Saying Burton is fond of colored people becomes ironic after a few lines being read. He does have some consideration and feeling for the motherly figure of mammy, he does think colored people are good servants. However, they must "keep their place". They should not think themselves worthy of sitting at the same table as Burton which, by the way, he wouldn't do, not even for a million dollars. Thus, we can see how such "fondness" is limited by prejudice, by the belief that a colored person's place is beneath his own.
