They were in 2 different sides. The Abolitionists and the Pro-Slavery politicians. Some were heavily against slavery and started a movement to abolish it, some even tried freeing slaves. Pro-slavery politicians were in heavy support of it and did everything in their power to keep it alive in the South. It got to the point where all the Southern States threatened to secede from the United States in the late 1850s, and even caused Virginia to split into 2 states, Virginia and West Virginia.
I would say that the phrase only applied to white people mainly because blacks were not considered citizens and were mainly considered property. The phrase had good intentions but it took another 100 or 200 years for blacks and other minorities to gain equal rights.
