The South had long been a center of cotton production and a place with a large textile industry. The pay for the workers, though, had been low for all of that time.
Textile was known for being the lowest paying industry in the country and people would have to work for hours day in day out and they would still not earn enough. When he said this was wrong critics fought by claiming he was a socialist and what not.