By law they don’t have too
Slave-holder: Jackson was wealthy and owned a large plantation with many slaves.
His common man persona came from his time growing up poor and his time in the military. Though educated and wealthy, he was not as well groomed as earlier presidents and that is why commoners connected with him. A small percentage of Southerners had plantations or slaves, so many of Jackson's followers would not have this in common with him.