They did not want to share political and economic freedom with freed slaves.
The answer is the legislative branch
During the colonial times, only the rich and upper-class male citizens were able to learn in most if not all cases. We also now have universities and colleges that allow more people to attended, and do not restrict the based on wealth, ethnicity, or gender.