Assuming you're referring to the American landscape, the main reason why most of the buildings on the plantation were built at the front of the mansion was so slave owners could monitor the slaves.
Recognizing privilege is the simple part, but not the only part. It is more important to do something after you have realized your privilege. Recognizing your privilege leads to recognizing those who are not privileged in the way that you are. After you realize that, you must be an ally to those who do not have the privilege you do.
Hope this helps have a great day/night :)