The economic and social conditions of non slave-holding white southerners were often very bad, since they had to compete with free labor for many of the jobs that they needed to stay alive--meaning that they had very limited economic opportunities.
Madison Supreme Court ruling illustrates that U.S. law cannot be enforced if it conflicts with the Constitution as per the Judicial principle known as Judicial Review.