There were many reasons. For starters, it involved getting rid of a large number of Native Americans, and while some supported this, others opposed it as they were peaceful. Another thing was that this meant the introduction of new states and it deepened the division based on whether the states should be slave states or non-slave states. Also they involved war with Mexico and some wanted this and some didn't.
