Generally speaking it was the "bankers" who were opposed to Andrew Jackson's policies, since Jackson viewed himself as a "man of the people" who was generally against big banks.
Bankers were the most opposed to Andrew Jackson's policies as he was against big banks. During his presidency, he moved the federal government's money from the Bank of United States of America and into state banks.
