There would be too much innocents dead, and they feared setting an imperialistic or authoritarian viewpoint for other countries will cause the other countries to unite against them.
It impacted politics in the mid-19th century because of the large inflows of immigrants after 1845 from cultures that were different from the existing American culture.