They have been looked down upon for decades and possibly centuries. It had become a sort of a social norm in a sense to view them that way, hence all of the public opinion has shaped how we think of them today.
The Founding Fathers felt the right to petition and assemble which supported the organizing principle of popular sovereignty of the Constitution. The founding fathers included John Adams and Benjamin Franklin.