Political,religious and social reasons should not interfere with the rights to assembly of any person or group of persons. the right to assembly is a fundamental right protected i the first amendment, and a key component of the bill of rights.
Back then , African American citizen were heavily discriminated by society. For example, they did not allowed to enter the 'white folks' ' Restaurants. They don't even allowed to enter a same public restrooms as them. They also doesn't count as citizen and not allowed to vote.