Placing a confederate symbol on an office government building would send incredibly regressive ideals. It would remind everyone constantly of the ideals the Confederacy, and it could possibly send the message that those ideals are welcomed in the government. It could also signify a rebel takeover, as the Confederacy's secession was essentially a rebellion.
After The Union had defeated The Confederacy, the right to vote was guaranteed to all races. This was possible because the Union was in favor of equality.
