Can you tell what story this is from
It is a story parallel to what is happening in the Usher house.
The palace in the description represents Roderick Usher, whose hair used to be blond, as the yellow banners that wave in the air; through his eyes (the windows of the palace), a harmonious mind could be perceived, and wise words used to flow from his mouth, described as a door of pearl and ruby. However, gloom befalls on Roderick; his hair turns gray, his eyes are reddened by tears, his thoughts are discordant, and his mouth utters only the wild laughter of a madman.
The answer is the Climax of the story.
I think it was when they started the point of telling stories of the witches and what they did with the people it just started to spook them out and they feared that maybe they were honestly true. And I think at that time it really wasn't the best time to stand out and/or say/do something out of their league because they would think you were a witch.
(And we all saw Sleepy Hollow sooo) :)
Hereś something I wrote a few days ago. It meets the requirements, but might not be what you expected. If youŕe not satisfied with it, feel free to report this answer.