Yes, the Elders did take the memories of the community. The Elder's believed it would be morally right to exact each individual into what they called, Sameness. They also took colour, variety, and freewill away from the citizens of The Community.
I believe "think he had seen a ghost" is the best answer.
