This prediction could be almost exact. Social referencing is the process in which a child in his first years starts to understand the world around him, using as reference the way the adults who give him affection view it.
If his parents are reluctant to enter such a place, the child could slowly develop an aversion for that house, just for the fact that one of his caregivers does it.