Although there is no precise reference to the text, <em>"the middle of nowhere" is a phrase representing a very remote and usually isolated place.</em> So, the narrator and his family must think that his dad's home is far away from them and possibly far away from civilization, hence they call it "the Middle of Nowhere".