Although there is no precise reference to the text, <em>"the middle of nowhere" is a phrase representing a very remote and usually isolated place.</em> So, the narrator and his family must think that his dad's home is far away from them and possibly far away from civilization, hence they call it "the Middle of Nowhere".
The setting of "Marigolds" is during the depression or the 30s. You know this because the story says, "The depression that gripped the nation was no new thing to us" and "the black workers of rural Maryland had always been depressed".
