I think it's the culture/genes they were brought/born into. Because it can be genetically made into your DNA or you might be raised in that certain household/culture that you're used to.
The way their tongue moves is indeed the simple explanation. For example, in France certain parts of their mouth sound certain letters differently than how it is pronounced in English. Being trained to speak one way, then having to do it the opposite way, is tough, to say the least!
I took a summer program in the summer where many people there were immigrants allowing for us the learn a new language as the summer went by and use it everyday