Barbarians are your answer.
However, they were not barbarians, but tribes from the North and other regions. They were given the term "barbarians" by the Roman Empire, for the tribes were inferior in build-up of their communities and fighting skills, and were extremely culturally different from the Romans (As we can see that the Romans did not call Greeks "barbarians" because the Greek civilization looked more built-up).
However, the Barbarians were only one of the factors of the decimation of the Western Roman Empire. Civil War, civil unrest, famine, governmental problems, social injustice, insane emperors, failure of leadership, etc, were all reasons for the Roman Empire to start collapsing.
-----------------------------------------------------------------------------------------------------------
While the western half of the Roman Empire started to crumble and fall, the Eastern half by the name of the <em>Byzantine Empire</em> continued to last, and carried on the Roman tradition. However, they soon fell during the Crusader's age to the Seljuk Turks, and the Roman Empire was non-existent until the so-called "Holy Roman Empire" arose. However, while it was called, " ' Roman" ", it had nothing in common with the Roman Empire other than in name.
-----------------------------------------------------------------------------------------------------------
So in this case, we must look at what you meant by the "Roman Empire". Do you mean the western half? If so, then your answer is Barbarians. If you meant the eastern half of the Roman Empire (Byzantine Empire), then it would be the <em>Turks</em>.
~
