I blame these things:
1. The plague that killed off most Romans.
2. Incompetent emperors.
3. Christianity (it had a pivotal role)
4. barbarians settling within the empire because of the Hun invasions.
More detail-
1. The Romans already were outnumbered by their subjects so that wasn't really the issue. The issue was that there were too few Romans to become emperor, to join the army, or to preserve their heritage. This is why that the Eastern Roman Empire became, in effect, a Greek state under Heraclius.
2. The incompetent emperors allowed things to happen that really shouldn't have happened. For example, under Vespasian the frontiers crumbled with the defeat of the Roman legion against Gothic knights at Adrianople. Emperors like Caligula, and Commodus were pedophiles, and lax emperors who did not care less about what was happening. Under them, the decline of power made headway that otherwise would have taken centuries.
3. Constantine gave Christians privelages. One such privelage was that they were allowed to travel the Imperial Road System for free. So, a massive increase of travellers wearing the roads down with no extra money to pay for it. It antagonized a lot of the old school, but at the same time it served as a force that kept the empire together for a little while.
4. Illegal immingration.
Barbarians were allowed to settle within the empire by lax emperors, and when the Germanic tribes grew large enough, the Empire had grown weak enough to fall victim to hundreds of thousands of barbarians.