There is no doubt that urban centres basically disappeared after collapse of the Western Roman Empire, and that population density went significantly down. Once the city-based specialists were gone, so was any ability of rural folk to buy anything that could not be produced by primitive methods in their own community. And without an efficient trade network, there was no way to import food if local crops failed. Hence, famines, which the previous empire was mostly able to hold in check by moving food over the sea at big distances.
A major problem of the Early Middle Ages was diminished security - all those Viking, Avar, Hun and Pecheneg raids were absolutely real, and their targets weren't "the 1%". Of course loot from the rich would be taken, but so would poor young women for sex and their children into slavery, and their meagre food reserves for the raiders to eat. That is what happens to settled people without an efficient defense of their borders.
We have had two big imperial collapses right in Europe within living memory - the Nazi Reich (by war) and the Soviet Union (economic). Ask the survivors if they "noticed". They absolutely did. I would even say that the working class "noticed" the most, as they usually had fewest reserves to survive the subsequent chaos.
It wasn't that different in the past.