They worked hard to establish a strong middle class, which is arguably what fueled the majority of the Roman Empire's growth. The rich class also learned that a strong middle class meant lots of taxes, and thus lots of money in the upper government. This money fueled their expansion, construction of major monuments, and their strong military.
This is in stark contrast to previous civilizations that were built mostly on slavery. This isn't to say that Rome didn't have its share of slaves. Slaves were used in the building of many monuments of Rome that still stand today, but most of these slaves were prisoners of war. They were mostly used in dangerous construction projects and then later massacred for entertainment when they lose their usefulness. Rome had plenty of slaves, but not nearly as many per capita when compared to earlier civilizations.
So to your original point, these were likely middle-class dock workers that worked at this port. Just like dock workers today they made a decent salary (this was before unions, now dock workers do surprisingly well for themselves) and lived in reasonable homes and ate surprisingly well.
In fact I would argue that many of them probably were demoted in social status after Rome fell in the 400-500s AD. With Rome no longer the superpower that it once was, the desire to trade with Rome goes down, which meant that fewer ships come into port, which is bad for the dock workers that work at that port. Many of them probably lost their job and this may be the contributing factor to the food changes we see over the next hundred years with these people. There is far less money moving through their city now and more people move to a lower social-economic class.
Imagine if the US immediately collapsed. The billions of dollars of trade that goes in and out of the Los Angeles port (the largest port in the US) would drop by 90% or more. Many of those dock workers, who right now are strong middle-class workers, would lose their jobs and unemployment country-wide would increase dramatically (probably well over 25%). This would make it difficult for dock workers to find new jobs and many would start going hungry or reach poverty status.
It isn't too far of a stretch to assume that this is what happened to Rome's largest port after Rome collapsed.
Interesting. What are the numbers?
After the collapse of the USSR, regional governments and foreign governments stepped in. If the USA collapsed, New York, Texas, and California would have something to say about it. My point is that "collapse" is rather nonspecific.
The billions of dollars of trade that goes in and out of the Los Angeles port (the largest port in the US) would drop by 90% or more.
That narrows it down a bit.
There was nevertheless a 30% drop in GDP. That's bound to have some effect.
I do wonder if the cost of more exotic food in Portus would have been significantly cheaper than 15 miles away in Rome.