You're forgetting health care costs, which have skyrocketed since the 80s. Interestingly, all of these costs are not part of official inflation numbers, so real inflation for middle class is much higher than the reports say.
https://fred.stlouisfed.org/graph/fredgraph.png?g=kipg
Another factor of two increase.