In this context, you do. You need material structural stability (which is nontrivial, even for partially bermed structures; digging down further is a different story too) and for human habitation. Even above-ground you see retaining walls everywhere in places like New England because dirt likes to move. Worse, you'll need relatively expensive materials, and ones that are OK with contact with a lot of moisture. Maybe plastics can be the answer to some degree, if ones that are structurally sound and not tasty to microorganisms can be employed en masse, but the default answer is probably steel, and that won't last forever (or even all that long).

Beyond that? People aren't generally high on dirt walls and floors. And smoothing stone to presentation levels in-place, ensuring regularity, etc. is not a trivial task.

It could be done, don't get me wrong. But we have wear and decay problems above the surface that probably pale in comparison.