Exactly. There's a vicious cycle of expensive payloads driving costs of launches up (extra scrutiny, fewer launches, less experience, less cost-reducing hardware innovation - the RL-10 alone is sixty years old and quite expensive to build by hand from thousands of components, for example) in turn again driving costs of payloads up (you want the best bang for the buck if the price per kilogram is high, so you use overengineered spacecraft designs and materials).

Fortunately, there seems to be no shortage of ways out of this - new large markets like massive LEO constellations and fuel depots for more capable exploration missions immediately come to one's mind.