>People do research and write papers because they have to produce something measurable, not because the research they do is actually interesting or important.

Yes, so much of this. I think it's a direct consequence of your next point:

>I wanted to stay in academia so I couldn't take any risks.

That's how boring research gets prioritized.