The author already touched upon a few hypotheses why the industry has arrived at this state. There's one I'd like to highlight: The author, now having gone through this experience, would actually be in a perfect position to found their own CRO and implement all their efficiency learnings there. However, they probably look back upon this experience and think it was super painful and would prefer to do fun stuff like developing software instead.
That, for me, is the main reason. There are smart, structured, pragmatic people who got acquainted with this industry, but they choose to leave it.
And that's sad!
So the solution would be to find a group of smart, structured, pragmatic people with a high tolerance for bureaucratic pain and give them the resources to make this industry more efficient.
And here's my plug: That's exactly what we're doing at OpenRegulatory [1], albeit for medical device software and not for clinical trials. Among other things, we've published all our document templates for free (also on GitHub), host a free Slack community for companies to help each other out, and offer a compliance SaaS with transparent and fair pricing. It's still a long way, but it's a start :)