I don't think this is accidental complexity, it's essential complexity. To the extent that it seemed easier in the past, it's because we ignored some of that complexity and paid the price. Putting up a truly production-grade service is fundamentally hard.
Now, I think it probably will get easier over time as we grapple with these problems. Integrating with auth, for instance, should be easier, and that can be solved with some more code, and formal and informal standards. It's not all essential complexity. But I think a good deal of it is, or at least it is from anything remotely resembling our current perspective.
I think this is the kind of essential complexity which we faced when developing operating systems. OSes are fundamentally helpers, which don't solve the application problem, but make the solving easier (a good OS makes it easier by much).
So we can and should use the results which we got from OS development.
I've been working on building something like this for a year now [1] and it's certainly quite different from how we do backend development normally,but that's also the point :). The tricky part is to find abstractions that are general and non-leaky so they can be leveraged to build a wide variety of software. Feedback appreciated!
There will always be a meaningful amount of effort involved in supporting your own systems, and that cost (be it in ongoing maintenance time, or in lost time because your own system failed) _can_ always be higher than paying to outsource the work to a trusted third party.