Looking forward to next few years when we can finally abstract away all the back-end techs.
Can you elaborate on that?
Once performance starts to matter (either due to scale or time requirements) abstractions always have tradeoffs you can't accept.
As long as compute is a meaningful percentage of spend, the trade off will matter.
If you use a cloud provider there are managed solutions for data engineering pipelines.