Let me automate that for you: removing pain points from data pipeline setup

I like this. A nice, practical solution to a real problem!

Disclaimer, I work for AWS. At some point there's just too many of these data streams, so a warehouse no longer does it and we turn to data lakes. The advanced version of what the OP is trying to achieve, without any actual manual work, could be an AI, and be called Lake Formation, https://aws.amazon.com/lake-formation.

Really great story. I feel like the same low key problem at my current workplace would instead have spawned a “let’s replace everything with a data lake”-project, spawning several years with an army of consultants and at least one new major external platform. All to solve these small pain-points that you can weed out if you have a passionate and talented team who are close to the problems and have freedom to build the solutions.

Nice writeup. I've used https://www.singer.io/ to good effect in the past, seems like this is a similar, maybe more manual/custom approach.