Hacker News Reader

1 - Hide your ETL logic in REST APIs and use common microservice patterns for monitoring, logging, etc...

2 - Use PythonOperator to call your APIs.

It's working really well for us.

This is a very bad idea. You cannot hide ETL in a Rest API, at most you can hide the T part and it's a lot more work. you need to build an extra service that reads the data from a database and does some padgination. It's way more complex than a one-liner to get data from a jdbc connection and way much slower.

I used to do that, but having to deal with timeouts in long running jobs was frustrating because i never knew if something was successful. Could bump up the timeouts, but theyre there for a reason.

Reply on news.ycombinator.com