Hacker News Reader

Comment by elchief | Hacker News Reader

I gave up on all these frameworks and just use Cron and Make and Python

Do you use some kind of library to help with transformations? If you have some volume using plain python can become a bottleneck no? Any tips in this area would be appreciated.

I'm not the person you were asking, but I, too, use mostly Python for ETL tasks as a good chunk of my job.

Python's speed, or lack thereof, is rarely an issue. Most tasks that I write/run finish in less than a minute, including the time for transformations. Some of those can get fairly hairy, as well. Most of the time is spent querying the DB and uploading to the remote.

For the tasks where the transformations are too complex, you can greatly decrease the time necessary by tweaking the queries (i.e., limiting rows to just changed rows, etc.). And frankly, once you realize most data doesn't need to update in real time, it doesn't matter how long the transformation step takes (as long as it's fewer than about 23 1/2 hours).

pyspark if big-ol' data sets

pandas otherwise

Airbyte engineer here. We use Dbt to normalise the delivered data at the final destination warehouse.

If it's not a destination warehouse, how transformations are handled depends on the connector implementation.

Depends on what you need. It's classic thinking you need a too big tool. Can imagine that setup working great for many cases.

This is cruel to your future self.

Reply on news.ycombinator.com