Hacker News Reader

Can you give more background on what your data pipelines are like? Are they mostly batch processes?

If so, I'd strongly recommend using a workflow tool like Luigi[0] or Airflow[1]. In a phrase, I'd say they're like "Make for data".

[0]: https://github.com/spotify/luigi [1]: https://github.com/apache/incubator-airflow

View on HN

Basically, we improved a shell scripts based ETLs with lambdas. Now we barely can maintain them, and when a something breaks, it takes an engineer an inordinate amount of energy and time to get them fixed. Since we process about 0.5 to 1.5 TB of time series(IOT) I was thinking about an architecture that combine AWS kinesis and airflow

Reply on news.ycombinator.com