Hacker News Reader

Comment by leblancfg | Hacker News Reader

That looks very interesting indeed, and love to see some more development in that space.

Would anyone care to explain how it differs from Airflow? I had dismissed Airflow some months ago for my use case (Windows, and large number of dependencies were an issue at the time), but would still like to eventually migrate my ETL scripts to a solid framework sometime in the future.

View on HN

With a little bit of work you could probably trigger tasks on a Windows Machine, in particular if the work you need to do is mostly script and/or SQL based. The main limit is that Gunicorn for the Airflow webserver is not currently compatible with Windows, but the scheduler should work fine. I believe it might be able to run in the Windows Subsystem for Linux, but I don't think anyone has tested it as of yet. (Source: I am an Apache Airflow committer)

Thanks! To be fair, I was looking at it from a single-user perspective, and was optimizing for:

    time it takes to setup and move my ETL to Airflow + time I will save in the future by using the framework

    time I will spend fixing and maintaining my already-existing ETL scripts

so I dropped the notion after I hit the first major snag. But I should put a reminder to take a look back at it in a few months!

Mara is designed to be simpler than Airflow. It positions itself between plain scripts and Airflow.

What features does it forgo to be 'simpler'?

The main feature which jumps out at me, I think it schedules on a single node, where Airflow can orchestrate across multiple nodes. Airflow is also starting to implement more enterprise-y features, like role-based access control, which should be coming to the next release.

(author here)

That's absolutely correct. Mara uses Python's multiprocessing [1] to parallelize pipeline execution [2] on a single node so it doesn't need a distributed task queue. Beyond that (and visualization) it can't do much. In fact it doesn't even have a scheduler (you can use jenkins or cron for that).

[1] https://docs.python.org/3.6/library/multiprocessing.html

[2] https://github.com/mara/data-integration/blob/master/data_in...

Reply on news.ycombinator.com