If you're considering Apache NiFi, you should also look at Apache Airflow and Uber Cadence to decide what model would work best for you.
I have used Nifi a little bit and Airflow not at all. Reading the home pages of the two products, it's hard for me to know when it would be more appropriate to use Airflow than Nifi.
They both schedule jobs and move data according to control flow topologies that you build in a GUI, right?
- Would love to see the ability to develop custom NiFi processors in Go/Rust/Elixir etc.
- XML is a big pain in the rear.
- Being container-aware is big win. Stateless is even better.
I see a good opportunity there for users like me to explorer NiFi's capability in the future.
> They both schedule jobs and move data according to control flow topologies that you build in a GUI, right?
Airflow on the other hand is designed to run scheduled jobs (whether it be batched or otherwise). The 'job' can really be anything - build / data processing pipelines, system configuration management pipelines and so on. In Airflow parlance, one can create connected DAG's as pipelines that massage the data in a way you intend it to.
They both share some commonalities but I do gravitate towards their use cases being subtly different and an important one highlighted above.