Ask HN: Are you using Apache Nifi?

The data streaming world is saturated. There is a real cost associated with choosing between Spark, Flink, Akka, Heron/Storm, IBM Streams, Edgent, or even NodeRed — to the extent that its impossible to evaluate everything. Usually, developers start from the most commonly used framework and go down the list to evaluate how it plays.

A streaming framework can be kickass but see no real adoption because of this. Hopefully that isn’t what’s happening here, or why someone is posting to HN to ask “who’s using us and why?”

I work for an IT consultancy firm, and we have some customers asking for it right now (mostly in the banking sector).

The reason is Horton works deprecated Apache Flume and bundled Nifi instead.

Personally I don't like it. I prefer other ETL tools like Talend. I also find confusing the user interface.

The good part is that NIFi processed are buffered, and you can keep metadata (like data provenance) with the records which is very helpful for compliance.

We use composable: https://composable.ai

We view it as a middleware layer and integration engine. Tools like composable save enormous amounts of development time, and simplifies maintenance.