Interesting. Any details on the things that you found particularly easy or particularly difficult to do with it?
- We wanted to collect files from various locations and push that in hdfs. Nifi seemed like a good way to build a self service setup. Once we setup source and sink, if we read + processed + removed file from destination, Nifi copied it again. We did not have control of always removing source files as soon as it was copied on destination. Components we used were GetFile, PutFile and some options in Conflict Resolution settings for these components
- Inspect file name, run a script to generate new subdirectories for partitions and place the file in appropriate partitions. Attaching a script was easy. Changing the destination path on the fly was not.
Some Complex cases - there are other ways to do this but setting this up in Nifi was a breeze
- Set up file collection from ftp, sftp, file copy from 20+ locations. This was painless, few minutes per source
- Add REST interaction within data flow easily
- Read CSV files and convert to Avro/Sequence files
- Read files and route part of data to different processors
We also ran into some strange bugs where Nifi got stuck in some type of loop and kept copying data over and over again.
We were able to do all this testing in 2 weeks. Give it a shot, it might work out for your use case.