Fluentd: the missing log collector

blog.treasure-data.com

50 points·kzk_mover··7 comments

Cool! Handling each event record as a structured data (JSON) is very interesting, because it can connect with MongoDB(DocumentDB) seamlessly for stream output or aggregation. This slide is written about a collaboration with Fluentd and MongoDB. http://www.slideshare.net/doryokujin/an-introduction-to-flue...

awesome slide!

Excellent, and I am going to try it. But I still think using something like Flume has some advantages mainly because of Hadoop eco-system. For instance you can plug-in the log data to HBase and use Hive to write to high level abstracted queries and run on Hadoop. I am guessing but seems like there are plugins on the way for various systems but not Hadoop. Update: Also Flume can use any data stream, for instance Twitter stream so not limited to log Analysis only.

Yep, we agree with your opinion CURRENTLY.

We're developing Hoop (REST API for HDFS) plugin, and also Stargate (REST API for HBase). It requires additional setup at the infra-side, but still very useful and performs well.

You can feed Twitter stream by writing in_twitter plugin. That will 30 lines of Ruby code :-)

Err... what? Am I missing something? Just use rsyslog and transfer the logs over TCP/IP to a central server. No need to re-invent the wheel.

It's more about log analysis than just transferring the files over. Apart from splunk there aren't many projects which handle this well.

It also seems to have many input plugins apart from standard syslog. And that's great. I think we're way past the time where lines of text just don't work well enough. First you format your data to text, then you want to get your original data back from that text by parsing... seems like a wasted effort.

(although there's graylog2 with GELF)

coooool!!

Reply on news.ycombinator.com