Confluent, a company for Apache Kafka and realtime data
linkedin.com
linkedin.com
In the same way that Hadoop is starting to feel outdated, but HDFS doesn't seem to be going anywhere -- I think we'll see a lot of innovation in stream processing frameworks in the next few years, but Kafka will just keep on going.
There's a missing piece in the realtime puzzle, at least one that I haven't been able to find, for which Kafka is an overkill - perhaps someone here knows of a solution:
I have tens of system endpoints connected through unreliable (Line-of-sometimes-occluded-sight, 2G and 3G WWAN, some are in vehicles so connections are intermittent).
I just want to consistently tail their logs in a bandwidth-efficient, connection-drop resistant way; and I can't find any standard thing that does this.
Kafka would fit the bill in general, but would require a lot of work (reading textual logs into kafka, querying kafka for new stuff across connection, reading from kafka and writing to text files) - and I'm not sure how well it deals with dropped connections.
My existing solution is to rsync the log directories (--append, --inplace) as infrequently as I can from an operational view, which is 1 minute. It is relatively bandwidth efficient (although could be much better), robust with respect to connection issues, and generally works.
However, it is less efficient than it could be: if directories have a lot of files, like /var/log often does, there's a lot of sync overhead. The delay is 1 minute instead of a couple of seconds (which is what you would get with a simple "tail -f" through a TCP connection), and it doesn't play well with common log rotation schemes (though that's relatively easy to work around).
Anyone has a better solution, kafkaesque or otherwise?
kafka sounds like an enormous overkill. If you want to store the logs locally while tailing, just add in a tee.
But it my case, I have "30 minutes on, 5 minutes off, 90 minutes on, 90 minutes off, 2 minutes on" kind of situations, in which anything that doesn't track what was already transferred and what wasn't, will lose data. (zeromq's buffers also have limited capacity and/or are tied to a process on the other side - if it restarts, buffers are gone).
If you don't mind using a 3rd party service, you could look into using Papertrail, Loggly, etc.
We're working on a database/cache/messaging system too, http://github.com/amark/gun it is dedicated to removing the pain I and other Javascript/NodeJS developers had when it came to managing/debugging databases (devops and sysadmin work is frustrating).
http://www.quora.com/RabbitMQ-vs-Kafka-which-one-for-durable...