miguno

Small and friendly correction: Kafka bills itself as "a distributed streaming platform".

Also, a heads up: we'll soon publish a blog post on the subject of "is it ok to (permanently) store data in Apache Kafka".

> No offense, but this just shows your ignorance of the functionality that you're leaving on the table by considering SQL a cough-legacy-cough solution.

No reason to get ad-hominem.

When I said cough-legacy-cough, I was referring to the situation you happen to be in when e.g. taking on a new project. If, for example, your company has been fully committed to Postgres since several years, your chances to change this and move to a different architecture (or even sticking to the same architecture but using a different RDBMS like MySQL) are pretty low.

Kafka can perfectly keep your data around forever. The only limitation is available disk space (and databases have the same limitation). I'm not implying that it is always the best idea to use Kafka as a long-term storage solution, but likewise a database isn't the silver bullet here either.

> so I'd probably be uncomfortable assuming that it won't do something and "tidy up" my very old data at some point in the future

Kafka doesn't "tidy up" your data unless it is configured to do so. What's true is that, by default, Kafka will keep your data around for a week "only" but that's a configuration. And most people change it to whatever fits their use case (some lower it to a few hours, some increase it to months or years; others configure it to keep data around forever, typically in combination with Kafka's so-called "log compaction" functionality).

> Since it's supposed to do this from time to time out of the box, you'd have to be very careful not to let anyone tweak the custom config to revert back to this behaviour. RDBMS won't delete things unless you tell it to more explicitly.

The DBAs I worked with would now say "Hold my beer..." ;-)

> While I think storing things in Kafka is fine generally, there's no way I'd not have a more perminant store of the data somewhere so that I can recreate the Kafka data store if I need to.

What's interesting is that more and more users (from my experience) are actually beginning to treat Kafka as the source of truth, and rather recreate other data stores -- like RDBMS, Elastic indexes, etc. -- from it. If you like RDBMS, you can think of Kafka as the DB's transaction log.

IMHO a lot of these discussion is about personal preferences, the situation that you are in (cough legacy cough), team skills, etc. There are often good reasons to use Kafka rather than RDBMS (in this context) but also vice versa, or completely different technologies of course (like blob stores, e.g. S3).

> I think they want to it to be durable in the sense that even if the events are already consumed they can still play them back to reindex elastic search (which is a thing you need to do periodically).

That (replaying if needed) is exactly what Kafka allows you to do, unless I misunderstood what you wrote.

To add to what nehanarkhede said: I'd rather see it as a testament to the powerful building blocks of Kafka and its Streams API that you can actually implement a project such as KSQL with very few engineers.

Also, the commits in KSQL reflect only parts of the work -- it doesn't include design discussions, code reviews, etc.

Lastly, keep in mind that the git repository was cleaned (think: squashed commits) prior to publication, see the very first commit in the repository's timeline. So you don't see the prior work/commits of other Confluent engineers that went into KSQL.

Disclaimer: I work for Confluent (www.confluent.io). That said, let me clarify that Kafka Streams is a component of the Apache Kafka open source project. There's nothing proprietary about it. The reason the links below point to docs.confluent.io is because the Apache Kafka docs for Kafka Streams are not ready yet; but they will when Kafka 0.10 is released, which will be the first official Kafka release that includes Kafka Streams.

> 1. How does it scale horizontally?

See http://docs.confluent.io/2.1.0-alpha1/streams/architecture.h....

> Do I have to deploy this on library on multiple nodes?

Kafka Streams really is a normal Java library. You include it in "your" Java application just like you'd include a library such as Apache Commons, Google Guava, or JUnit. There's no "deploying" of this library -- i.e. no need to pre-ship it to machines; also, you don't install a cluster or anything like that for Kafka Streams. It's a purely client-side library.

Interestingly, many folks I have talked to seem to be confused initially about the "library" aspect because I suppose, in the big data world, one has simply become used to (and pigeonholed by?) frameworks like Spark, Storm, etc. that you must deploy and operate separately (and which then dictate how they want you to write your own apps against them).

Does that make any sense?

> 2. Is RocksDB instance local to your consumer/ thread/ node?

Yes, it is local. But local state stores (RocksDB is but one choice for implementing these) are also replicated for fault tolerance. Details at http://docs.confluent.io/2.1.0-alpha1/streams/architecture.h... and http://docs.confluent.io/2.1.0-alpha1/streams/architecture.h....

Regarding joins: We have documented some information at http://docs.confluent.io/2.1.0-alpha1/streams/developer-guid....

I hope this helps! If these pointers above aren't sufficient, please drop us a note to dev@kafka.apache.org (http://kafka.apache.org/contact.html). We really appreciate any such feedback (code, docs, whatever) to ensure people have an easy and fun time to get started with Kafka as well as its upcoming Kafka Streams library.