Introducing Tank, a very high performance log | Hacker News Reader

Introducing Tank, a very high performance log

medium.com

12 pointsmarkpapadakis12 comments

Wish the post talked real numbers, as "very high throughput and very low latency" means very different things to different people. For example, a 1 millisecond latency is "very low latency" for web related things, but might be a lifetime when comparing it to a top of the line trading application.

You are right, I should have elaborated on those claims, and I plan to add a relevant Wiki page to address this shortcoming. However, you can use the tank-cli bm command to benchmark client to broker to client (end to end) latency.

We get sub-ms (a few hundred microseconds) latency here, and even on a very busy Tank broker (it takes effort to get there), I don't foresee situations where you can get over 1ms latency for any operation.

For throughput, Tank relies on sendfile() and readahead() for very high throughput - and thanks to the very tight packing of the messages, you will likely saturate your disk/network easily.

You may also find this interesting: https://github.com/phaistos-networks/TANK/issues/14

It would be much less confusing if you didn't compare a storage mechanism (append-only log) to message transporters (Kafka).

Would love to see a non JVM kafka alternative. But, I wish people would stop building infrastructure with C/C++ when there are safer alternatives like rust and go.

Really? Safer? How about writing things in languages that are more mature instead of hoping on the newest bandwagon.

Said the assembly programmer to the C programmer.

Given that Kafka is a message router, what don't you like about Fluentd (Ruby) or Heka (Go)?

Besides the fact that ruby performance is terrible and heka was deprecated and abandoned by Mozilla due to design flaws (https://mail.mozilla.org/pipermail/heka/2016-May/001059.html)?

I'm not to defend Heka, as I don't use it, but those "design flaws" only concerned development, not operations. If you read the message you quote, you would see that Mozilla is still using it.

And I take it you have measured Fluentd's performance, so you say so from experience, not flail around the typical opinion about language that can safely be attached to Java as well?

How does this compare to ZeroMQ?

This is a message broker. You can publish messages to it, and it will retain them for as long as you want(it supports retention and compaction semantics), and you can consume the messages, starting from any message(messages are identified by sequence numbers). This is similar to Apache Kafka and - to a lesser extent - Azure Events Hub )

Reply on news.ycombinator.com