Open-Sourcing Twitter Heron
blog.twitter.com
blog.twitter.com
Comparing Heron to google millwheel is interesting because of the design choices they made. Heron support at least one and at most once message guarantees but at Twitter most job ran with acked turned off so it was at most once with acknowledged data loss ( they had a batchjob doing mop up work to pick up missing data). Google on the other hand implemented exactly once semantic by doing idempotent sinks/ watermarking and managing out of order messages plus deduping support. Since both Flink and Spark will be implementing Apache Beam (millwheel's predecessors) model, only reason I see someone picking heron instead of Flink/Spark is that they are operating at massive scale that flink/spark dont support yet
Flink certainly scales just fine, for what it's worth. Flink 1.0 is quite good, and I'd consider what I'm doing "massive scale"; the ease of 1MM+ QPS with decent p95 latency via Flink surprised me compared to other systems that I investigated in this space. Most hip-fired benchmarks, including that awful Yahoo! one that everybody cites, use Flink poorly.
Rest of your comment is great and I couldn't agree more. Spot-on analysis. Twitter made a misfire here buying out Nathan Marz, neglecting Storm in favor of Heron while the rest of the field advanced (notably Google's open source work and Flink), announcing Heron which is so much better but keeping it to the chest for a while, then losing out on both of their streaming engines in time. Storm and Heron both feel too little too late, particularly Storm's recent (vast) performance improvements which a lot of folks I know kinda shrugged at and which is kinda too bad.
The Dataflow/Beam/Flink stuff is the compelling horse right now, to me. Just my personal opinion.
Why are Yahoo! benchmarks awful? How did they manage to use Flink poorly?
For people who don't know what he is referring to, check this: https://yahooeng.tumblr.com/post/135321837876/benchmarking-s...
Ditched it and decided to do it in Spark with Scala (making it a good excuse to learn Scala). With so many real time options popping up and around, deciding which to pick is getting harder and harder.
Edit: Not so sure what they are using for networking. Seemed like cpp-netlib at first, but I don't think so now.
Their default event loop implementation uses libevent, and they're using protobuf in some of their higher-level networking classes, but the networking code itself seems to pretty much be plain sockets (with a thin portability layer on top in a few places).
https://github.com/twitter/heron/blob/master/heron/common/sr...
For anyone curious, it's https://github.com/twitter/heron
Getting Started: http://twitter.github.io/heron/docs/getting-started/
Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Scala nicer than both). These days, it comes down to whether the JVM memory overhead is a big deal on the specific project, and if not, for the class of projects discussed in the previous paragraph, I'm probably choosing Scala (but if my teammates object, then it's back to Java).
IOW, "Let's say you want largely random pauses to impact latency non-deterministically." I'm not knocking GC, but it has serious consequences in systems that make this a weird assumption with which to start, particularly the "big" GC languages that you named. Scala in particular with its immutability generates a lot of garbage if you are not careful, and we were constantly fighting GC pressure (and, indirectly, our achievable concurrency and efficiency) at a household name Scala app. The number of JVM developers who are aware of this and capable with the memory subsystem -- i.e., off-heap strategies such as that used in Flink (relevantly) and some of the clever speedups in netty -- are dwarfed by the number of JVM developers in the whole, so you really need someone who understands these problems to efficiently scale a JVM language.
There are better systems languages if you can move away from GC, such as Rust. And modern C++ is fantastic for systems, and it's too bad it gets overlooked because of bam, your first assumption there. Most systems at Google are in C++ because they invested into the infrastructure that you need to support it. It should tell you something that they're heavily involved in each new version of C++.
(I'm talking about systems, not applications. GC is a viable tradeoff for programmer productivity in applications.)
Yes. That's not a problem to find and hire one or two of these guys to write a core. And then hire a ton of regular Java developers to write a ton of code you need around that core, educated and managed by those two smart guys.
Compare it to the problem you have going C, C++, Rust, Nim way. Now you need to hire a ton of guys who can write good C++ code (because it must be good to not to crash every 5 seconds, at least), or even worse - a ton of guys who can write any (not even good) Nim code.
There are three focus areas of wrongness in your comment (maybe four if I'm feeling particularly culturally trolly) but I'll stick with that one and leave the others to others.
Jed, once again you are laying claim to experience you do not have. If you ever once had touched Scala at Foursquare it would be a different story. This is just a bunch of tired platitudes.
- Erlang (this fits your criteria at least as Java...)
- OCaml (concurrency isn't the best)
- Haskell (also fits your criteria very well)
- D (easy to use C libraries and sometimes C++)
- Rust (not GCed, but why is GCed a requirement?)
- Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else)
Java only seems like a good choice if you ignore all the languages that are a better choice than Java. IMO Erlang seems like the choice here (or Elixir if you don't like Erlang syntax).
After that, what's left?
At the moment, none of the above languages have any significant talent pool to draw from. If you're operating at "Twitter-scale" (I guess that's now a thing), you also need to be able to spin up a team of people that can get to work quickly without having to learn a language as they go. That eliminates a great number of languages just for logistical reasons. Rust might get there (and I hope it does!), but it's not there yet.
Maybe Twitter already has a significant number of skilled Java engineers, so they decided to use what they already knew really well. There's nothing wrong with that. Choosing any of the other languages you listed would have been a far riskier strategy.
With Java, you can directly import a lot more, especially since the vast majority of OSS in this ecosystem is written in a JVM language.
That said, I'm hoping to deploy my first Rust project to production soon. It's come a long way for being such a young language.
You could be right. For a number of the languages I mentioned I banked on really good FFIs allowing easy usage of C libraries. Perhaps in some domains, including stream processing, there are more and better Java or C++ libraries for them to build on.
Another reason to run on the JVM is that often your applications run on the JVM, and the JIT -- which is getting better and better, and will get incredibly good in Java 9 -- can really do cool stuff when it optimizes across the app/infrastructure line. I've just seen a paper[1] showing 3x performance boost when rewriting parts of SQLite in Python, and letting the JIT optimize the DB together with the app.
Aside from Erlang which is far too slow (most Erlang apps use C for the data plane)
Edit: You are still right. The hiring issues are relevant, and apply to Erlang also.
Pet peeve: OCaml's concurrency is pretty good, it's parallelism that it struggles with.
it's also worth nothing that Java comes with the JVM and Hotspot JIT, which earn it a bit of swagger.
- OP mentioned performance. Erlang is terrible for - Haskell
- OP said non-esoteric.
- There's a significant ramp up before you learn to deal with the compiler and lifetimes.Erlang's CPU performance is not bad IME (though admittedly I can't think of anything I've implemented in both Erlang and Java to compare). But that's beside the point; you pick Erlang for networked and I/O bound tasks. Twitter mentioned running Heron on several hundred machines—at that scale, single-thread performance starts to matter less and Erlang's scaling efficiency closes the gap.
> Haskell
Haskell may have been esoteric in 2005, but it's hardly so anymore. (Also, I mention Mercury, and Haskell is the language you decide is too esoteric‽)
> Rust
So you're saying they wouldn't use it because they didn't know it. I suspect that's the real reason they chose Java—not because it is necessarily the best language suited for the task, but because it's the best of the languages Heron's developers knew.
Erlang is very slow when it comes to CPU bound tasks. It's nowhere near Java for tasks like this.
C# runs absolutely fine on nix.
There's something of an inertial effect with Java, where people continue to write Java because everyone else writes it (and thus it continually improves). What other platforms do you know of that have a similar track-record of performing in large-scale software?
I like coding with Java 8. There's lots of new syntactic sugar that let you cut down on all the boilerplate and cool new standard library features like Java 8 streams that let you do functional programming. The best part of Java 8 streams is they perform really well. You get more readable code along with better performance.
I'm curious what you found unpleasant about it.
But honestly...? It's taught in schools a lot so more people are familiar with it. The greatest disservice to an entire generation of programmers is Java being standardized at Universities.
We run Kafka Mesos https://github.com/mesos/kafka allows for as a service implementation for broker clusters for users and apps.
Vagrant for running example https://github.com/elodina/heron/blob/v2/contrib/kafka9/vagr...
I'd appreciate if it were called Spark Microbatching, but we can't have everything.
How many streaming analytics use cases are ok with the JVM, ok with 10s of millis of latency, but not ok with 100s of millis of latency?
Microbatch latency immediately rules out several useful applications of streaming. I also didn't say anything about the JVM nor it being okay for my purposes. You did.
I can back up my statements on streaming from Flink to Samza to Dataflow/Beam to Storm to MillWheel to Spark "Streaming" and back again because it has been my primary focus (literally thinking of nothing else) for a couple years. Please accuse someone else of FUD because that's a relatively veiled way to say "you don't know what you're talking about," and I assure you that you're (condescendingly) wrong. I think you're also coming at that angle from interpreting me as negative on Spark Streaming. Read carefully.
I've run spark streaming jobs at 250ms batch times. This comment you made, right here, is why I used the word FUD.