frankmcsherry

It's a good question. Part two goes in to this a bit more (the post was too long). But, the performance seems to be due to a few things:

1. Rust/C#/etc. vs Java. Stuff just goes faster with less effort.

2. The model is more "programmable". Rather than select from pluggable pre-fab algorithms, with certain trade-offs already made for you, you are able to write your own code where you need it. The system runs your code and really not that much else.

Stay tuned for when we actually push that content out.

I'm sorry, this hasn't been my experience, or the experience of anyone I know writing stream processing systems. Having written one each over a GC and not (Naiad, and the timely dataflow in Rust linked above), dev time on the former was disproportionately spent fighting GC perf issues.

The folks I know doing stream processing over the JVM have garbage collection (and avoiding it) as perf issue #1. For example, you can read about Flink's approach to avoiding GC at

https://flink.apache.org/news/2015/05/11/Juggling-with-Bits-...

You can read a more scientific discussion of GC vs region allocation here:

https://www.usenix.org/system/files/conference/hotos15/hotos...

In my experience, concurrent access to shared data is relatively rare in a well-designed data-parallel platform. Data are exchanged rather than shared, at which point ownership of memory transfers from one worker to another. Rust is great at understanding ownership, and at managing memory in this sort of situation.

You can contribute to:

https://github.com/frankmcsherry/timely-dataflow

https://github.com/frankmcsherry/differential-dataflow

Or, just tell your friends. :)

Better yet, write some python / pandas / dataframes / whatever_the_cool_kids_need layer on top, and rule the next big data drama cycle.

Shameless self-promotion for big data in Rust:

https://github.com/frankmcsherry/timely-dataflow

https://github.com/frankmcsherry/differential-dataflow

Nothing much useful to contribute, except that they mostly work, are largely unsafe-free (unsafe in some sorting, and a Drain replacement), and build on 1.0 stable.

SIGMOD 2015 proceedings are open access, meaning the conference has arranged with ACM to provide the content to pretty much everyone.

That being said, the submission is to someone's (Adrian's) interpretation of the work, and is arguably more useful than just linking at the pdf itself.

The paper contains the motivation: "For our needs at Twitter, we needed a stream processing platform that was open-source, high-performance, scalable, and was compatible with the current Storm API."

I guess 3/4 isn't bad.

Oh, it's coming around, alright.

https://www.usenix.org/conference/hotos15/workshop-program/p...

> What do we need to do to prevent people like you from writing an article like this again?

I thought it was a great read, and totally didn't take the "Haskell programmers are liars" literally ...

... until I saw #4 on your list and that this example was actively discussed on the mailing list and ultimately determined to satisfy it.

Haskell is really nice. Keep looking for a better example. :)

yeah, totally. there's now a method `embalm(&mut self)` run just before the binary data gets written. That and `exhume` would be great places to go to and from little-endian.

Would you find it equally worrying if stack-based languages claimed they improved over x86 because you needn't worry about clobbering values in registers? I guess it depends on how well you know the code and its context. Might still be logic bugs.

And man, functional languages. It's pretty much 100% logic bugs there. I worry most about those sort of programs.

Well, I'm looking at the code and it looks like it doesn't report false positives. I understand that you expect it to hash the content down and store only the hash, but it doesn't do that.

EDIT: I suppose my point was just that there is a difference between "I think you chose a bad name for your class" and "This is actually not true and not possible". It seems both true and possible (and arguably inappropriately named).

Yes, you can also write safe code in an unsafe language, but until you are literally willing to code review every program anyone writes in your unsafe language of choice, there is value in having safe languages where we don't have to rely on your throughput and accuracy.

There is a real difference between a safe language, and a safe subset of an unsafe language that gets verified by some guy (no disrespect). The former is (everything else being equal) just way better for everyone involved, especially for the guy.

This probably isn't crystal clear from this post alone, but for PageRank the 42GB working set has a good mix of read and write. Specifically, about 14GB of target rank vector gets +='d all over the place. If they got hammered randomly, then lots of paging and writing would happen.

That's a good point, thanks.

I've added the timings from the original post to the tables in this post, which also makes clearer the intended point: the existing systems don't yet have measurements for data at this scale (except Facebook's modified Giraph, which I want to hear more about :))

I was formerly in the same lab as Leslie. As a side project, we (Tom Rodeheffer, mainly) used TLA to verify the correctness of Naiad's coordination protocol. The protocol is very simple (distributed reference counting), and my initial "proof sketch" was about two paragraphs long (and very unconvincing, apparently :). Tom's TLA+ version was closing in on 200 pages, available at

http://research.microsoft.com/pubs/183826/paper.pdf

A short publication about it, with some reflective discussion of why it was hard to write down, is at

http://research.microsoft.com/pubs/199767/clock-verif2.pdf

My understanding is that part of the problem was that Tom needed to explain (re-prove) many meta-facts we take for granted.

The thrust of the post wasn’t meant to be that it is surprising that you can even do these things on one core; sorry if it came across that way.

Rather, it’s that when doing the work on one core (with a way simple implementation) you go faster than many of the measurements that popular scalable systems put forward as evidence that they improve on the state of the art. Which sort of invalidates that evidence. Some of the papers end up left with fairly scant evidence, which should be a serious issue for researchers.

GraphChi and X-Stream (http://infoscience.epfl.ch/record/188535) are much more sophisticated implementations, ones you might not be embarrassed to be beaten by, so they don’t make the same point. And they aren’t any faster than the systems above, and so have the same issues (though perhaps less explaining to do).

With all due respect, your code seems to be a copy/paste of my code, where you've removed the Pair and Option implementation (allowing the alternate default I discussed in my second blog post) and replaced my string `ColumnarVec` with your string `SimpleSerialize`. So, thank you for measuring my code for me? :D

The reason the new numbers look better for Option with the Copy implementation is that you are now writing 16 bytes for each Option. They are a 50-50 mix of Some(0u) and None, which I wrote out in 5 bytes on average (always a 1 byte "present/not", and then 4 bytes of data on average). The Copy implementation is just writing 3x as much data and padding the throughput numbers, rather than reporting something more like goodput.

I sense that this isn't the right place to shake this out, so I'll stop posting. If there is a better way to follow up, let me know.

[edit: I've got you through github, thanks! I'll buy beers when I swing by TU next]

I totally left the #[inline(always)] out from the code, in the interest of explaining. That's what prevents the copying! :D

And also what Steve said about Rust being rly rly smart, or at least move semantics opening the door for that.

Hi, author here.

Unless I misunderstand your post, I think you may have missed important parts!

More is going on than "just wrapping the data of the original vector as a Vec<u8>". That is what happens when we are handed a Vec<uint>, but for other inner types (pairs, vectors, etc) there is more to do (and, actual code presented showing what that work is).

1. A Vec<(u8, u64)> definitely ends up as a (Vec<u8>, Vec<u64>), thereby avoiding padding you'd have if you just wrote out the elements as structs. You absolutely end up saving space, and part of doing this is definitely copying data. (responding to "as it does not actually copy any data").

2. The types themselves can be vectors, corresponding to a struct with owned pointers (whose elements can also own pointers, etc). This is not something that just casting the source array will deal with. It's important here that Rust hands you ownership, as otherwise it would be totally inappropriate for us to claim the underlying memory (which the code does). That part, recycling owned memory, is one of the big performance wins (about 2.5x faster for me than invoking the allocator each time I need to mint a new array).

3. "Copy" doesn't appear in the first post. The types don't have to be Copy, and indeed Vec<T> is not Copy, even if T is. Not sure where you got that from. But, no such requirement; yay!

Read part 2 for more about Copy, and how when your type is Copy you get a free implementation that does what you suggest (keeping each struct intact), except you can mix and match with Vecs and Options and stuff like that.

Hope this clears up some of the not-sense-making. Feel free to holler with other questions.

Cheers, Frank