Hacker News Reader

I like the direction Spark is heading. I am happy to see that they look at Spark in the same way compiler developers look at programming languages. There are huge optimizations to be made in this space. It's insane how inefficient our current systems are when it is related to big-data processing.

View on HN

I think, as experiments published in last year based on Spark, the inefficiency stems from the assumption that CPU cycles are abundant w.r.t. RAM and bandwidth. So, nobody focuses on optimizing program itself as much as to reduce memory or bandwidth usage.

Seeing the talk today, I think the Spark team is very aware that CPU cycles haven't become more abundant, while I/O has. There was an entire slide to this effect. I don't know enough about the space to say more than that, but it seems it's definitely on their mind. One of the things I remember being mentioned is operating on data in a much more efficient binary representation than native Java memory model, allowing better use of cache and such.

That's not an assumption, it's a fact.

Reply on news.ycombinator.com