Hacker News Reader

Comment by ateesdalejr | Hacker News Reader

Wow! Multi-terabyte heaps? That sounds like way to much stuff in memory. When would you need multiple terabytes in RAM? I mean come on... What kind of application even needs that. Though I do admit that javascript needs this kind of GC implementation because the hangs JS causes are a real pain.

View on HN

Multi terabyte heaps would move the bar for Big Data way to the right and let most of us just ignore the whole thing.

Basic space versus time trade off. In eight years we’ll laugh about how stupid people were for doing it, and in another 8 laughing at the people who didn’t.

One of the reasons things like memcache became a thing is because we hit a point where same-switch tcp/ip round trips were smaller than average seek time for your hard drives. I gotta say, it’s a little crazy that it was faster to ask another machine a question than to look it up yourself, but there it is.

Run Apache Spark on the box with 256+ GB of RAM. The GC stalls by 60 or more seconds are quite typical for heavy queries. I hope new GC will bring improvements.

For Spark/etc I only care about throughput. It can stall all it wants. Exception (possibly) being spark streaming.

Analytics jobs, some database use cases, machine learning, simulation... plenty of things could make use of huge heaps.

And TB memory is merely enough for lots these use case. We can now skip a lot of complicated scaling issues with merely a few boxes.

The problem is memory price / GB not only hasn't gone down but goes up.

Reply on news.ycombinator.com