Comment by chockchocschoir | Hacker News Reader

Comment by chockchocschoir | Hacker News Reader

Parent Full thread

chockchocschoir

> Where the large scale storage clusters shine is when the sheer scale of data won't fit in anything else, i.e. there's no other (sane) choice.

And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be!

On that note, there is this great website for seeing if there are servers that can handle fitting all your data in RAM: https://yourdatafitsinram.net/

Systems like Power System E980 can handle up to 64TB RAM, which is a lot of data. Just like parent said here, do try to fit things on machines like this before even trying out large-scale storage clusters, because they are a hassle to deal with and generally not worth the cost.

Edit: For the Cloud-hosting crowd out there, I took a quick look at https://instances.vantage.sh/ and sorted it by memory. Largest instances you can get at AWS is "U-24TB1 Metal", which comes with 24TiB of RAM (and 448 vCPU so the computation itself gets as fast as possible too [granted you can parallelize it]), which should fit most cases of "big data" I've seen in the wild today. Unsure about running costs though, but I'm 90% sure it's cheaper to have that instance for a couple of hours than the cost for having to deal with storage clusters, which are also slower.

>> And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be!

That's why the "big data" industry also encourages collection of absolutely every bit of data you can find. They want you to need their tools. You may not think there's a use for it, but vague promises of AI finding needles in the haystack are used to get you to keep it and bloat your system.

This is why the first, possibly most important phase of a data analytics job is to filter down to the subset of the data that you actually need to compute the result.

Collecting it all on the front end can be very effective, as long as you can easily filter. If you can't, you are just causing yourself more problems.

> Systems like Power System E980 can handle up to 64TB RAM, which is a lot of data.

Doesn't that kind of system cost millions of dollars?

Hadoop and the like are popular because they run on COTS hardware that costs peanuts. It hardly makes any sense to argue that spending over a million dollars is enough to drop a solution that's primarily adopted because you do not have that kind of money to throw at a problem.

You could also use SSI (single system image) system architectures, including distributed shared memory, to seamlessly (at least wrt. software implementation) scale up from efficient single-node processing to an arbitrarily large, clustered system. AIUI this is how things are traditionally done in mainframe processing, and it is also regarded as a very meaningful possibility in HPC.

That concept has fallen out of favour in HPC, but it'll be interesting to see if it makes a comeback.

As recently as 2016, an SGI UV3000 rack-scale SMP machine with 16TB of RAM was the sort of thing you'd see on a trade-show floor - now you can get that much memory in a 4U chassis, and things will only improve further if/when Optane DIMMs take off.

I can share with you that my solar-powered Pi3B+ hosting yourdatafitsinram is holding up quite well, being linked to on HN - a funny contrast to the kind of systems it links to in the table.

Even a 10+ year old DL380G8 - can hold 1.5TB+ RAM and 24/48 cores and that hardware is dirt cheap on the second hand market.

> I can share with you that my solar-powered Pi3B+ hosting yourdatafitsinram is holding up quite well, being linked to on HN - a funny contrast to the kind of systems it links to in the table.

If all a server does is serve mostly static content from memory then it's not expected to handle a demanding workload beyond networking.

Therefore I fail to see the point of trying to downplay someone else's needs for TB of RAM just because you serve a site on a raspberry pi, as if that's the full extent of what anyone needs to do with a computer, particularly when the OP mentions big data.

You totally misinterpreted the point of my text.

Reply on news.ycombinator.com