Hacker News Reader

Comment by gambler | Hacker News Reader

>In-memory databases are not economically scalable.

SAP HANA is widely used for very large data sets.

How are you defining "very large"? I've used HANA and even SAP doesn't make claim of real scalability -- the practical limitations are in their own documentation. And in real scalability testing, it struggles long before you reach those limits. For sensor data, I wouldn't use it for more than a few terabytes, and even then it has a few sharp edges when it comes to performance if you are not careful. No one is going to be putting petabytes into it, which isn't that much operational data these days.

And to my point you quoted, SAP HANA is extremely expensive to operate compared to alternatives. The licensing costs alone will kill you, never mind the hardware requirements.

This checks out. A single 12 TiB RAM u-12tb1.metal on AWS EC2 goes for $30.539/hr. For a month for one (24 * 30), that's a bill of $21,988.08 USD.

A single i3.16xlarge, with 15.2TB NVMe SSD goes for $4.992/hr, or $3,594.24 USD a month. 16.33% of the price.

I'm not going to argue whether an in-memory database (measured in ns) wouldn't give you performance improvements over fetching data from an SSD (measured in ms). But not everyone needs that speed or can afford it for that price.

Sources: https://aws.amazon.com/blogs/aws/now-available-amazon-ec2-hi...

https://aws.amazon.com/ec2/instance-types/i3/

https://aws.amazon.com/ec2/pricing/on-demand/

At a high-level, it seems expected that since the outside world (around the DB) operates on the same tiered storage (cache > RAM > disk > backup(>n)), economically it doesn't make sense to replace any tier by its direct superior, because the bottleneck becomes the outside world (any machine/infra that's not able to keep up anyway).

Not with storage but I/O, that's what the makers of the Cell processor (PlayStation 3) had in mind originally (it was culled before the final design though): huge IO, non-stop feeding the beast. The arch-goal was to be able to link Cells together to make a "network of CPUs" (network becomes sockets link) able to parallel executions. A hard (wild?) computer science dream/problem. (the final Cell CPU has none of that iirc). I think NVLink is a decent comparison of that purpose/design.

Now running a whole infra on RAM, not just a DB, that makes more sense on paper, and I guess that's what devs do daily with e.g. test environments, like a bunch of containers over a ton of RAM. At a small enough scale, upgrading storage tier makes sense as cost becomes negligible.

For a business / at scale, short of extremely specific applications where you'd indeed have not just a DB but whatever calls it also on RAM --- and remembering that this is not economical to serve "more" users or "faster", since you'd scale horizontally for that --- the use-case or endgame of whatever this DB serves should have that speed as a hard requirement. Likely to be 'one' monster itself, like, a supercomputer? Assembling deep learning datasets from real-time feeds on-the-fly? Skynet? Big brother? :) Jokes aside, one needs a beast to feed that'll take no less to justify a 7x more expensive RAM-based anything at scale.

Normal folks, I think we'll do with caching the hell out of our data for cheap, for now at least. Until RAM becomes abundant and CPU/storage/IO extremely expensive by comparison. (was RAM ever abundant? I can't seem to remember a time when I could just buy without counting, unlike storage or FLOPS relatively to everything else).

I missed the part about "petabytes a day". I'm curious, which companies stores petabytes of data a day in a single database?

Facebook was hitting 4 petabytes a day of user data back in 2014. Back then, there were only a handful of giants needing that. Today, multiple systems are now being scaled to handle that sort of data volume. Think automotive fleet systems and other IIoT applications. Mobile networks (and security systems for them) managing tens of millions of devices.

https://research.fb.com/facebook-s-top-open-data-problems/

Generally these tend to be systems that work with machine generated data, my experience is with sensor data generated by automobiles (automated car efforts).

Naive solutions tend to either summarize the data, store as logs and then run batch processes to index in some form (or leave unindexed and just brute force the computation), or limit the incoming data rate to whatever could be indexed.

These can work for some use cases, but make it very difficult to operationalize these data sources (i.e use them to make real-timeish decisions).

Even human generated data sources (fb / twitter etc.) can generate something close to that data rate.

Reply on news.ycombinator.com