Hacker News Reader

Comment by dd444fgdfg | Hacker News Reader

what I fail to fathom today is, when I think back to the late 90s, I had various production rdms, SQL Server 6 (or 7, can't remember), running on something like a Pentium 4, with millions of rows of data in various tables... and the performance was... fine. not an issue.

View on HN

Today you need a Cassandra cluster fronted by a Kubernetes cluster to get similar performance. The good news is that you don't need a couple boxes under your desk, you can pay AWS tens of thousands a month for it.

Single machine postgres is mostly fine when you're working with less than hundreds of million of rows, less than a few hundred gigs of data, don't need HA, and don't need more than a few thousand QPS. Outside of those constraints is why new things exist imo.

Of course, the overwhelming majority of all businesses (including web-centric businesses) work fine within those constraints.

Sure, small businesses are fine using postgres. I would argue that by developer market share though most (or perhaps just many) developers have to worry about these issues, since the companies that have scale have to higher many more developers.

There's also the argument that businesses hoping to scale might be better off assuming they will need it to save themselves work down the road.

Computers are very fast and fast storage is very cheap these days.

Your Data Isn't That Big https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

> Your Data Isn't That Big

This is an extremely important lesson. You don't have Big Data, unless you're one of a handful of huge companies.

Years ago I joined a mid-size public company and inherited one particular service. Management was adamant that it had to migrate away from MySQL onto Cassandra due to data volume. But there were no metrics being tracked, it was driven by "we have to do it because everyone is doing it".

I started tracking actual usage, growth and performance and it didn't seem to point to a problem that needed solving. So I halted that project (to their credit, they listened to engineering input so I was able to).

A few years later when I left, it was still perfectly fine on MySQL. Nowhere close to running out of capacity. Today it's been almost ten years and while I'm not there, what I hear is that it's still fine.

"But my data is more than 5TB! Your life now sucks - you are stuck with Hadoop. You don't have many other choices (big servers with many hard drives might still be in play), and most of your other choices are considerably more expensive."

My point is that large companies that do need to scale hire many developers. FAANG likely hire north of several hundred thousand engineers alone, and according to this article[1] there are only something like 1.5 million software engineers in the US.

[1]: https://www.daxx.com/blog/development-trends/number-software...

Even combined, FAANG is probably only around 150k programmers. A vast majority of the worlds 24-26M working software programmers do not work at them, or even at the top 100 such companies.

PostgreSQL handles 100k QPS without any issues unless your queries are very complex and I know many people who run multi-TB databases in PostgreSQL without any issues either. But it all depends on your workload of course.

Yeah, I probably could have bumped both those numbers an order of magnitude.

Well, millions of rows of data is tiny these days: if you have 100_000 IOT devices, each generate on average 10 telemetry points a second, that's already 1M writes per second. Storing these telemetry for a week and you already have 600 billions rows.

Of course that's a oversimplification, but the data generated today is mind-boggling large compared to the 90s.

> but the data generated today is mind-boggling large compared to the 90s.

For what percentage of companies would you say your statement is true?

If you have 1M writes per second, you have a data collection problem, no matter the technology. Doing it on a single node is not a great idea just because it'd be a huge SPOF.

In most cases, you have colossally fewer than 1M writes per second, and you usually distribute them over several boxes. the IoT avalanche of writes in particular is naturally sharded, that is, all these writes can occur independently, and then you can replicate statistics into an analytical RDBMS.

It's a large write load that needs to be transactionally ordered, like a stream of purchases in a large online store, that can be a hard(er) problem for an RDBMS.

Yup, I agree with you. Most businesses can work fine with traditional RDBMS, even on TB scale if you have a large box with lots of RAM and SSDs.

My example was just one use-case where one should consider using a more unconventional DBs like Cassandra or HBase.

:s/these days/at your particular job/g

It is not and issue today either. Modern RDBMs have become better optimized and hardware has become faster so a lot of NoSQL is premature optimization where people optimize for having 100+ TB databases when their database actually is 100 GB.

Reply on news.ycombinator.com