Hacker News Reader

Comment by tharkun__ | Hacker News Reader

Actually they're even older.

When I talked to my dad about RDBMS he was like "weeeelll, sure, there's things like DB2 UDB that do relational but performance wise, nothing beats reading the data straight by key in exactly the format you need.".

DB2 UDB: "Initial release: 1987; 34 years ago". I.e. what he would rather use and is sort of the NoSQL equivalent is _even older than that_.

Background: he started off with 360 assembler and worked all his working life on IBM Mainframes and the various technologies in and around it. They had it all and they had it before it came to "us". We're just re-inventing most of these things on much cheaper and more open hardware and software.

View on HN

what I fail to fathom today is, when I think back to the late 90s, I had various production rdms, SQL Server 6 (or 7, can't remember), running on something like a Pentium 4, with millions of rows of data in various tables... and the performance was... fine. not an issue.

Single machine postgres is mostly fine when you're working with less than hundreds of million of rows, less than a few hundred gigs of data, don't need HA, and don't need more than a few thousand QPS. Outside of those constraints is why new things exist imo.

Of course, the overwhelming majority of all businesses (including web-centric businesses) work fine within those constraints.

Sure, small businesses are fine using postgres. I would argue that by developer market share though most (or perhaps just many) developers have to worry about these issues, since the companies that have scale have to higher many more developers.

There's also the argument that businesses hoping to scale might be better off assuming they will need it to save themselves work down the road.

Computers are very fast and fast storage is very cheap these days.

Your Data Isn't That Big https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html

> Your Data Isn't That Big

This is an extremely important lesson. You don't have Big Data, unless you're one of a handful of huge companies.

Years ago I joined a mid-size public company and inherited one particular service. Management was adamant that it had to migrate away from MySQL onto Cassandra due to data volume. But there were no metrics being tracked, it was driven by "we have to do it because everyone is doing it".

I started tracking actual usage, growth and performance and it didn't seem to point to a problem that needed solving. So I halted that project (to their credit, they listened to engineering input so I was able to).

A few years later when I left, it was still perfectly fine on MySQL. Nowhere close to running out of capacity. Today it's been almost ten years and while I'm not there, what I hear is that it's still fine.

"But my data is more than 5TB! Your life now sucks - you are stuck with Hadoop. You don't have many other choices (big servers with many hard drives might still be in play), and most of your other choices are considerably more expensive."

My point is that large companies that do need to scale hire many developers. FAANG likely hire north of several hundred thousand engineers alone, and according to this article[1] there are only something like 1.5 million software engineers in the US.

[1]: https://www.daxx.com/blog/development-trends/number-software...

Even combined, FAANG is probably only around 150k programmers. A vast majority of the worlds 24-26M working software programmers do not work at them, or even at the top 100 such companies.

PostgreSQL handles 100k QPS without any issues unless your queries are very complex and I know many people who run multi-TB databases in PostgreSQL without any issues either. But it all depends on your workload of course.

Yeah, I probably could have bumped both those numbers an order of magnitude.

Today you need a Cassandra cluster fronted by a Kubernetes cluster to get similar performance. The good news is that you don't need a couple boxes under your desk, you can pay AWS tens of thousands a month for it.

It is not and issue today either. Modern RDBMs have become better optimized and hardware has become faster so a lot of NoSQL is premature optimization where people optimize for having 100+ TB databases when their database actually is 100 GB.

Well, millions of rows of data is tiny these days: if you have 100_000 IOT devices, each generate on average 10 telemetry points a second, that's already 1M writes per second. Storing these telemetry for a week and you already have 600 billions rows.

Of course that's a oversimplification, but the data generated today is mind-boggling large compared to the 90s.

> but the data generated today is mind-boggling large compared to the 90s.

For what percentage of companies would you say your statement is true?

If you have 1M writes per second, you have a data collection problem, no matter the technology. Doing it on a single node is not a great idea just because it'd be a huge SPOF.

In most cases, you have colossally fewer than 1M writes per second, and you usually distribute them over several boxes. the IoT avalanche of writes in particular is naturally sharded, that is, all these writes can occur independently, and then you can replicate statistics into an analytical RDBMS.

It's a large write load that needs to be transactionally ordered, like a stream of purchases in a large online store, that can be a hard(er) problem for an RDBMS.

Yup, I agree with you. Most businesses can work fine with traditional RDBMS, even on TB scale if you have a large box with lots of RAM and SSDs.

My example was just one use-case where one should consider using a more unconventional DBs like Cassandra or HBase.

:s/these days/at your particular job/g

The original Unix DBM file was basically NoSQL in a SQLite type package and it was developed in 1979.

https://en.wikipedia.org/wiki/DBM_(computing)

I worked on DB2 UDB in the 90s, and learnt a lot about DBs through all the improvements in isolation levels. Those were the days when you'd actually have to make some judgement calls about setting specific isolation levels to improve performance. Nowadays you just chuck an SSD and several TBs of memory into the works (or talk to someone about some massively horizontally scaled solution, and distract from the real problem at hand).

Oracle introduced RDBMS in 1979 based off Codd's paper in 1969. Mainframes were all flat file not object-relational, but you could finagle them to be sort of that. Object-Relational didn't come into play until the 90s.

The main issue with object relational as the "main" database is its hard to run reports on. Reports (for biz) is half the reason the system exists. I'm not saying ORDBMS don't have a place, but RDBMS is much better suited for real-time data analysis / reports. I think ORDBMS is popular because it's easy for non-database people (who don't know 3rd normal form or SQL) to store stuff in.

Let's not erase the work of Stonebraker et al whose efforts through the 70s were a lot closer to Codd than Oracle's. Ingres's query language, Quel, springs to mind: being directly based on the relational calculus it was far more readily composable than SQL (which might be charitably described as "borrowing ideas from the relational model") was or is.

You may be overlooking System R from 1974:

https://en.wikipedia.org/wiki/IBM_System_R

Well, even plain old filesystems are some weird hybrid between key-value stores and graph data bases, too.

it's sometimes very helpful to think of filesystems as a streamable blob store with a hierarchical key

That's one valid perspective. But don't forget that you can also seek and mmap etc. And hard-links are a thing, too.

Things like fuse also show that it's easy to view other databases as filesystems, too. As a silly example, you can 'explode' /etc/passwd into individual files per entry. Or you could expose even a relational database as a filesystem.

For the latter, you could do something simple and expose tables as directories and rows as files. Or you could do something dynamic, and eg represent (read-only or idempotent) queries as file-names, and their results as file contents.

That wouldn't be so much a traditional filesystem, but would be accessible via a filesystem interface.

I hear AWS DynamoDB is basically just a "key value store interface" to S3 file storage.

Reply on news.ycombinator.com