Hacker News Reader

Comment by tetha | Hacker News Reader

Yup, experiencing that at work quite a bit. People are currently arguing about big data solutions for a data set of 300Gb or so, growth in the area of 100ish Gb / year. I'm still arguing that this could be an in-memory dataset for postgres, and we wouldn't need that with a good schema.

Or other people are wondering if we could replace all of our relational databases with kafka. While complaining about inconsistent data sets. Well let's talk about advantages of a good relational schema first.

Maybe I'm turning into a grumpy old admin/DBA... but mariadb or postgres used well just solve so many problems.

View on HN

>> Maybe I'm turning into a grumpy old admin/DBA

No you're not; you're saving the company tens or hundreds of thousands of dollars over the coming years.

> Or other people are wondering if we could replace all of our relational databases with kafka.

Unless you're using your DB as a data transport, dear God why. And if you are, dear God why. I work with Kafka and it's very good for a certain subset of problems. Replacing RDBMSes is not one of them.

I once asked if Postgres would have any negative performance impact from holding 1.8m records and once the laughing stopped and they realized I was serious I got a good lesson on how robust Postgres can be.

I once worked for a company that maxed out a 32 bit autoincrement primary key on a mysql table. Relational databases can hold a lot more than people give them credit for.

Yep. I worked at a company that had to migrate to 64-bit IDs. This was with MySQL 5.0 on servers with HDs, not SSD. Some of those "alter table" commands took all night!

Oh man, I hope they saw that one coming ahead of time.

A colleague describes this problem as "running out of numbers".

I managed complex datasets of a few TB with standard Unix tools just fine. If you are lucky, you have a sustained read and processing speed of 200M/s (or more with, if you are put in some work), which allows you to sift through a TB in a good hour.

Sometimes I am wondering, how people will rationalize the layer bloat in 5-10 years, when hardware went through a few more upgrades.

Reply on news.ycombinator.com