Hacker News Reader

Comment by jtchang | Hacker News Reader

Does it blow anyone else's mind that database servers are so powerful nowadays that you can just load half a billion rows into a SINGLE TABLE and pretty much expect things to work? Obviously there are caveats but all in all a nice chunk of RAM backed by a large SSD goes a long way.

View on HN

As a long time skeptic of the NoSQL craze that seems to have (happily!) peaked, I have to say that no, I'm not very surprised by this.

Databases (real databases) are using indexing technology that's been proven over not just years but decades of use and refinement. log2(500000000) is still only 28. Double that and it's only 29. And these databases' indexes are typically btrees with much higher branching factors than 2.

One of my favorite articles on this is Dennis Forbes' "The Impact of SSDs on Database Performance and the Performance Paradox of Data Explodification" (http://www.yafla.com/dforbes/The_Impact_of_SSDs_on_Database_...).

I'm more impressed with how cheap it is.

An SSD is on par with a 15K disk, but delivers 10X the perf, throw 10 SSDs in an array and you've got what used to cost $100,000 for about $5K

But SSDs that cost the same as a 15K disk do not have anywhere near the life time of the spinning disk due to write endurance limits.

So load half a billion rows but don't update it too often!

SSDs live long enough. You get magnitudes more IOPS performance out of a single SSD than a RAID of 15k disks, it consumes magnitudes less power, it costs less, and it needs less maintenance. It's a complete no-brainer these days to put an SSD in your database server.

Given the cost of power and space and vs. the number of additional drives necessary I'm not sure that longevity is worth it.

I was impressed by stuff like this in the 90s ;-)

What's impressive now is not the software but that the hardware is so cheap.

My telemetry table is >2B rows and growing. 5B is starting to worry me; .5B, not so much. This is even backed by ( not much ) spinning rust.

Hopefully you don't have an auto-increment int (2^31) as a PK, or you'll be worrying sooner than you think.

That's one reason why switching auto-increments to bigint does help (a lot):

http://dev.mysql.com/doc/refman/5.5/en/integer-types.html

That is interesting to know, I have been setting my index's to bigint for a long time.

Ugh. No. This uses a multi-column natural key. I have a tendency to avoid synthetic keys when I can.

Reply on news.ycombinator.com