Hacker News Reader

Comment by voidlogic | Hacker News Reader

This varies by the amount of RAM, if the total RAM being talked about is <= 256 GB (or in some cases 128) this statement will usually be false. But for RAM totals above 256 GB (and in some cases 512) it is almost always true. But these watermarks will change over time so this assumption must but re-evaluated from time to time.

View on HN

$WORK anecdote:

We had a quartet of legacy-ish databases with sharded customer data (had some interconnection, but we sharded in the application-layer, so the database didn't have to move any data internally). It was a bitch to maintain, but it performed OK, by throwing fairly beefy machines at it.

We began a quest to replace the database with MongoDB/HDFS/Riak/CouchBase/Cassandra/whatever. We wrote a script that converted our data to JSON for easier ingestion and began hammering away. Until we discovered that by converting it to JSON (and simplifying some parts of the data-model), our dataset had actually become small enough to fit on a single server in memory (still ~100GB).

So we ditched the cluster-thingie and moved everything to CouchDB. Pricy hardware, but the saved man-hours recouped that within a month or two.

(And in half a year's time, the database will have outgrown whatever hardware we can reasonably throw at it, so then we'll have to start over for real...)

But why throw it in couchdb when you could use a rdbms ?

It must be web scale.

We don't need a RDBMS; we get along quite well with key/value + keyscans, making it a lot easier to switch to a database with roughly similar features later (CouchBase, Cassandra, HBase, possibly others)

OK, slightly ignorant question here, but SQL is kind of a standard for RDBMS's. As far as I am aware no similar starts exists in the NoSQL world, so I would have assumed that it would make switching harder. If you are just using it for basic stuff, then "SELECT * FROM table" is pretty standard across RDBMS's.

Reading this: (And in half a year's time, the database will have outgrown whatever hardware we can reasonably throw at it, so then we'll have to start over for real...)

You need a rdbms. Field names won't be repeated for each query and you will push later the time to 'start over'.

That's odd. When I last checked, CouchDB took a lot more memory. Unless you are doing something very odd with how you model your data ("select (key, value) from attributes where recored_id = ?" ... why are you even using SQL then?).

I expect that slimming down the data model was the main factor, or you simply didn't want to use SQL at all (or your data wasn't really structured enough).

Reply on news.ycombinator.com