Comment by msiebuhr | Hacker News Reader

Comment by msiebuhr | Hacker News Reader

Parent Full thread

$WORK anecdote:

We had a quartet of legacy-ish databases with sharded customer data (had some interconnection, but we sharded in the application-layer, so the database didn't have to move any data internally). It was a bitch to maintain, but it performed OK, by throwing fairly beefy machines at it.

We began a quest to replace the database with MongoDB/HDFS/Riak/CouchBase/Cassandra/whatever. We wrote a script that converted our data to JSON for easier ingestion and began hammering away. Until we discovered that by converting it to JSON (and simplifying some parts of the data-model), our dataset had actually become small enough to fit on a single server in memory (still ~100GB).

So we ditched the cluster-thingie and moved everything to CouchDB. Pricy hardware, but the saved man-hours recouped that within a month or two.

(And in half a year's time, the database will have outgrown whatever hardware we can reasonably throw at it, so then we'll have to start over for real...)

That's odd. When I last checked, CouchDB took a lot more memory. Unless you are doing something very odd with how you model your data ("select (key, value) from attributes where recored_id = ?" ... why are you even using SQL then?).

I expect that slimming down the data model was the main factor, or you simply didn't want to use SQL at all (or your data wasn't really structured enough).

But why throw it in couchdb when you could use a rdbms ?

It must be web scale.

We don't need a RDBMS; we get along quite well with key/value + keyscans, making it a lot easier to switch to a database with roughly similar features later (CouchBase, Cassandra, HBase, possibly others)

OK, slightly ignorant question here, but SQL is kind of a standard for RDBMS's. As far as I am aware no similar starts exists in the NoSQL world, so I would have assumed that it would make switching harder. If you are just using it for basic stuff, then "SELECT * FROM table" is pretty standard across RDBMS's.

Reading this: (And in half a year's time, the database will have outgrown whatever hardware we can reasonably throw at it, so then we'll have to start over for real...)

You need a rdbms. Field names won't be repeated for each query and you will push later the time to 'start over'.

Reply on news.ycombinator.com