Hacker News Reader

Comment by ConstantineXVI | Hacker News Reader

I'm new to both (and noSQL in general), first heard about CouchDB a week or two ago. Would Couch be a good fit for building a logging system, or is there something else I should look into?

View on HN

I wouldn't use it for a write-heavy logging system ( and I am a big fan of it).

Well actually it depends on what type of logging. If you plan on pouring megabytes of data/sec second into it continuously you might want to find something else. I don't think CouchDB would be a good candidate for that. It might be if you plan on archiving your post-processed logged data for a long term storage but you have to be smart about its format.

Yeah 'logging' can mean a lot of things that it is hard to give a better advice without more information.

Other DBs/tools you might want to look at:

* rrdtool http://oss.oetiker.ch/rrdtool/

* MongoDB http://www.mongodb.org/

* Redis http://redis.io/

* Custom format + plain text files

So, both MongoDB and Redis are DB's optimized for keeping stuff heavily in memory. I wouldn't use either for heavy logging.

RRDTool is nice, but only for data where you don't mind losing records.

For straight logging, SQL is actually a better solution than a lot of those options. If you need weird queries, something like Elastic Search or Solr on top of your SQL DB can buy you quite a bit of freedom. As can denormalizing your SQL.

One other thing to consider of course, is something like syslog.

HBase might be a good solution, too, but I don't know a whole lot about it. CouchDB would work fine but you'd want to add some operational automation and a proxy to roll over between temporary databases since deleting old logs from a single database won't fully reduce its size and, as stated above, compaction isn't terribly fun or fast but you can avoid it by just flat out deleting old log databases.

I'm not an expert on CouchDB (and I would love someone to correct where my approach is wrong).

I could see having a CouchDB cluster per co-location, recording a document per logging event. Then back haul through replication to a central service for analysis/dashboard use.

The only tricky part I see is to delete old logging data requires a compaction period and according to http://wiki.apache.org/couchdb/Compaction it's entirely possible for the compaction process to be slower then write data and never actually finish.

Edit: Also the back end server doing analysis may need to be a BigCouch cluster to handle the load (depending on size) https://github.com/cloudant/bigcouch

Absolutely it would be great for logging. Assuming that you don't need atomicity for your logs (eg, I want to see that log line reflect in my reports _right_ _now_) Couch is a perfect way to dump a huge amount of data really quickly. For reporting, the map/reduce system of Couch is a great fit. There's an initial learning curve -- as with anything, really -- but you'll find that Couch would be a great way to get logged data (which can be pretty unstructured) and report on it easily and quickly.

no, it wouldn't. It would be a horrible fit, imo. You don't need versioning info for logging, and the extra compacting of the database is another aspect, since with logging heavy applications you will write all the time...

I love CouchDB but i definitely wouldn't use it for logging.

You don't need to worry about the versioning for a document that you never update. You can just blast the logfiles into the database and not worry about conflicts. Similarly, database compaction is really the biggest concern when you're updating documents, not just dumping new ones into the database. I would expect a very linear growth for a logging app.

ok, my initial thought was one document per logfile, containing an increasing amount of loglines, but that's probably the worst approach possible. One document per logline which is appended to the database should be much better.

The problem doesn't come with the inserts, it comes with the deletes, where CouchDB will keep some metadata and stubs around to support its replication use cases. In a pure logging setup this is perhaps less than ideal.

atomicity doesn't mean, or imply, instantaneous.

However, a document update in CouchDB is fully ACID (including the A of Atomicity). People often confuse transactions around multiple changes with ACID for reasons that I cannot fathom. CouchDB does not have support the former (they don't scale well in clusters) but absolutely supports the latter.

Couch's map-reduce is quite slow for ad-hoc stuff. It's excellent for frequently viewed data (as you can get stale views, and new views are sort of incrementally done), but it's slow to set up.

No. CouchDB is good for storing important data, slow at writing, and bad at ad-hoc querying.

MongoDB is lightning fast at writing (provided you haven't got indexes, or your indexes fit in RAM), good at ad-hoc stuff, and a little more durable than /dev/null. Just remember to back up occasionally.

Cassandra is a good fit for write heavy workload, as it is optimized for write...

Reply on news.ycombinator.com