Logging the cloud with SimpleDB | Hacker News Reader

Logging the cloud with SimpleDB

peecho.com

32 pointspveugen14 comments

SimpleDB is good for a lot of things, but the unpredictable costs (since you're charged for box usage which varies) and network latency make it unsuitable for logging at scale. (like, what if you get 100's of error messages a second?)

We use a decidedly low-tech scheme. We log everything to files on local storage. Then we use a command line tool to periodically sync these to S3. Each instance gets its own folder.

When we want to query, we have several options. We can run good old grep or tail on one instance -- or on several instances with pssh. We can use the S3 command line tool to process multiple files on an admin machine. Or we could use Hadoop to mess with files in S3 (haven't done that yet...!)

Not as snazzy as having each log record stored in a structured fashion, but it's a lot cheaper than SimpleDB.

I put together a similar logging setup for our servers recently; except I'm using CouchDB for my store and NLog (said servers are C#) for the library. Reasons I'm using Couch instead of SimpleDB:

1) REST+JSON API made integrating it super-simple (had to add a single line to NLog to set MIME types to make Couch happy)

2) We use SDB in our service; so if SDB blows up, not only would things break, we couldn't easily get at our logs to see why.

Biggest drawback I've found to Couch in this context is that it loves to devour disk space; you quickly learn to cron some compaction jobs.

Why not use something like rsyslog and have it transfer the logs via udp to a centralized rsyslog server? Very easy to setup.

The first motivation for this setup was to make our logs easily parseable to extract stats from them; Couch seemed like a natural fit to store them. Plus, Couch's sane API essentially means we're able to grab the logs from any tool/language with minimal effort. I'm well aware that this isn't necessarily the best/proper way to handle logs, but we're all fairly pleased here with how it's worked out.

Thanks for your addition. At Peecho, we figured that if SimpleDB goes down, logging would be the least of our troubles. Read here how we tried to maintain our uptime: http://www.peecho.com/blog/minimizing-downtime-on-amazon-aws...

-------- Please vote for Peecho in the race for the Accenture Innovation Awards: http://goo.gl/3xLlC

Was just going to post this link -- you guys put a lot of time into your AWS posts and they are excellent. You have a really nice/robust AWS deployment there.

Thanks!

Logging seems like the perfect candidate for a relational database. It's a write-heavy system with need for powerful ad-hoc query capabilities. Am I missing something? Why put this in a key-value store?

You are right, a relational database could be used. However, SimpleDB is a very simple system that has a much higher availability than any relational solution that we could think of. In general, it is just there. Next to that, it does not have many features, but it supports just the stuff you would like to do for logs. On top of that, we write many more logs than we read, and we don't care so much for realtime consistency in logs. And it is cheap.

What about a highly-structured ACID datastore designed primarily for short transactions reading & mutating small sets of random rows makes you think that? Log data is semi-structured and very heterogenous. I would choose the filesystem far before an RDBMS.

Maybe RDBMS isn't the /most/ efficient way to store the data, but the advantage of being able to write simple and powerful queries against that data is huge. I seem to lose that with file or key/value based storage. Am I wrong? (not trying to criticize, I am genuinely curious)

Also, I don't need to worry about files across servers and rolling log files that file logs usually entail.

I'm not suggesting that a K/V store is better, just that an RDBMS isn't a good solution to this problem either. How do you effectively utilize the query capabilities of an RDBMS with semi-structured data? You're better off sticking the data in a flat file and using text processing tools to work with it.

How do you get around the 10GB/domain limit?

What do you do when your fleet gets big enough that you start getting throttled?

What's the query latency?

How do you delete old log data?

What is the cost like? What's the logging volume?

Reply on news.ycombinator.com