Hacker News Reader

Comment by ajkirwin | Hacker News Reader

You aren't running this in the standard language/caching/db setup, then?

You're storing EVERYTHING in memory? Isn't this kind of.. well, stupid, frankly.

While the comment above may not have the best manners, it does bring up an interesting point. I certainly wouldn't mind hearing more about HN's setup.

If you're really curious and have the time, you can download the source: http://arclanguage.org/install

It'd be interesting to read a write up on the concepts used (such as the in memory database) and the server restart mechanism. It'd be hard to tell from the source for non Lisp programmers.

An in-memory database is pretty simple - it's just a bunch of hash tables.

A server restart mechanism is also pretty simple. Write a cron job that activates every few seconds to ps aux and grep for the server name, and if it's not there start the server. That won't catch server hangs rather than deaths, but that's exactly the problem that happened here.

Thanks, I understood it uses hash tables - I've done the same myself, but never on such a large scale. This post sums it up: http://news.ycombinator.com/item?id=513259

Dude, frack no. Having everything in a database means you are tied to the speed of disk, instead of the speed of ram. The biggest issue with being purely ram resident is handling multi-threaded updates across your ram resident dataset.

You could always use Erlang...

"Dude, frack no. Having everything in a database means you are tied to the speed of disk, instead of the speed of ram. "

database are not tied to a storage medium. there is no reason why you can't run a DB (key-val or even a full RDBMS) in RAM.

If everything is purely RAM resident, wouldn't you lose it all when the server gets wedged?

No. You journal the changes to disk. You therefore only need disks fast enough to keep up with the journal. If you get into a situation where your disks can't keep up with the journal, your site is probably big and popular enough that you can afford to hire DBAs to go from there.

That's why you use intelligent caching.

For a site this small, there's absolutely no reason the entire database shouldn't be cached in RAM. 32GB of RAM costs about $800. That buys you plenty of time to not have to worry about caching, and instead gives you more time to work on interesting features. For a single-person operation (or even a few people), you have to spend your time wisely.

But of course, any sane database will cache as much in RAM as possible anyway. But instead of dying when you run out of memory, it just evicts the least recently used page.

Reply on news.ycombinator.com