Paul Buchheit: The secret to making things easy: avoid hard problems | Hacker News Reader

Paul Buchheit: The secret to making things easy: avoid hard problems

paulbuchheit.blogspot.com

33 pointspaul32 comments

Want an example of "simple but less reliable"? Here's one (in one or two easy steps): 1. All updates go in to memcached, but not the database 2. (optional) A background process occasionally copies entries from memcached to the db. Without this, the values will be completely lost when memcached restarts.

Paul - here's an interesting take on your recommended solution: Use solid state ram and forget about step two. You get fast a read/write, without the risk of data loss if the machine restarts.

Here's a pretty cool board that will take DDR SDRAM, and supply it with a constant source of power (imitating solid-state memory): http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=1

Those who insist on a database, could even store the database on this board. Best of both worlds.

Here's an insane graph showing the database transaction rate for the board vs. a conventional hard drive: http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=7

Solid-state drives greater than 128GB are in OEM testing now. Expect server economics to turn upside-down in about 12 months, when persistent storage seek times get 100X faster.

Yeah, solid-state storage is clearly the future for most apps. Flash is over 10x cheaper than DRAM though, so it'd be nice to see some high-performance flash drives as well. It's also nice to not have to worry about losing data during a prolonged power outage.

No need to wait, really. Addonics' SATA/CF adapters for your $80/8GB flash cards cost $35. Even cheap compact flash does random seeks 40x faster than platters. Throughput is a little slow but what the hell, for 115 bucks? If I were building a server farm right now I'd try those things out.

This is a cool thread.

So please spec out a couple servers for us. How would you get a bunch of memory (32GB, 64GB even) into one of them and still have it be a good web/app server?

My current favorite is the Altus 600 from Penguin Computing. Cheap chassis with dual sockets and 16 RAM slots. More slots means cheaper chips. 2GB chips for that board are about 300 bucks. Throw two 2.2GHz Opterons and 2x 300GB drives in there. Add an SATA controller card, 4 SATA/CF adapters and CF cards (in the CD-ROM bay), and you've got a nice machine: 1U, dual processors, 32GB of RAM, 600GB of platter, 32GB of flash for about USD$7500.

Expensive, yeah. 5 grand of that is RAM. You could get away with 4GB of RAM and still have a fucking fast web/app server as long as your active dataset fits on the flash.

$215: http://www.newegg.com/Product/Product.asp?Item=N82E168201343...

Brings the memory cost down to around $3440.

Wrong speed but yeah, that's the ballpark. I used Crucial to price the RAM.

It's too bad that the transfer rate is so low. It should be pretty simple to build a very fast device which spreads the blocks across a large number of flash cards. I suppose you could do a raid array of addonics adapters, but it's unclear how that would perform (and it might be a mess dealing with a large number of them).

For many database applications, I don't think the transfer rate matters that much.

Regardless, the DDR-ram solid state solutions have faster transfer rates than a 4 drive raid0. The only drawback is that they cost about $100 per GB.

Oh, and the onboard battery will only let you unplug your machine for 8hrs. After that, the state is lost.

Does SATA / CF have any advantage over a pack of USB drives?

I am not sure what out-of-the-box solution can you use to read/write packs of flash memory in parallel, but I'm pretty sure there exists something (you could do it on the application level as well).

Edit: USB RAID-0 array benchmarks here:

http://arstechnica.com/reviews/hardware/flash.ars/8

http://www.bigbruin.com/reviews05/thumbraid_5

The advantage is that you're not going through the USB bus. The spec says 480Mbit but in real life it's much much lower. Does your rackmount server have quality USB2.0 (heh) or the cheapest 1.1 crap the manufacturer can find (likely)?

On the other hand I don't have hard numbers, just some bonnie tests on random flash sticks I had lying around. Check out the newer USB sticks that are "Readyboost Ready", such as Corsair. They might be quick enough.

The problem is that not many people benchmark the write access time for flash. It's pretty bad, often 50ms+. There is a product I know of, the Zeus IOPS, which gets extremely fast write speeds to flash. Unfortunately it costs about $250/GB. For that price, you can get a RAM based solution, which is even faster. Ones like http://www.hyperossystems.co.uk/ can come with a battery backup and a hard drive that it will dump to and restore from if the power goes out. Price is about $220/GB. The prices sound high, but if you're trying to scale a database, it can potentially solve an otherwise intractable problem without any software changes.

There's also the i-RAM ( http://techreport.com/reviews/2006q1/gigabyte-iram/index.x?pg=1 ), a battery-backed RAM disk. My employer uses it for an app where disks simply couldn't deliver the latency we needed, and yet it had to be fault tolerant enough to survive a power failure.

Edit: wow, totally did not see Sam_odio's original post, linking to exactly the same product. Oops. And yCombinator doesn't allow us to delete comments...

An even better secret to making things easy: only try to solve problems that exist.

This is a nonexistent problem: "The problem? Most databases are still storing my name using the same techniques that they did 15 or more years ago -- they seek the disk head to some specific location and then read or write my name to that location. These databases rely on the one operation that DID NOT dramatically improve in the past 15 years! They still perform like it's 1991."

I don't think PostgreSQL is the only database that uses write-ahead logging.

Post some numbers. How many updates per second can you do?

I don't know. If that becomes a problem I'll put the write-ahead log on a faster disk to buy time to look into it.

Why can't you just succumb to peer pressure and get rid of that database? :)

I'm not sure. First I'm going to get rid of that operating system. It has lots of features that aren't relevant to my web app.

It's somewhat amusing, but also mildly depressing, to see these remarks despite the fact that I explicitly stated (in bold, even), that "I'm not advising you to ditch your database!"

And yet somehow that gets translated into "get rid of that database."

Don't be depressed. You started something that's spawned (as of now) 28 comments. Great thread.

Sorry it's mildly depressing. Unfortunately, any thoughts you bring up related to a polarizing issue are going to end up being framed in terms of the polarization. That's just the way it is; some things will never change.

On the up side, you are going to learn a lot about essay writing, and about how concepts do/don't get through to people. These are very valuable skills.

But wait, didn't you say it was a non-existent problem?

Someday updates per second might be a problem. The nonexistent problem is "they seek the disk head to some specific location and then read or write my name to that location." I may have missed it, but I haven't seen you admit yet that you were wrong. You've been informed now that reads use a memory cache and writes use a write-ahead log, right?

I'm probably at least partially wrong about everything..

It's true that PostgreSQL uses a write-ahead log, but it's still going to do that seek when it checkpoints, correct? I don't think that it will be able to sustain high transaction rates (100,000 tps for small transactions should be very doable). If you find that this isn't true and that someone has demonstrated these rates, let me know -- it would certainly be good news.

Yes, it will still do that seek when it checkpoints, but I seriously doubt it will do one seek pert transaction at that checkpoint. Still, I doubt 100,000 tps can be achieved with PostgreSQL.

As for being wrong, that's one of the privileges you lose with celebrity. Everything you write in your blog is going to be preceded by "The guy who wrote gmail says..." When you say something that looks like it might become a popular misconception, guys like me will jump up and down, scream, and tear at our hair. Or maybe we'll just post terse, cold, unfriendly corrections. We still like you, though.

My main issue with your post is the way it comes across as saying databases are overengineering for 99% of web sites, but mainly deals with a performance issue that won't matter for 99% of web sites. I would have no problem if you framed it as high tps vs low tps for certain classes of application, not as simplicity vs overengineering. Using a relational database is a fine default choice for web apps today, just as you expect to have an OS on your server by default.

Fast disks are only faster by a factor of two.

I think I could address any such performance issues within a factor of two growth of updates per second.

167 small transactions per second on MySQL 4.1 / InnoDB / FreeBSD / FFS / SCSI, which looks like one per rotation. Presumably the time required to write a block at the end of the write-ahead log.

5000 per second with --innodb_flush_log_at_trx_commit=0, which only writes the log to disk once per second.

The MySQL documentation claims that this configuration does crash-recovery correctly, though you may lose the last second's worth of transactions.

BTW, in case anyone is thinking, "why would I need more than 167 transactions per second", imagine that you have a web site that displays everyone's "last active" time (for chat maybe). If users are performing some activity once every 30 sec, then you will be pretty much be at your limit with about 5000 simultaneous users.

Robert, did you test MySQL with a parallel load? To give it the best chance at maximizing tps, it will need to be able to combine multiple pending transactions in a single write. Also, can you tell if it's updating it's b-trees? (if not, it may not be able to sustain these rates)

I ran the test with 1,000,000 INSERTs (taking 200 seconds), with the innodb cache size set to only 200,000 bytes. So there's a fair chance it updated the b-trees on disk.

Loved this point:

"The thing is, a lot of those difficult problems are irrelevant for 99% of products. For example, "real" databases can handle transactions that are too large to fit in memory. That was probably a really important feature in 1980. Today, you can buy a computer with 32GB of memory for around $5000. How many GB transactions do you suppose Twitter performs? My guess is zero -- I suspect that their average transaction size is closer to 0.0000002 GB (messages are limited to 140 characters)."

Reply on news.ycombinator.com