Comment by mattlanger | Hacker News Reader

Comment by mattlanger | Hacker News Reader

Funny to see this on Hacker News -- I wrote it. Which means I should also take full responsibility for the performance complaints that have shown up in this thread.

Someone in the thread mentioned that they're "not sure how you could screw up a web app this simple." I'm entirely sure how we screwed it up.

My friend came up with the idea, designed the layout, and I built it. It was a spare time thing, just for fun. When we launched we seeded the database with a couple hundred questions and put it out there. Somehow (thank you Twitter, Tumblr, etc), the site saw 10,000 uniques in its first day, which exposed the fact that I'd over-normalized the database to death, and the site could hardly function. We launched at 11am, and by 3pm I was on the phone with mediatemple to upgrade the server just so there'd be enough memory to allocate an ssh session.

I spent a sleepless 72 hours after that rebuilding the architecture to try and make it scale, which it did -- for a while.

But now we've got a database with 13 million votes in it, and the app must cross-reference those votes with a user guid so as to serve fresh content, and it's simply outgrown whatever fixes I was able to make during that marathon coding weekend.

I haven't done any work on it in months, yet every so often it pops up somewhere on the web and guilts me into thinking I should probably re-architect it once again...

So if I don't sleep or leave my apartment this weekend I've got you all to thank :)

So if I'm getting you right, you think that it is of tremendous importance that a user not be asked, in their 5 question run, whether to choose Good or Evil twice.

And you're willing to take a MASSIVE performance hit to guarantee that, for all users.

OK, let me take a wild stab in the dark: you've got over 1,000 unique pairs in the database. You ask the overwhelming majority of users 5 questions. The chance of a user getting 5 unique questions, if you just pull them randomly out of the DB, is about 99%. And that would be Pretty Darn Fast to calculate.

Or, here's a cheap trick for you -- partition your table of questions into 5 groups. A really simple way is to take primary key mod 5, but if you want to do it better you just hash the input and then take mod 5, then store that in a group column. For the first question, choose randomly from group 1. For the second question, choose randomly from group two. etc. By construction, it is now impossible to ask the same user the same question twice.

Which means you don't have to store a user's prior votes at all. Just store counts.

There, enjoy your million-fold performance increase.

Well, it's not just that we wanted to avoid asking the same question twice in a 5 minute span... we do have users that spend literally HOURS just answering questions (instead of paying attention in class). So in order to maximally squander their education, we wanted them to receive new questions all the time.

>>So in order to maximally squander their education, we wanted them to receive new questions all the time.

Sometimes I think this is the true reason the internet was created.

(additionally, you can substitute education with noun of choice)

It's cute and pretty, and pretty nice for a spare-time project. I like it well enough already. I'll just wait for the traffic to die down a bit.

Reply on news.ycombinator.com