Comment by moot | Hacker News Reader

Comment by moot | Hacker News Reader

Parent Full thread

You're absolutely right -- it is negligible. When you're serving upwards of a petabyte per month, 23 GB isn't exactly a lot!

> On the other hand, if saving single bytes was significant, there would be a lot more potential in the source by shortening CSS class names etc. rather than picking a short domain name.

Also spot on, but the point I was trying to make is I was given the choice of choosing a longer domain and a shorter one, and the shorter one resulted in smaller page size, which does result in some (though as you put it, negligible) savings in terms of transfer. CSS/JS refactoring/pruning would definitely be a better bang for your buck if your goal was solely to reduce page weight, but my primary goal was to decrease request overhead and this was just a side benefit at no additional cost to me.

As an aside, I would say the non-technical benefit of the longer domain (4chan-cdn.org) would have been avoiding user confusion, but I feel this is mitigated since visiting 4cdn.org directly bounces you to www.4chan.org and our custom error pages make 4cdn.org clearly 4chan related.

Right, if you use a short domain name "because why not" for something new that's alright. It was just the "it adds up" part that I was commenting on.

Do you have (well, you're moot, so -- want to share) some more data of 4chan's current size? I'm sure lots of people would be interested in hearing about that. (This would probably make a great individual post.)

I'd be very interested in this... I used to visit 4chan all the time back in the day, not so much now, but I do enjoy reading statistics about the site.

Thanks for this article moot.

I've been meaning to write a post about all of the weird stuff we do in the interest of maximizing our limited resources. We've always had to stretch things as far as possible given server, financial, and time constraints, which has led to some interesting/unorthodox "solutions."

That's what interests me about 4chan's story... How you do that is amazing.

What is your caching like, and do you use precompression? In fact, how do you serve that many pages full-stop? We (HN) need to know!

We write pages to disk as compressed HTML, and make use of nginx's gzip_static and gunzip modules to serve them. So every time a person posts, we regenerate the applicable reply and index HTML. The high postrate boards are rebuilt with a daemon on a timer, since past a certain point (~1 post per second) it's wasteful to regenerate on demand given how long the script takes to run.

We essentially think of flat files on disk as a cache for the database, and don't employ a proxy cache or any other common HTTP proxies. It's a little unorthodox, but it works well for us.

Flat files on a disk == awesome

We've had a surprising amount of difficulty maintaining this over the years, since FreeBSD's NFSv4 implementation kind of sucked for a while, and it doesn't support mounting a memory partition (tmpfs). But you can trick it by mounting the memory partition using nullfs, and then nfsmount-ing that.

We used that memory-partition-over-network thing for a while, but actually switched to SSD-over-network because it was faster than the memory partition. I spoke with a FreeBSD maintainer about it and he said what we were doing was so unsupported/unoptimized that he wasn't surprised.

We run into weird FreeBSD edge cases pretty often where there are few people, if anyone, who can answer our questions. Sometimes I wish we'd gone with Linux, but after ten years the hassle of switching doesn't seem worth it. Thankfully 9.2-RELEASE has been pretty good to us.

Have you considered memcached with nginx HttpMemcachedModule? If so, why didn't you decide to use that instead? Seems like all of 4chan's active posts could easily fit in memory of server with a moderate amount of ram (especially if compressed).

Why even write the html to disk? 4chan flushes history so aggressively I'd think you could fit all the high-traffic stuff in memory.

if there's a power-blink and the posts get lost? Its 4chan.

Without fsync, its effectively that; but you get atomicity (with unlink etc) and can use multi process etc.

Reply on news.ycombinator.com