Hacker News Reader

Comment by quchen | Hacker News Reader

> 50 bytes may not seem like a lot, but when you’re serving 500 million pageviews per month, it adds up.

That's a void argument in the article. If they were serving pages with 500 bytes each this would indeed be a huge improvement, but no page is 500 bytes. I just opened 4chan.org, and the markup before <body> is already 1,836 bytes. The entire frontpage of /b/ is 114,428 bytes, and saving 50 is absolutely negligible. On the other hand, if saving single bytes was significant, there would be a lot more potential in the source by shortening CSS class names etc. rather than picking a short domain name.

EDIT: According to http://www.4chan.org/advertise, there are a total of 575,000,000 monthly page impressions.

View on HN

You're absolutely right -- it is negligible. When you're serving upwards of a petabyte per month, 23 GB isn't exactly a lot!

> On the other hand, if saving single bytes was significant, there would be a lot more potential in the source by shortening CSS class names etc. rather than picking a short domain name.

Also spot on, but the point I was trying to make is I was given the choice of choosing a longer domain and a shorter one, and the shorter one resulted in smaller page size, which does result in some (though as you put it, negligible) savings in terms of transfer. CSS/JS refactoring/pruning would definitely be a better bang for your buck if your goal was solely to reduce page weight, but my primary goal was to decrease request overhead and this was just a side benefit at no additional cost to me.

As an aside, I would say the non-technical benefit of the longer domain (4chan-cdn.org) would have been avoiding user confusion, but I feel this is mitigated since visiting 4cdn.org directly bounces you to www.4chan.org and our custom error pages make 4cdn.org clearly 4chan related.

Right, if you use a short domain name "because why not" for something new that's alright. It was just the "it adds up" part that I was commenting on.

Do you have (well, you're moot, so -- want to share) some more data of 4chan's current size? I'm sure lots of people would be interested in hearing about that. (This would probably make a great individual post.)

I'd be very interested in this... I used to visit 4chan all the time back in the day, not so much now, but I do enjoy reading statistics about the site.

Thanks for this article moot.

I've been meaning to write a post about all of the weird stuff we do in the interest of maximizing our limited resources. We've always had to stretch things as far as possible given server, financial, and time constraints, which has led to some interesting/unorthodox "solutions."

That's what interests me about 4chan's story... How you do that is amazing.

What is your caching like, and do you use precompression? In fact, how do you serve that many pages full-stop? We (HN) need to know!

We write pages to disk as compressed HTML, and make use of nginx's gzip_static and gunzip modules to serve them. So every time a person posts, we regenerate the applicable reply and index HTML. The high postrate boards are rebuilt with a daemon on a timer, since past a certain point (~1 post per second) it's wasteful to regenerate on demand given how long the script takes to run.

We essentially think of flat files on disk as a cache for the database, and don't employ a proxy cache or any other common HTTP proxies. It's a little unorthodox, but it works well for us.

Flat files on a disk == awesome

We've had a surprising amount of difficulty maintaining this over the years, since FreeBSD's NFSv4 implementation kind of sucked for a while, and it doesn't support mounting a memory partition (tmpfs). But you can trick it by mounting the memory partition using nullfs, and then nfsmount-ing that.

We used that memory-partition-over-network thing for a while, but actually switched to SSD-over-network because it was faster than the memory partition. I spoke with a FreeBSD maintainer about it and he said what we were doing was so unsupported/unoptimized that he wasn't surprised.

We run into weird FreeBSD edge cases pretty often where there are few people, if anyone, who can answer our questions. Sometimes I wish we'd gone with Linux, but after ten years the hassle of switching doesn't seem worth it. Thankfully 9.2-RELEASE has been pretty good to us.

Have you considered memcached with nginx HttpMemcachedModule? If so, why didn't you decide to use that instead? Seems like all of 4chan's active posts could easily fit in memory of server with a moderate amount of ram (especially if compressed).

Why even write the html to disk? 4chan flushes history so aggressively I'd think you could fit all the high-traffic stuff in memory.

if there's a power-blink and the posts get lost? Its 4chan.

Without fsync, its effectively that; but you get atomicity (with unlink etc) and can use multi process etc.

You are looking at the traffic sent by the server to the client but cookies are sent by the client to the server. Upstream vs. downstream. Cookies are much larger proportion of the request. Also the response can't even be sent until the entire request is retrieved.

Excellent point... we can also safely assume the average user's upstream connection is quite a bit slower--likely on any metric other than latency--than his downstream.

In addition, let's not forget to multiply the cookie overhead by the number of images on the page... likely to be quite a large number given 4chan's love of images and enormous, eager-loaded discussion threads.

the markup before <body> is already 1,836 bytes

All those bytes are justified because they are rendered on the screen provide direct utility to the user. The extra 50 bytes does nothing but slow the page down.

there would be a lot more potential in the source by shortening CSS class names etc

There is no either/or here: this can and should be done as well.

Is there a good CSS minifier which looks at the CSS as well as the HTML, and shortens class names automatically? In most cases I don't really care what the class names are in the final HTML, as long as they match up with the CSS.

I realize there are a lot of things that could go wrong, such as * correlating the HTML and CSS for an entire site instead of just one page * dealing with third party dependencies that require certain class names to be used

Just wondering if there's any work already done with this approach.

I don't think there's anything that will change the class names, but there are minifiers

http://yui.github.io/yuicompressor/css.html

Changing the class names would be difficult as it wouldn't pick up any dynamic class names, like

    var className = 'user_' + user.getState(); //'user_deleted' or 'user_active'

But that's the same problem as minifying JS libraries already, where you have a set of public symbols that should not be altered. An exclusion list works well in that case where you could just put the class named that are for dynamic generation (you probably don't do that with all classes in the stylesheet).

No need, because you can gzip compress, which will make even better savings than that (because it can also compress and save space taken by tags etc, not just class names).

You could do that and gzip though; it would probably yield some savings.

An obfuscator generating code à la iocc efficiency would be quite neat (e.g. http://www.ioccc.org/2012/endoh1/endoh1.c)

>You could do that and gzip though; it would probably yield some savings.

Nope. That's classic non-engineer thinking. The same kind of thinking that does "optimization" without profiling.

Let me introduce you to my friend: http://en.wikipedia.org/wiki/Diminishing_returns

"some savings" says the parent. A diminishing return is still a return.

And 0.0000000000001 of a dollar is still money. Yet it's meaningless to do anything about getting it...

I can say the savings would probably be greater than the 50 bytes he saved on the domain name. Compressing is not magical, if you compress class names you're still storing at least one instance of each name plus overhead.

Sure, but that's just by pointing image URLs, etc., to a shorter domain vs. a longer one.

There's also savings in not sending cookies with every image request - which saves them a honkin' 46 terabytes.

Plus, he himself admits that this isn't the first place to start optimizing - it's just where he chose to.

That's still 25GB/month of bandwidth they save. Who knows, maybe that nudges them out of a high-cost data tier.

My point was that "saving 25 GiB" is way below anything they can possibly care about. How much traffic does 4chan handle in a single hour? I think it's pretty safe to assume that 25 GiB is significantly lower than one standard deviation of their monthly traffic, so cutting costs with it is also not feasible.

Ok, so let's assume you are right. Did he make the wrong choice? The shorter name seems strictly better to me- whether we feel it is trivially better, it saves traffic and the domain name has lost no information.

If he reworked old code just to save 50 bytes, that would probably be a mistake, but it sounds like the work was being done anyway and he had the choice to save 50 bytes OR use a longer domain name.

> If he reworked old code just to save 50 bytes, that would probably be a mistake, but it sounds like the work was being done anyway and he had the choice to save 50 bytes OR use a longer domain name.

Basically this. I thought it was an interesting side benefit that came at no additional cost to the main benefit of greatly reducing request overhead for static resources.

You often pay less per GB the higher you get

> 50 bytes may not seem like a lot, but when you’re serving 500 million pageviews per month, it adds up.

This is rubbish. What matters is how many packets of data go through the network. It is packets which are the unit of transfer and which are handled by intermediate nodes. The difference between say 2340 and 2290 bytes is of no material consequence whatsoever, it may cause one less packet to be sent but probably won't. To think it does have a consequence is to demonstrate a complete lack of understanding of what happens in the network and in endpoints. None of these '50 byte savings' accumulate anywhere in any meaningful or measurable sense whatsoever. So no, it doesn't "add up" to anything.

And if you're going to downvote me explain why I am wrong and how the benefit of these mythical 'savings' can be demonstrated.

Packets are not padded to the MTU, so a link literally needs more time to send a longer packet. If you make a response 50 bytes shorter, the last packet will always clear a 1 Gbps link 0.37 ms sooner, even before you shave a packet off 3% of your responses. How much you care depends on what you're doing and how bursty it is, but it's certainly measurable at scale.

If the few bytes saved by this new domain seems worth commenting on what about the very large number of unnecessary whitespace characters they are serving?

They have around 100 images on a single page, and google analytics increases the 50 bytes with 1KB. This does indeed result in 100kb overhead

I think you're confusing the two benefits.

What the thread OP is talking about is the decrease in page size from the URLs included in the page source. Choosing a shorter URL for the static domain versus a longer one resulted in a rough savings of 50 bytes per page (size), compressed.

You're referring to request cookie size, which was also decreased significantly (CloudFlare still sets a single cookie, unfortunately), which results in the big savings of ~100 KB upstream per page load.

Kind of off-topic, but how do you feel about doing a conscious MITM to your users(by using Cloudflare), being 4chan home of Anonymous and all that?

4chan logs IPs and provides them to the cops if the ask, so it's not like users who want to remain completely anonymous can do so without being behind seven proxies anyway.

Yeah... people shouldn't ever assume that "anonymous" means "anything goes"

Providing IP's to law enforcement is not the same as allowing a third-party to do a MITM. Cloudflare can inject whatever content they want to the pages they serve.

> > 50 bytes may not seem like a lot, but when you’re serving 500 million pageviews per month, it adds up.

I thought you were quoting me here, I wrote the same text yesterday only with a figure of 100 million pageviews. That was weird to see.

Reply on news.ycombinator.com