Hacker News Reader

Comment by jasonkester | Hacker News Reader

The reason that Google is choking on this 30M pageview/month site is because analytics for a 30M pageview/month site is hard.

Things like unique visitor, unique referer, unique whatever tracking just explode when you have to compare each IP address you get with 20M-entry list.

If you try to do it on your webserver, you'll have a 3 hour window each day where a significant portion of your CPU and Memory is not available for serving web traffic. If you try to do it elsewhere, you get to ship 20gb daily logfiles around.

It's just not any fun. Which is why even Google has difficulty with it.

Running your own analytics is great for relatively small sites. For big ones it starts to bog down once you hit a certain size.

View on HN

Most stats providers compute unique visitors using client-side cookies; not entirely accurate, but quite cheap. You're right though, it's a lot easier to use an existing (and free or nearly free) tool than to build your own.

I agree it's a lot easier. But Google Analytics has a drawback I can't accept: several sites I visited make me process scripts from Analitycs, and that takes a noticeable amount of time and bandwidth. When someone visits my site, she's giving me her time. I'd rather not waste it.

That, and the fact that I prefer to process my logs on my computers. As a matter of principle (decentralization, privacy, freedom, blah blah).

Now, I barely know Analytics. But it looks cool, so I'll check, and see if I can reap most of their benefits myself. (If I can't, I'll have a tough choice indeed.)

Yep, GA used to have the occasional problem with that, though I've been told some of it was due to a bug in the browser (Firefox's) code that decided what to put in the "Waiting for xyz..." spot -- if that's what you're referring to.

(Something about that bar not being updated until the first byte of the stream was read -- so if a static.foo.com was hanging connections, it would get stuck blaming whatever loaded right before it.)

GA now has asynchronous tracking though, which solves this pretty nicely. Though I'm not a JS guy, my understanding of it is that instead of sourcing the script and then issuing the tracking calls, you put some instructions into a global variable, and then source the script. Nothing blocks on the script's loading, so even if GA is down the rest of the page is supposed to load correctly. Once the GA script loads, it looks at the global and runs the tracking commands.

I do use Firefox, so I probably hit that bug. Also, with NoScript, forbidding GA causes no problem.

But still, I would have liked the analysis process to be completely invisible from the client.

Reply on news.ycombinator.com