Hacker News Reader

Comment by heipei | Hacker News Reader

As much as I applaud every alternative to Google services, I would think about these kind of analytics twice when running any kind of developer-centric service. Chance is that your target audience is running an adblocker / DNS blocker of some sort and that this will heavily skew the analytics results: The absolute numbers will be a lot lower and you will only capture folks not savvy enough for adblockers. I removed any external analytics about a year ago when I realized that more than 50% of visitors to my service were blocking Google Analytics and that GA numbers had become meaningless (cross-referenced against access.log).

View on HN

Keep in mind that the Internet actually has more bot traffic than human traffic these days. A 50% discrepancy between Google Analytics and access logs could be explained entirely by bot traffic with no ad-blockers involved.

It could be ad-blockers, though. And I want to emphasize a point you mentioned briefly: it very much depends on your target audience. An anecdote from several years ago, around with IE was still the plurality of traffic on most sites but no longer the majority. Two web properties run by the same company, with different audiences. On one of them, IE did not crack the Top 5 in browsers, and was actually outnumbered by Opera. On another, the entire Top 5 browsers were different versions of IE, and Chrome and Firefox combined were less popular than IE6, IE5.5, and AOL's browser (which apparently was a reskinned IE7).

The moral of the story is, what's true for everyone may not be true for you, and vice-versa.

The vast majority of bot traffic identifies itself though. Search engines from all over that you've never heard of, but they're all showing accurate user agents. I don't attribute every access by far, of course, but for a small website (with peaks to high HN ranks, so it's not unknown or that badly linked) it's manageable to tail -f the access log and see that the traffic you're getting matches what you expect after e.g. posting a link in a chat group. About 99% of traffic in such a case fall within the categories "human-looking user agent visiting this newly created URL that I just shared in the group" and "random bot traffic".

The vast majority of what I guess is "robotic, but javascript-running" traffic identifies itself as the (headless) browser it most likely uses, i.e. Chrome.

When such crawlers use a distributed network of machines to crawl, there's no easy way to identify them as such, other than _guess_ that they're robotic in nature due to them coming from "data centre" like IPs (i.e. AWS, Azure, GCP, waht-have-you).

While such requests may well be users using those "data centre" services for, say, a personal VPN... the overall pattern is what gives them away.

Not all users look at a page and then start requesting (maybe from a separate IP, mind you, but still from the same provider yet with the same UA) each URL in that page in succession.

Crawlers who want to stay under the radar are much, much harder to detect as such nowadays.

En masse, with a popular website, sure. But from my limited traffic website where I sometimes look at individual requests by hand and see that it's my friends from residential Dutch IP addresses loading the page, I'm quite sure these are no secret masqueraded internet robots. Hence I conclude that, since much of the traffic (in periods that I look at randomly, the vast majority I never look at) is identifiable as human or as clearly a crawler, such unidentifiable bot traffic is a very tiny percentage.

I'm sure it'll be there, lots of people will be interested in what differences a page shows from different vantage points or to robots, but it's probably less than a percent of all robot traffic.

Though checking the thread above mine, I'm not exactly sure what I was replying to. Didn't it say something about robot traffic that wasn't identifiable as such? I don't see that anymore.

This relates to an article I wrote and shared here recently about marketers being obsessed with bad data - ie looking at GA even though the data in GA is heavily skewed and potentially entirely misleading due to issues with adblocking, etc.

https://news.ycombinator.com/item?id=25016532

What tools are you using for internal analytics? Streaming logs into something like GoAccess?

Streaming structured JSON access logs from nginx into an ElasticSearch via logagent-js, and then query via kibana. But goaccess is fine if you just want quick aggregate information!

Thanks, may I ask how you are processing visitor locations? IP through some geo services?

nginx has a geoip module that you can use.

How many people using adblockers block analytics rather than just ads?

Fundamentally I am much more concerned about blocking tracking that actual advertising. So I have my blocker set based on that. I'm pretty sure GA gets swept up as part of that.

The default block lists on many adblockers will also block common analytics platforms.

If you're interested in how many visitors are using an adblocker you can use Admiral's [1] free analytics on your site.

[1] https://getadmiral.com

(Full disclosure: I'm a co-founder at Admiral)

Boy, is your company's product nicely done but annoying! :)

Any plans to make the "continue without disabling" button harder to click in the future?

The design and functionality of the Engage modal is actually up to the publisher.

One of the local news stations uses them on their website. They have a tiny "continue without supporting us" link that blends in nicely with the text around it. The weird thing is I always see that link while my wife never does. I don't mean she can't find it; it simply isn't there. This is using Chrome on Android. No root or anything special. Doesn't matter if her phone is on wifi or mobile. On the other hand, I see that link every time. It's been that way for about a year and I have never been able to figure it out -- although admittedly, I haven't tried that hard.

The publisher can choose to remove the option to bypass the modal after a certain number of visits or after a certain period of time.

Good to know. Thanks.

Getadmiral.com didn’t even get past DNS blacklists, so not sure what exactly you’re reporting: Blocked by 1Hosts (Lite), ABPVN List, Anudeep's Blacklist, Energized Blu, hostsVN, Lightswitch05 - Ads & Tracking and oisd.

I’m not seeing where the free analytics is on the homepage (at least on mobile).

But I’m curious - do you have any off the cuff stats on what is typical as far as % of users with Adblock across various types of audiences?

Tech-related sites can have 40%+ desktop adblocking. Mobile blocking is usually <10% no matter the vertical. The less tech-heavy the site is, the less adblocking but it's usually at least 10% on desktop.

Also, on the pricing page there's a "Get Free Analytics" button that you should be able to click. If you have any issues you can let me know: james at getadmiral.

Admiral's paywall is so glitchy on mobile, when I don't have an ad blocker installed but it frequently thinks I do, that I've learned recoil in disgust from any brand that uses your service.

That's unfortunate! Usually when this happens the user has a network filter, enterprise firewall, or something else on the network that's blocking some traffic. Publisher's have an opportunity through our platform to offer subscriptions as an alternative to whitelisting so you can view the site without ads. They can also issue you a free subscription.

Products like Admiral are why the Internet sucks. Everybody treating the Internet like their own little capitalist fiefdom consuming everything you do so you can be repackaged and resold a million million times.

I'm old enough to remember an internet that respected the sacred open ungated plain of knowledge. Publishers lost the privilege to monetize when they insisted on tracking, whether the visitor paid or not.

Weren't you the toxic sludges trying to DMCA your way out of EasyList and co?

How do you sleep at night?

Reply on news.ycombinator.com