Hacker News Reader

Comment by steventhedev | Hacker News Reader

This is why I use server side analytics almost exclusively. You just can't rely on every browser running all your javascript.

View on HN

With ad blockers, user tracking gets harder and harder, and this sucks when you want to understand the effects of changes on your own website, such as changing user menus, moving content blocks from a position to another, or even to a different page, etc. The value of partnerships with other websites gets harder to evaluate too.

Hosting my websites on Linux environments, I never stopped using server-side generated statistics (based on Apache's access logs) with tools like AWStats. Used together, I feel that client and server-side stats give a much better image of what really happens on my websites.

But in the end, I feel awstats isn't enough, and I'll take a look at other solutions that have been pointed out in these comments. Thanks!

Is Apache reporting back to the apache foundation about how you are using your apache software? Maybe your expectations about user tracking are the problem.

I'm pretty sure they aren't, but I must admit, I have never checked. I supposed that would be known by now... One thing is sure, access logs are located on my own VM, and are processed locally by awstats. The only thing that binds requests to an actual user is the IP address, and one must contact ISPs to have more information. I don't have the authority to do that.

What interfaces akin to Google Analytics read web server logs?

Piwik can. Run a self-hosted instance or pay them to run it, and it handles server logs no problem.

You do give up the ability to get live stats, but you get better performance and the ability to track more visitors (read: people like me who have blocked GA for years...)

Added bonus: Referrer spam is automatically blocked by default.

https://piwik.org/log-analytics/

EDIT: Almost forgot GoAccess, if you are okay with a terminal app and want live stats (can also be scripted to generate HTML reports) - http://goaccess.io/

Piwik is astonishingly good. It'll give you trouble if you have a lot of traffic, but for moderate sites it's pretty much a completely self hosted drop in replacement for Google Analytics. You can track conversions, set up campaign goals and view statistics overlays for example. Not all features are available when using the log import backend, as compared to the javascript thingy, but most are and it'll give you an overview on how your ad blocking customers are doing.

Quick note that with hourly, or more frequent, cron jobs you get quasi live data. Mentioned a little about logimport in a recent post on stripping external calls and pointless js/css/font loads from our site. Otherwise well put, same experience here.

Btw, tail -f /log.log is always fun for live data...

Yeah, I build a system in the late 1990's that did something like that. Required a dedicated server though, the web server farm would copy over their log files to this analytics machine with cron jobs, which would then dynamically aggregate that data and update the stats cache.

Charles, Snipcart co-founder here. As mentioned by the end of the post, we're actively looking to expand our analytics game a bit. Piwik seems like a good place to start, so thanks for sharing. Cheers!

Try running it alongside GA and comparing stats. I did this for a while and switched to Piwik as it consistently counted more visits (legitimate visits, obviously).

Also, please consider adding your Google Analytics use to your privacy policy. They require it, even though most people ignore this requirement and there is no enforcement.

Side note, I just tried @ replying to your companies post on Twitter and got a @your account may not be allowed to perform this action' http://i.imgur.com/9uoEn9B.png Strange, never saw that before...

We received some mentions in the last minutes, and we just reviewed our account settings, nothing seems problematic on our end. Weird indeed. But thanks for the feedback!

> (legitimate visits, obviously)

That's an important point; I'm seeing referral spam in my GA reports which I never noticed before 2015. I manually discount it when compiling reports, and my understanding is that these spammers hit GA UAs at random without even loading your website.

It's incredible easy to filter this out in GA. Just set a filter or segment that only counts traffic from your known hostnames.

aside from the ELK stack, you can also send views/events from your server to GA (this is what I do). Staccato[0] is my go to gem for this

[0]: https://github.com/tpitale/staccato

keep in mind that you can push stats into GA from the server. I actually made PoC for pushing server stats to the GA - https://github.com/hippich/server-analytics

The same way you can gather all available stats on server side and push these yourself. You can override IP too. Obviously not everything will be available, like screen resolution, but you still can capture a lot of stuff.

How do you filter out crawlers, bots, pre-fetchers, etc.?

I've personally never even tried using server-side analytics because I assumed there would be so much noise it wouldn't be worth the trouble.

Every well-behaved bot (ie, almost all of them) has its own user agent string. You can just grep -v those out.

I wish that were true, but I'd estimate that less than half of the bot hits on my website properly identify themselves as bots. Most of the crawlers running on DigitalOcean/AWS/etc. seem to use a user-agent string lifted from one of the common browsers.

Check for the order of the headers and the TLS protocols it claims to support – those are useful for identification.

I must be getting old, never? There's a lot of techniques like doings stats on blank pixels and such that have been around for a long time. A lot of ad blocking techniques start at the DNS of the hosted resource, so having a single domain for everything (if practical) can present some reliable information.

Reply on news.ycombinator.com