Hacker News Reader

Comment by jwr | Hacker News Reader

I turned off Google Analytics, because I realized that it doesn't actually report any useful or actionable data, just vanity metrics, and many of them of dubious quality.

I run a SaaS and what matters for me is paid subscriptions. "Visits" (even if by humans, which is hard to tell) really do not matter much. Yes, I do want to increase conversion rates, and run bandit experiments, but I'm better off doing that myself.

What also matters are search terms, but Google's search console (or tools, or whatever it's called this week) provides that.

Turning off Google Analytics was hard to do psychologically — the Fear Of Missing Out is strong. But it turns out I'm not missing out on anything, except some dubious vanity data. And I'm making the web a better place in the process.

View on HN

Analytics may give you the number of paid subcriptions but that is not the reason it exists. Modern analytics systems are build to give you a statistical view of all events/interactions leading up to a conversion or, more important, to a conversion that did not happen. With this information it is possible to optimize all touchpoints a user has with your services.

It is easy to track all aspects of subscriptions including recurring revenue. The data quality depends on the data you send to google analytics and is not "dubious" but in your own responsibility. And google analytics is really good in separating bots from human visits.

If google analytics did only report vanity metrics to you, you most likely did not use it the right way. Maybe you missed the segmentation tools to find groups of users for whom your service did not work out?

> google analytics is really good in separating bots from human visits

That certainly wasn't my experience.

But more generally: I optimize touchpoints using automated Bernoulli bandits with multiple variants. And I track the metrics that really matter (like signups, MRR, churn, etc) very, very carefully. My point was that just adding GA doesn't bring much value, and makes the web worse.

Another example of how page visits don't matter: it's easy to get a huge spike of HN users clicking through. But if my SaaS has no relevance to HN users (except as a technical curiosity), this doesn't matter at all. It won't change my revenue, so it's irrelevant.

Unless you run a site with ads, focusing on page visits doesn't make sense: it's like measuring the performance of a supermarket and getting excited about increasing traffic on a nearby highway. Could it influence your sales? Possibly. Is it actionable? Nope.

It's more like the supermarket keeping track of what isles people are going down and what items they are looking at. Which supermarkets would do if they could because it's extremely valuable data.

> Which supermarkets would do if they could

They can, and they do. Here[1] is a random example of applying that to a supermarket (the other pages on that vendor's site shows some of the other ways the tech can be used).

Solutions tend to use a mix of video camera data and cellular data (wifi and bluetooth beacons[2]) to track in-store movements and behavior.

Done well, they can also connect individual store journeys to specific point of sale transactions, and understand hoe variations of in-store behavioral patterns ultimately influenced actual transactions. Which opens the door to running in-store optimization experiments very similarly to how you'd run A/B optimization tests for websites/apps.

[1] https://retailerin.com/en/retailerINfor/customer-behaviour-a...

[2] https://www.thinkwithgoogle.com/marketing-resources/retail-m...

Very cool!

GA used to be good in separating the bots from the humans, but now I don't think so. I see a huge number of visitors from Dublin that seem to be AWS sourced bots (and Asburn, both have bounce rates like no other city). Millions of visits, from a place where we would expect thousands.

GA's "Filter bots" setting simply applies the IAB bot filtering list[1]. The resources required to operate a page rendering bot[2] used to be a high enough bar that the ones doing so were large actors and would make their way onto that list pretty quickly.

The IAB list is still a helpful baseline (if overpriced if you want to lease the list itself[3]), but all it's doing is applying suppression based on things like known bot IP ranges and user agents. It's far less effective than it used to be since it's so incredibly easy and cheap nowadays for anyone with an interest to spin up a rendering bot. Now you've got to supplement it actively with your own set of filters and heuristics if you really want to get rid of bot traffic polluting your data.

[1] https://iabtechlab.com/software/iababc-international-spiders...

[2] Scraping bots were cheap and easy to run before, but didn't render the GA javascript code so never showed up in analytics. It's only bots that use something like a headless browser to render the page that show up in GA, and those have only become commoditized and cheap/easy relatively recently.

[3] Google applies the IAB list to your traffic for free if you check the setting for it, but if you want to use the IAB list yourself you have to pay $4k - $14k annually to lease it from IAB.

The bots got better and harder to detect.

You can actually set analytics to exclude traffic from a certain ISP or set of IPs on the View's Filters page. There is some legit traffic that comes from data centers, but only a tiny fraction.

I also don’t think google analytics is that useful. It’s one of many possible tools for user tracking, and far from the best one. Thought it’s “free” so I guess that’s why it’s more popular than much better tools.

But the fundamental problem is that analytics provides data and information, when what people want is knowledge and wisdom. But you need to do the work to get it. No amount of analytics is going to tell you that your Facebook ads are underperforming their potential because your buy button is hidden by an overlay in the built-in Facebook browser, especially if Facebook is your best performing channel overall. Analytics can’t tell you what’s not there. There’s no substitute for having developers and marketers actually purchase your product, themselves, on the channels your customers use. And too many companies don’t do that.

Just a hypothesis I have, that most e-commerce companies are leaving millions to tens of millions of dollars on the table by not giving all their employees a corporate credit card and having them purchase their product on it, say, once a month. They’re losing out on far more than they’d end up paying for the few instances of fraud.

For a personal project everyone basically ignored every post I made about it. I used server logs to generate analytics and noticed people hit the first page and left (I assume they spent seconds and left).

I then broke up the main page to many smaller pages and notice people still leaving right away but my documentation page got more overall clicks since the main page didn't massively overload them

I guess analytics don't give you answers but know what pages they tend to click on and what happens when you make a page more simple/more heavy you can figure out a solution

However I didn't end up finding a solution and I'm planning to rewrite my project. I have a better idea how I should introduce it to people next time

I feel the same way.. mostly. I want one thing from GA and that is user flow. I've first hand seen this help a startup run landing page experiments and very quickly to improve their conversion rates. (The startup was selling a luxury consumer product)

This won't work for everyone and will be of little value to many. However that tool just doesn't seem to exist on most if not all of the lightweight alternatives. Matomo is the only alternative I have seen implement this feature, though those with more experience with the alternatives will hopefully show me I am wrong on that.

The main product I work on does not use google analytics, but we do use mixpanel to see what our paid users are actually doing with the SaaS. We actually don't care who each user is, we just want the aggregate data. We believe this data is important for retaining paid subscriptions and attracting new ones. Let me explain.

There's one feature that 100% of our subscribers use. I mean, it's the main thing we do and everyone who subscribes needs that function. Even without analytics, we know that we have to continually make that feature faster and smarter to stay ahead of our competition. We know that if we fall behind our competitors, our subscriptions will dwindle.

But, then we have a bunch of other features that help customers solve some edge case problems around that main thing. It's very important for us to know which of those functions are being tried, used, and reused (or not). Not all customers use these features, but some are tools we have that nobody else in our space does. So, having insights on which ones are getting traction and which ones need improvement help us spend our marketing and engineering time better to attract new customers and retain our existing ones.

I can't imagine not having any analytics. I feel like we need them to continually make small course corrections that ensure we're providing the best value to our existing customers.

> We actually don't care who each user is, we just want the aggregate data.

> It's very important for us to know which of those functions are being tried, used, and reused (or not)

Aren't these 2 at odds with each other, or else how can you tell when the same person re-uses a feature? Surely you need some kind of user identifier for that?

Yes, good point, I should have been more clear. You're right that we track unique users, but we do so in abstract. I suppose with some work we could pull together enough data from different sources to determine who a particular user was in a particular session. I just meant that we don't do that and it's not easy for us to do that because we don't care about that type of data. We only care about what each abstract user does in the sense that we want to know the aggregate of how many users did that thing.

Additionally, unlike google analytics, we do support the browser's "do not track" flag. So if a user doesn't want to be tracked at all, we completely respect that.

So far GA is answering two important questions for me - which marketing strategies are actually working (because it's hard to tell when you've got multiple going at once), and also making sure my marketing is actually hitting the geo-regions I need it to.

That said though once I know which marketing tools are effective, there's nothing more that GA does that CloudFlare couldn't just tell me anyway (i.e. am I getting more or less traffic) and I'll probably drop it as it's one less dashboard to look at - like you said that conversion to subscriber _is_ the ultimate metric for success.

This was my experience as well. Early on, when I was still learning, it helped me learn the importance of extremely low Time-To-First-Paint. But now, I just default to good designs that are easy to read. There's nothing new to be learned that GA could provide insight on.

Also, early on, I found the Referrer tracking to be useful for discovering the reach of my projects and to help me get into conversations with users on other sites to help them with using my software. But that feature of GA eventually became useless when Google did nothing to address Referrer spam.

> I realized that it doesn't actually report any useful or actionable data

The actionable part never occurred to me, but makes so much sense. What action could anyone really take, based on the data presented by Google Analytics? On top of my head I can actually think of anything you could easily get from server logs.

Examples I saw in a website I launched last year: * High bounce rate showed that people did not find the content engaging / not what they were looking for, confirming a hypothesis I had; this was particularly true for certain sections of the website. * Time spent on pages was another metric that was helpful to find problems in some sections. * Easy slicing of traffic sources by referrer and country, showed me that most of my "good" traffic was coming from facebook, so I invested more time there.

To optimize the bounce rate metric even further a funnel can be created and measured. This allows to see bounces for each part of the funnel up to your conversion(s). It is often really valuable to see how a change on your website improves one part of the funnel but impairs another one, and than to think about the reason why this happens.

There are millions of possible answers to this question... Examples:

- Conversion rate grouped by browser and browser version to find browser specific bugs. - Total revenue per user per marketing channel / search keyword / ... to optimize budget allocation. - Revenue by mobile OS version share to decide testing procedures to not optimize for users that don't contribute to your bottom line. - Client side loading times per user location and provider to optimize infrastructure placement. - ...

One of the reason Google Analytics (and logging solutions like it) is popular is that people don't have access to server logs.

Server side logs aren’t an accurate way of doing analytics and bring up compliance challenges (storing access logs and using them for purposes other than security etc.)

> Server side logs aren’t an accurate way of doing analytics ...

Based on what? Server side logs records what the server itself sent. That sounds like it should offer far more accuracy than what a JS only solution would do.

That being said, if the code for analysing the server logs is lousy, it's not going to help. :)

The identifier in server logs is IP address, which is useless for distinguishing users. On the corporate side, NAT hides all the visits of many people behind a single IP. On the consumer side, ISPs change IP addresses all the time so a single person will show up as multiple IP addresses.

Note that I'm just talking about distinguishing people, not identifying or tracking. A basic question like "how many people came to my website today" is more accurately answered with client-side analytics than by analyzing web server logs.

> A basic question like "how many people came to my website today" is more accurately answered with client-side analytics than by analyzing web server logs.

How can client side analytics be more accurate, when ad blockers stop visits from even being registered by client-side analytics?

Before looking at accuracy, you already have compliance challenges with logging IPs / user agents and using it for analytics purposes.

Please don't try and change the goal posts. You said:

> Server side logs aren’t an accurate way of doing analytics...

What is your thinking behind that statement, as it sounds incorrect?

What compliance is challenged by logging user agents?

Can you talk a little bit more about why server side logs aren't accurate for analytics? I was planning on doing basic analytics (pageviews mostly) using Cloudfront access logs stored in an S3 bucket and queried through Athena for a site I'm working on which uses the AWS stack.

I know Cloudfront logs can sometimes drop, but is there a more important reason you're talking about?

I think with server-side only is harder to filter bots, crawlers and get accurate bounce rates or session length times.

Shameless plug, you might want to have a look at https://github.com/ownstats/ownstats

You mean unpopular?

Popular. Because adding lines of javascript is much easier than processing server log files.

I find time spent on page is a great way to measure performance of redesigns on page. There are numerous actionable points.

It might be, assuming two things:

1) That you actually care about this metric. I don't, I do not get paid by the number of minutes spent on pages, I get paid by the number of signed-up subscribers who use my software to make their workflow easier. I can (and prefer to) use bandit testing to measure the performance of redesigns.

2) that it can be reliably measured, which I don't think it can.

We are a SaaS company also, and time spent reading manual/tutorial pages is important to us.

With regards to point two, being reliably measured sounds to me like perfection is the enemy of adequate. Perhaps in low volumes you can't measure certain stats like this reliabily but in large volumes I think it's useful.

Tom Gilb once said that "everything can be measured so accurately that the result is better than having no measurement at all" or words to that effect. I'm sure he phrased it better. In this case, some users aren't measured, which drags down the accuracy of the measurement.

Even if the numbers are off by quite a few per cent, I can easily see how some site operators might benefit from knowing e.g. that visitors close one tutorial much quicker than the others.

As long as your measurement is a random sample everything is okay. Even if it is not, it is much more information than you had before. You just need to keep it in mind when evaluating conclusions. We are not talking about drug tails here and no one dies if the measurement is not 100% accurate.

I am able to measure everything 100% accurate. But this is really really expensive. It's a trade-of.

Unless your page is google.com, where more time spent on page means the experience is worse, not better.

Context is important, and it's one metric amongst others that should be used.

The point is, there are clearly actionable points GA can offer. It's disingenuous to think there are not.

I agree, saying that there are no actionable points in GA is like saying analytics in general are useless.

Counter example: I search for information on a subject. I get many interesting and relevant results and therefore spend more time on the page.

This makes it even more obvious that a longer or shorter session length is not clearly better or worse, it depends on a lot of other things.

Hook into onbeforeunload and call your own javascript to post back to your own site to let you know when they've left.

I know how I'd do this, but have no reason to do it.

Well, you don't specifically need GA for it, but tracking referers to sales, for example...is often actionable. Similar for a funnel view to see where visitors drop off. Especially combined with some A/B testing.

Good for you! This is the right method. Analytics -- in the sense of watching mouse clicks and reference urls -- are only trying to close the feedback loop FAR sooner than when it actually matters.

Depends how big you plan on getting. Be infinitely easier to hire someone who knows how to use GA to make decisions, and historical data will be useful at that point.

Do you not find value in adding events to your pages so you have user funnels for conversions? Or do you think bots cause too much noise for this?

Reply on news.ycombinator.com