Hacker News Reader

> improve your product significantly without as much subjectivity

It's fake objectivity that often leads to a race to the boottom.

Between "engagement" as a terrible proxy for user happiness/success and the idea that it's more important and better if nobody can fail to use your app's "UX" than it is if it to enables a smaller number of people who take the time to learn to do really great things, all wrapped up in the VC driven dream of hypermegablitzscaling, all a reliance on analytics does is guarantee mediocrity.

I want software created by people who care enough not to look at the data.

Analytics data are like emotions: a valuable source of information which should not be used as a direct controlling signal. It should be first thought through.

I didn’t say it was objective. I said it allows you to be less subjective. I have no idea how anyone could think otherwise unless you’ve never had, for example, analytics show you your visual design or interactive flow is literally broken on specific but rare device/platform/browser/etc combos. (Just one example.)

As usual, it’s possible to be stupid and abuse tools, but what I wrote presumes you’re not doing something stupid. If you think it’s impossible to not be stupid when it comes to this tool, this is false.

I've personally never been in a situation where analytics drove change. Even mere A/B testing seems like a ghost in (most?) organizations, where management or designers decide to force through their personal preferences.

That said, I would love to join such an organization. Any suggestions on where to find data driven jobs?

> Any suggestions where to find data driven jobs?

My experiences suggest a large measure of caution. Many organizations describe themselves as data-driven and many actually do try. However humans are humans and many will be “data driven” when it aligns with what they want to do, otherwise it’s off for “more data” or a “new perspective”. In other words sometimes data driven means use data when it supports my already staked out position. I hate to be so cynical but I can’t help it.

One might argue that your cynicism is (drum roll please) data driven.

It may be less used in driving website design, but I assure you it is very very widely used in marketing to determine which campaigns are working.

Amazon?

The second half of this article explains Amazon's process around metrics, well worth a read:

https://commoncog.com/goodharts-law-not-useful/

At my previous job, A/B testing was widely used to check which of the small UX tweaks works better.

The problem there is many uncontrollable factors contributing to the data. If engagement is found to be better with A on the day, does that mean B will never be better? It doesn't.

A/B testing is often conducted unscientifically or with insufficient sample size and timeframe.

Sometimes the cards fall where they fall, and your small UX tweak wasn't involved. It's tempting to conclude A/B testing delivered a valid result. As an experiment, if you deliberately made A and B identical, then A/B test as normal, you will still get a winner. They won't be equal.

This is fair; valid experiments need to run for weeks, not hours, to account for cyclic variations, should not be confounded by some events (like holidays, elections, etc) that severely change relevant user activity, etc, and need to bring large enough amounts of data points for the results to be able to be statistically significant.

Well yeah you have to actually do it right.

The pendulum has swung too far. We quantify everything now, chasing numbers to the point of losing our sense of quality and craftsmanship. There's a huge difference between "going with your gut" and the ability to assess, qualify, and reason about something. To understand how and why something works or doesn't. Whether from first principles or experience.

Numbers provide a happy path for consensus. If the "why" of your project is to please as many people as possible, then web analytics probably works great. I'm not arguing they're categorically useless, only trying to surface the unspoken costs I see bear out in our industry from relying on them so heavily.

The original post proposes they be abandoned. So I’m arguing against that.

> Before web analytics, going with your gut was the only option.

The analytics people insistent denial that usability research ever existed seems to have reached a new level.

What I wrote wasn't an insistent denial, try steelmanning next time. Extrapolating the magnitude of problems from user research is hard, since you're still sampling a distribution.

If the already-released version of your product has so many problems that you have to use computational techniques to extrapolate their magnitude, then you have already lost.

You can find problems like “oops, the Android they use in India on phone model X has chrome that blocks the screen every week so they can’t get passed the login flow”

It’s about better sampling of the actual distribution. User testing has huge selection bias.

What you wrote was just denial. The insistent part comes from every time those discussions going this same way.

Anyway, and FYI, one of the most robust findings in user research is that the magnitude of problems isn't really something important to know.

You could hire a decent QA department.

Then you'd run the app on all screen sizes before each git push and notice that the button doesn't always show up, or isn't always clickable.

The point is that there is a swath of failure modes that only become apparent when you actually have instrumentation point at them first. QA can’t fish these out.

Absolutely, and a good reminder that your web analytics is only one source of data you can use when making decisions. Sites and apps generate so much interesting quantitative and qualitative information that is important to compare and validate assumptions.

"It's very easy to be fooled by a vocal minority of people who complain about your app sucking when the majority are getting value from it."

Does this apply to "feature requests" by a minority whereby the majority then has to accept those "features".

Yes! Early stage businesses can get bogged down in "next feature folly" - keep adding and adding as each new (bespoke?) client comes on. Then the product has features baked in for narrow cases and turns off the larger part of the bell curve.

My line in the sand is: never build a feature on the first request. Ideate, research, measure others opinion of the same features.

One should have a tight beta group for this.

> show you lost 90% of your users because your button doesn't show up on certain screen sizes

How does analytics do that? A mobile usability scanner might pick that up, but analytics?