Hacker News Reader

Comment by cosmic_cheese | Hacker News Reader

Absolutely. I think that over-reliance on analytics has had a similar effect; compared to relying primarily on proper usability research and in-person user studies, doing that creates a detachment and distance that further removes the project from reality.

How great the impact is depends on how management uses the data. If they use it add resolution to broad strokes from studies and the like, there may be no negative impact at all, but in my experience it's much, much more common for management to read analytics like tea leaves and interpret it in whatever way best fits the individual's/team's biases/agendas.

View on HN

>> I think that over-reliance on analytics has had a similar effect

I lost track of how many times the directors would tell us, "Since the analytics say A, we should do B." then the research person says, "Sure Jim, great idea, lets get a few rounds of useability research and confirm it first."

Then the always predictable thing happens: Users never align with your analytics. Seeing data and seeing someone struggling to do something basic with your interface is totally different. When directors see these videos, it really makes them see how important research is and not just relying on data to make decisions. Right now, anything the research team wants, they usually get - its had that profound of an impact on our leadership team.

> Users never align with your analytics.

Any good theories for how this happens (i.e. why the data fails to capture the struggle)?

Not GP, but I can humbly share mine.

I think it's because a lot of analytics are data points oriented instead of being workflow oriented. So you can see that feature A is not being used a lot, but it's very important in a particular flow. Feature B may be used a lot, but it can be only important for a particular class of users while very detrimental mentally for another class.

People use software for a needs, but rarely I've seen a need being highlighted when interpreting analytics data.

Analytics don’t capture the “why” of what they measure, on the user side. Indicators may move in the seemingly right direction for the wrong reasons. Analytics typically can’t tell you what the user wanted to achieve. Knowing the “why” gives a better basis to decide on what changes to try, or to realize what’s actually wrong with the user-facing side of things. Another reason is that they tend to measure an average where in reality there is no average user.

Interpreting data is hard.

Eg, if you have a bunch of airplanes returning from a war, you might be tempted to armor the areas where they were hit. But you want to armor planes where the holes arent because those are the critical areas.

If you can read literal bullet holes backwards, hundred dimensional preference vectors say something, but we generally have no clue.

I think the answer to this is roughly ... the content of an applied statistics degree, or a research methodology practicum. There are so, so many ways that quantitative measurement of human phenomena can mislead, from measurement error to conflation between trends/sample sizes/effect sizes, to p-hacking, to good old bias (in interpretation or in deciding what tools to use to smooth or normalize data), and thousands more.

My problem is I can skip all those steps and 8 separate meetings by applying a little common sense to software design.

Unfortunately, orgs often treat devs like this as indulgent or wasting time. They need it to be done the inefficient way because that's the only way they have visibility and control.

Yes. And in most of those meetings where the researcher invalidated the leaders, the leaders have left the meeting saying (and thinking? I’m not sure) they were validated. So because of the leaders we have we should skip the meetings and research in both cases.

Simple example: feature X is rarely used (thus we should get rid of it)

Hold on, why is it rarely used?

Is it hard to use? Is it hard to find? Was it poorly named? Does it work right? Does it get me 80% of the way there? Does it get me 20% of the way there? Do I rarely need it but when I do need it it's a huge time saver? Am I hesitant to depend on it because I fear it will be taken away in a future update?

And following this, I'm sure we've all seen the pattern of manufacturing the data required to justify the removal of features. Bury feature → usage numbers tank → oh hey look nobody uses this any more, guess we can safely cut it!

> Simple example: feature X is rarely used (thus we should get rid of it)

Oh man, I remember a very specific example of this: many years ago now, Google Chrome pushed an update that got rid of the option on the menu bar for "Close Tabs to the Right". I remember looking into the Google issue tracker where people were complaining, and some PM provided a "data-driven" justification: when people opened the context menu, they only clicked on the "Close Tabs to the Right" option 1-2% of the time.

It's a great example of why data without context can give you the wrong answer. Of course the option is used relatively very rarely; you only need to clear out your tabs every once in a while, compared to creating new ones or managing tab groups! But it's still an essential task. It's like saying filing your taxes isn't important because you only need to do it 0.2% days of the year.

For what it's worth, I definitely don't organize my tabs in a way where "close to the right" would be helpful in cleaning them up.

It's useful when starting a thread of research of research as new tabs will be opened to the right (Ctrl+Clicking a link). So after it's done, close to the right can be quite useful.

I do. I open a search page, middle click a bunch of links, then want to close all of the ones to the right because I found one that told me what I needed and don't need the rest of them.

Probably a perfect example of why it's so useful for developers to talk to users

Not only to understand the usage of features, but also use cases perhaps to design even better features

I do all the time because the feature exists. I would treat it like a sliding line in my tabs bar. When I'm done with a tab for now it goes to the right, and eventually everything to the right gets closed once I haven't used it in a while

I don't like that they have that data in the first place. But then again I don't use Chrome.

Conversely: Feature X is often used... Because to drive adoption someone made it mandatory for a certain workflow and it nags everyone else so they "engage" with it just enough to make it go away. (Looking at you, "AI" buttons...)

> I think that over-reliance on analytics has had a similar effect;

The over reliance on analytics combine with a lack of metrics and proper accounting.

If you are renting all your infrastructure knowing how people use your app, and what the COSTS of that are is kind of a big deal. If your high dollar client is your lowest margin one, thats a problem that is technical and financial as well as a product insight.

And that infrastructure your renting, it stopped making sense for a lot of orgs to do that almost a decade ago, but here we are where everything is in the cloud because capacity planning is a lost art and was a great throttle on the insanity that the post is describing.

Reply on news.ycombinator.com