Hacker News Reader

Comment by transcriptase | Hacker News Reader

I’m convinced 90%+ of telemetry is simply to have usage metrics to show managers, rather than actually being used to improve products by identifying inefficiencies and bugs. And it’s a malevolent spectre if you’re the 1% of users who actually use a niche feature in something, since developers will use that knowledge to take it away.

Despite a 100-fold increase in hardware power, and languages/frameworks that handle 80% of the workload a developer 15 years had to think about, applications these days have fewer settings, open and run slower, and improve far less rapidly than anything written in C/C++/Java from 1990-2005 despite phoning home with novels worth of telemetry every few minutes.

Monitoring network traffic and DNS requests through a firewall and pihole, it’s downright hilarious that it’s the worst products that top the charts in both, even when being online is not a necessary component of the software. Meanwhile the software that just works, and works well, might check for an update once a month, and the changelog generally reveals new useful features and mentions performance increases.

View on HN

> And it’s a malevolent spectre if you’re the 1% of users who actually use a niche feature in something, since developers will use that knowledge to take it away.

Yes, but also no. If my resources are not infinite, knowing that an "expensive-to-maintain" feature is used by just 1% of users, it is an incredibly important piece of information when I'm gonna have to decide where/how to recalibrate my (team's) effort.

Product managing is also this: I have to know what works and what doesn't. Maybe I will use that knowledge to understand how to increase the 1% to 5% with a specific business case. Maybe it won't make sense and I will have to decide if I can keep it or not.

I agree that vanity telemetry is bad for users, but killing ALL telemetry is bad for everybody.

I don’t disagree with anything you’ve said, and this is only tangentially related, but I wonder how much the long-tail of usage is impacted by the likely correlation of power-users/niche features, and people who firewall/pihole telemetry.

I also wonder how much of a chicken-egg effect there is on the trend of developers hiding niche/power-user features that they would prefer to sunset deep into settings menus and then that setting seeing negligible use.

That is usually information you already know from other sources

(e.g. is the 1% feature a feature that is pivotal to getting new users?) (e.g. is the 1% feature a feature that is key for a specific audience within my total addressable market?)

But you have know from somewhere that the 1% is, in fact, 1%...makes sense?

Ok, but if we focus on results, why has software usability and functionality declined with the rise of telemetry.

I personally think it is because program managers rely on telemetry instead of stakeholder research and industrial design, either because they don’t know better, or because management thinks it will cut costs.

Besides, telemetry is a feature that 0% of users use, so it should be the first to be cut.

Subjectively speaking, while software I use these days has a lot less settings, I also tend to need to change settings a lot less than back in the day, defaults tend to be just right a lot more often. So at least for me it seems to be working, at least somewhat.

> Subjectively speaking, while software I use these days has a lot less settings, I also tend to need to change settings a lot less than back in the day, defaults tend to be just right a lot more often. So at least for me it seems to be working, at least somewhat.

But there's a trade-off there—as, ideally speaking, most software has become easier for most people to use mostly satisfactorily right out of the box, it becomes all the harder to adapt if you're one of the people for whom it doesn't work mostly satisfactorily, since there's no real culture of design around the idea that a user might want to use software in the way that they, rather than the designer, intended. Even if it's fair to write off those niche users, there are still the most users for whom the software works mostly satisfactorily, but can't be modified to be perfect, again because the culture of user customisability just isn't there.

Right. There used to be a design philosophy of making the defaults fit the majority of users, but to expose plenty of settings so that more advanced users, including those who have grown into the software, can still do what they need to do, and can work around design decisions that are bad for them.

That seemed to be a reasonable compromise to me.

In fact, it's something the US Air Force figured out after spending substantial time and resources.

That is to say: There is no such thing as the Average Man.

Make something that will generally fit most people, but then also add enough customization features so each man can tune it to his liking. There is a reason car seats and steering columns are adjustable.

The dichotomy isn't between telemetry and no options vs. no telemetry and all options. Somehow a decision needs to be made what to include because everything has at least an opportunity cost, and historically that has usually been made by "common sense", i.e. what whoever is in charge feels like is the right set of options. That can work out great if you're very much like that person or able to influence them (and they want to listen), but if you aren't, too bad, and the vast majority of people out there is not in either position. Late 90s/2000s me was probably a huge lot less like those people than many here on HN, and I guess that explains some of the difference in experiences. I generally seem to be quite a lot like "most people", which makes an approach that targets the middle of the bell curve work well for me. I'm not a big fan of embedding surveillance technology in apps, but I'd say the people who do are on to something and what's missing may just be a more transparent and privacy-preserving way of doing it.

My subjective experience has been the opposite. Modern software tends to be inflexible and limited, and those limitations absolutely adversely affect me. There isn't a day that goes by where this sort of thing doesn't interfere with what I'm trying to accomplish.

If you only care about users for which your software works, then most users spend almost all of their time as “power users” (= ones that have used the software for more than a few days)

Focusing purely on out of the box experience looks good for quarterly metrics (subscriber growth with low initial churn) but you end up with a population of users that are desperate to switch away.

At that point, you are one viable competitor away from being screwed. Worse, as users defect, telemetry will tell you that usage of the features associated with switching is declining, so you’ll double down on stripping out the functionality that you should be investing in if you want to defend your market share.

See also: Firefox.

Unfortunately that's the conclusion that most managers will draw, without asking more questions.

But knowing that only 1% of users use this feature is not enough to make that decision:

- maybe 1% of your users only use it, but they are the ones that pay the most money

- or it's the most prolific users so it generates half of your content

- or it's a niche feature used by power users, but those users are the ones bringing other users, introducing you to companies and doing free support

- or it needs to be used by only 1% of the users by design, and it's enough to be useful (E.G: bug report form)

- or your report it wrong, it's one percent of all accounts, but 10% of actually active users

etc.

Most telemetry will not tell you that of course.

Well, tracking users by cookies or fingerprinting or whatever can tell you that by combining data from different sources. Now we've looped back around to being in favor of the most intrusive methods.

That's my point, if you want really useful information, you need identifiable users, and for that, you need consent.

So ask for it.

All data will be biased and incomplete anyway, so at least acquire it in an ethical manner.

As I said that is a "piece of information", not the whole story. Making an axing decision with ONLY that information would be outright idiotic.

Telemetry has to be designed as well, to piece as much as you can, and then you complete the full picture with other sources. Maybe not everybody does this, but I don't think that this practice should be "shocking" for anybody.

yea you sound like someone who is competent and will actually thing things through and make a rational decision. Many people are not and will latch on to the information "only 1% of people use it!" and refuse to listen to why that may not matter"

> Most telemetry will not tell you that of course.

Why...? If that's the case, it sounds like the problem is too little telemetry, not too much.

Well, in Europe, if you want to know how much a user pays, you need to tie telemetry to their account containing personal information, which is forbidden by GPRD unless they give their explicit consent and they have a process to delete/rectify/download all of it.

Since most telemetry are optout and don't bother about giving users access to their data, they can't legally do it.

That's my point really: useful telemetry is really, really invasive. So just ask consent, and accept most users will tell you no.

If you chose what's convenient instead, you will not only have a mixed bag of data, but also you have decide to basically force yourself on people, in an age where a part of your userbase is getting upset over things like using the wrong pronoun on them.

I tell yes to Firefox and VLC to report on my usage. It tell no to Google and Microsoft. They want their cake and it it: behave badly, and then have we trust them.

> give their explicit consent

They already need to agree to Terms of Service when paying and starting to use the software.

And that is not explicit consent.

Explicit consent needs to be standalone and refusing it cannot stop the user from doing anything that doesn't actually really require said consent.

Right. A mention in the ToS does not equal real consent. I would only add that it also has to be informed consent. Saying something like "can we collect data from you to improve our products and services?" is also not soliciting real consent, because it tells the user basically nothing about what is being asked of them and how it will be used.

You think that keeping track of "This paying user uses this and that feature, and is thus on the correct price plan" is a violation of the GDPR, assuming it's mentioned in the privacy police and maybe ToS?

I don't know what does or does not violate the GDPR, and I'm not talking about metering services. I'm talking about telemetry for installed software.

I'm not even talking about what the law says, because the law is very inadequate on these matters (in the US, anyway). I'm talking about, from a common-sense point of view, what "consent" is.

In my view, it can only be counted as "consent" if I have been informed exactly what it is that I'm consenting to, and I am asked for that consent.

ToS don't count for a number of reasons, starting with the fact that nobody reads them, it's unrealistic to expect people to read them (because everything comes with lengthy ToS documents and if you really read them all, little time would be left to do anything else), they are generally difficult to properly understand if you're not a lawyer, and they are are usually intentionally vague of this issue -- which means the "informed" part of "informed consent" isn't satisfied even if you do read them.

I think the Principle of Least Surprise should apply here. If software is going to do something that users won't notice happening, is intrusive, and isn't pretty obvious from the nature of the software, the right thing to do is to tell the user about it and ask for permission.

Software that doesn't do this is adversarial to the user.

That seems interesting, when thinking about pro rated subscription fees -- if a feature isn't being used, it'd cost nothing. But with telemetry disabled by default, the company couldn't detect that the feature wasn't being used, and thus the customer would pay more. And if opting in to telemetry, the customer pays less and the company gets telemetry.

the example I like to always use is when EA kept abandoning Madden (especially online franchise mode and gameplay) for like 10+ years straight (while making money hand over fist on MUT).

One time there was this new mode, for like 1-2 years, of 3v3 online. it was probably the most fun mode madden has ever had. it just needed a few tweaks. sensible stuff. But it would've been amazing!

but no. because it was difficult to get to in the menu system, fewer people used it and few people knew about it. so of course telemetry is going to say few people used it. but it had a TON of potential! a product owner made the wrong decision to cut that feature, and guaranteed they used misleading telemetry to justify it.

Pay users for research. They are not your labrats. Is this really even a question?

The problem isn't that you have a feature used by 1% of the population. It's that you have like 60 features each used by 1% of the population, but with not much overlap. So all the 1% features get axed and suddenly you have only 50% of the users.

If you axe 60 features and there are 20 features left that are used by that 50 percent, then that might be a legitimate tradeoff from a business perspective.

Ah yes, the Firefox effect.

As I said that is a "piece of information", not the whole story. Making an axing decision with ONLY that information would be outright idiotic.

> So all the 1% features get axed and suddenly you have only 50% of the users.

In practice things rarely get that far, thanks mostly due to the complete farce that is "data driven design". Such an approach to design would naturally lead to the axing of every unpopular feature if it were followed mechanistically, but in reality "data driven design" actually refers to the practice of mining to support whatever you (the designer/programmer/manager/etc) have already decided to do, and the data is ignored for anything else.

It actually works like this:

I form a subjective judgement against some feature which I dislike (maybe because it's a pain to maintain or just because it doesn't suit my preferences for any reason.)

Then, I look in the telemetry and start making charts of things until I can find some combination of the data that seems to support my prejudice against the feature. I bury all the presentations of the data which failed to support or even refuted me, I pretend I never saw them.

Then I write up my report claiming that the feature should be removed for objective scientific reasons, and my fancy charts and graphs prove it! It's totally not just my subjective prejudices driving the design, no Siree...

Besides the privacy/consent concerns, this is why telemetry is bad for users. It devalues dialogue between the users and developers and replaces it with pseudo-scientific arguments based on nothing but the developer's preferences. That's what it's really for, an excuse to never talk with the users.

> If my resources are not infinite, knowing that an "expensive-to-maintain" feature is used by just 1% of users

Only 1% of mobile phone users need 911 regularly. Lets remove that feature.0

> Only 1% of mobile phone users need 911 regularly. Lets remove that feature.0

If this is the type of PM you are accustomed to, fire them. Because that's a dumb argument.

You know that a lot of legislative and regulatory effort goes into making sure 911 works on as many phones as it does right?

> If this is the type of PM you are accustomed to, fire them.

People's expectations about how projects treat 911 are born from those hard requirements. If it didn't have those requirements, you would find that every PM is the kind of PM that would let the service suffer. Maybe not scrap 911 altogether, but certainly deliver a less robust and reliable 911 service than they do now.

Or if you're Google, promote them (see pixel 911 issues)

Unfortunately there are many PM who are basically that type.

> Product managing is also this: I have to know what works and what doesn't.

You can't use a blunt usage metric to tell whether a feature works or not. It only tells you that the feature is popular for some reason. That reason might be that it works, but often the reason people use a feature a lot is it's shoved down their throats.

It's a vicious cycle. Say you are Product Manager for Feature X. Your bonus/promotion/career progression depends on some vanity usage metric for feature X. The more monthly actives or new users for feature X, the better it is for you. So, naturally, you're going to do whatever you can to funnel users at that feature. Stick it prominently on the main screen. Highlight it in flashing red. Spam the user with notifications begging them to use it. Make competing features low-contrast gray on gray. Whatever you can get away with. Then the next monthly metrics report is in: The feature is now being used by +15% of users since last month! Incredible! Your bonus buys you a Tesla. Management's conclusion is: "This feature really works and users must really like this feature." So, the product focuses more on developing that feature.

Now scale the above out to every feature in the app, each with their own competitive and motivated Product Manager. This is how we get apps that are all-telemetry dumpster fires and with more and more flashy features fighting each other to grab your attention and fewer and fewer useful 1%-features.

I've personally witnessed meetings with multiple furious product owners literally shouting at each other about where their feature's button gets placed on the home screen because that placement made the difference between 5% feature usage and 25% feature usage. Where was the end user in all these discussions? Not present. End users are to be milked for metrics gains.

> You can't use a blunt usage metric to tell whether a feature works or not. It only tells you that the feature is popular for some reason. That reason might be that it works, but often the reason people use a feature a lot is it's shoved down their throats.

I never said that was the only source of information, but indeed a useful one. Thinking a good PM only uses that one bit of info means you have known really bad PMs in your work.

The product managers know this, but the higher-ups (who control the money flow) tend to get tunnel-vision on a single metric, and that metric is often going to be the one most understandable to a layman. So that's the one that gets juiced.

> killing ALL telemetry is bad for everybody.

I don't think that a good case has been made for this. Telemetry does bring benefits -- it's cheaper than the old way, and captures useful things that couldn't be captured without it. But it also comes with some costs that are far from insignificant, especially for users.

I don't see any real reason to think that on the whole, telemetry has been beneficial for users. I think a stronger case can be made that it's beneficial for software makers.

>killing ALL telemetry is bad for everybody.

If you want my data you can fucking pay me for it.

Otherwise, GTFO. I'm here to use your product, not give you data that has nothing to do with your product without my permission.

but telemetry HAS to do with my product, not you.

Your needs are none of my concern.

If you want something from me, pay up.

You cannot separate the two. Telemetry is primarily about the user. What features do they use? What is their pattern of usage? Etc.

So it's about the user as well as about the software. If it weren't, the only telemetry that would be of any actual value to you would be crash reports.

How much work is it for firefox to maintain their text encoding conversion? How much work is it to maintain the menu entry that lets users control it?

> I agree that vanity telemetry is bad for users, but killing ALL telemetry is bad for everybody

It's good for my privacy

I think that 90% of telemetry is useless (eg. how often a feature is used says nothing about how important it is), but telemetry related to errors and crashes is extremely important.

My experience with shipping software is that if 50 people encounter a bug, maybe 1 of them will click the "feedback" button to report it. If you don't have a feedback button, it's less than that. If you don't have a technical audience, it's even less again.

So if you want to ship high quality software, then automatic error reporting is essential.

> how often a feature is used says nothing about how important it is

Exactly. But that doesn't mean it's not important. If you think something is important, but it's not being used, it's something to look into.

On a related note, exploiting security vulnerabilities is often non-deterministic. There might be thousands of errors or process crashes before the exploit is successful, and if at least some of those get reported to your telemetry servers it becomes a goldmine for discovering zero-days which are actively being exploited.

Microsoft has people who hunt for evidence of zero-days in their telemetry, for example.

I think opt out error reporting that prompts the user to send a report and lets them review it first is very worthwhile.

I would be much more inclined to click a "report bug" button if software didn't put itself into an adversarial position by engaging in shady telemetry tactics.

But, especially these days, you pretty much have to regard all software as being a bit hostile.

I don't bother to report bugs because I know I'm just speaking to /dev/null.

If you want to ship high quality software, listen to your customers and make it clear their words really matter.

No, telemetry does not count; that only reinforces I'm speaking to /dev/null.

It’s telling that you have products absolutely jam packed full of telemetry, yet their “community support” forums will have years old threads with hundreds of replies about specific unresolved bugs and small tweak requests that go completely ignored despite getting bumped weekly.

This disconnect is common, stark, and very obvious to all. I've long been genuinely amazed that companies allow this state of things to persist. It makes them look really terrible, and certainly doesn't help to make the case that users sacrificing some privacy to the software maker actually results in benefits to the user.

And to add to the general disdain for users, many of the larger company support forum are staffed exclusively by volunteer “experts” that aren’t even associated with the company or have a direct line of communication to it. Heaven forbid someone working on the product actually sees what their users desperately want them to know.

You’re thinking too simply.

I think you’re correct in that the telemetry is to show managers, but once it’s in their central log aggregator, there’s another team that then uses that and other aggregated logs from other teams to make business decisions

Metadata is useful in building profiles on people.

Master Data Management pipelines and systems make this processing quite easy.

Third party companies like Accurint will pay for data.

Very rarely is it just how much time are you spending in various features.

Reply on news.ycombinator.com