Telemetry required? Ask users first
dev.blog.documentfoundation.org
dev.blog.documentfoundation.org
Despite a 100-fold increase in hardware power, and languages/frameworks that handle 80% of the workload a developer 15 years had to think about, applications these days have fewer settings, open and run slower, and improve far less rapidly than anything written in C/C++/Java from 1990-2005 despite phoning home with novels worth of telemetry every few minutes.
Monitoring network traffic and DNS requests through a firewall and pihole, it’s downright hilarious that it’s the worst products that top the charts in both, even when being online is not a necessary component of the software. Meanwhile the software that just works, and works well, might check for an update once a month, and the changelog generally reveals new useful features and mentions performance increases.
Yes, but also no. If my resources are not infinite, knowing that an "expensive-to-maintain" feature is used by just 1% of users, it is an incredibly important piece of information when I'm gonna have to decide where/how to recalibrate my (team's) effort.
Product managing is also this: I have to know what works and what doesn't. Maybe I will use that knowledge to understand how to increase the 1% to 5% with a specific business case. Maybe it won't make sense and I will have to decide if I can keep it or not.
I agree that vanity telemetry is bad for users, but killing ALL telemetry is bad for everybody.
You can't use a blunt usage metric to tell whether a feature works or not. It only tells you that the feature is popular for some reason. That reason might be that it works, but often the reason people use a feature a lot is it's shoved down their throats.
It's a vicious cycle. Say you are Product Manager for Feature X. Your bonus/promotion/career progression depends on some vanity usage metric for feature X. The more monthly actives or new users for feature X, the better it is for you. So, naturally, you're going to do whatever you can to funnel users at that feature. Stick it prominently on the main screen. Highlight it in flashing red. Spam the user with notifications begging them to use it. Make competing features low-contrast gray on gray. Whatever you can get away with. Then the next monthly metrics report is in: The feature is now being used by +15% of users since last month! Incredible! Your bonus buys you a Tesla. Management's conclusion is: "This feature really works and users must really like this feature." So, the product focuses more on developing that feature.
Now scale the above out to every feature in the app, each with their own competitive and motivated Product Manager. This is how we get apps that are all-telemetry dumpster fires and with more and more flashy features fighting each other to grab your attention and fewer and fewer useful 1%-features.
I've personally witnessed meetings with multiple furious product owners literally shouting at each other about where their feature's button gets placed on the home screen because that placement made the difference between 5% feature usage and 25% feature usage. Where was the end user in all these discussions? Not present. End users are to be milked for metrics gains.
I never said that was the only source of information, but indeed a useful one. Thinking a good PM only uses that one bit of info means you have known really bad PMs in your work.
But knowing that only 1% of users use this feature is not enough to make that decision:
- maybe 1% of your users only use it, but they are the ones that pay the most money
- or it's the most prolific users so it generates half of your content
- or it's a niche feature used by power users, but those users are the ones bringing other users, introducing you to companies and doing free support
- or it needs to be used by only 1% of the users by design, and it's enough to be useful (E.G: bug report form)
- or your report it wrong, it's one percent of all accounts, but 10% of actually active users
etc.
Most telemetry will not tell you that of course.
So ask for it.
All data will be biased and incomplete anyway, so at least acquire it in an ethical manner.
Telemetry has to be designed as well, to piece as much as you can, and then you complete the full picture with other sources. Maybe not everybody does this, but I don't think that this practice should be "shocking" for anybody.
Why...? If that's the case, it sounds like the problem is too little telemetry, not too much.
Since most telemetry are optout and don't bother about giving users access to their data, they can't legally do it.
That's my point really: useful telemetry is really, really invasive. So just ask consent, and accept most users will tell you no.
If you chose what's convenient instead, you will not only have a mixed bag of data, but also you have decide to basically force yourself on people, in an age where a part of your userbase is getting upset over things like using the wrong pronoun on them.
I tell yes to Firefox and VLC to report on my usage. It tell no to Google and Microsoft. They want their cake and it it: behave badly, and then have we trust them.
They already need to agree to Terms of Service when paying and starting to use the software.
Explicit consent needs to be standalone and refusing it cannot stop the user from doing anything that doesn't actually really require said consent.
I'm not even talking about what the law says, because the law is very inadequate on these matters (in the US, anyway). I'm talking about, from a common-sense point of view, what "consent" is.
In my view, it can only be counted as "consent" if I have been informed exactly what it is that I'm consenting to, and I am asked for that consent.
ToS don't count for a number of reasons, starting with the fact that nobody reads them, it's unrealistic to expect people to read them (because everything comes with lengthy ToS documents and if you really read them all, little time would be left to do anything else), they are generally difficult to properly understand if you're not a lawyer, and they are are usually intentionally vague of this issue -- which means the "informed" part of "informed consent" isn't satisfied even if you do read them.
I think the Principle of Least Surprise should apply here. If software is going to do something that users won't notice happening, is intrusive, and isn't pretty obvious from the nature of the software, the right thing to do is to tell the user about it and ask for permission.
Software that doesn't do this is adversarial to the user.
I personally think it is because program managers rely on telemetry instead of stakeholder research and industrial design, either because they don’t know better, or because management thinks it will cut costs.
Besides, telemetry is a feature that 0% of users use, so it should be the first to be cut.
But there's a trade-off there—as, ideally speaking, most software has become easier for most people to use mostly satisfactorily right out of the box, it becomes all the harder to adapt if you're one of the people for whom it doesn't work mostly satisfactorily, since there's no real culture of design around the idea that a user might want to use software in the way that they, rather than the designer, intended. Even if it's fair to write off those niche users, there are still the most users for whom the software works mostly satisfactorily, but can't be modified to be perfect, again because the culture of user customisability just isn't there.
That seemed to be a reasonable compromise to me.
That is to say: There is no such thing as the Average Man.
Make something that will generally fit most people, but then also add enough customization features so each man can tune it to his liking. There is a reason car seats and steering columns are adjustable.
Focusing purely on out of the box experience looks good for quarterly metrics (subscriber growth with low initial churn) but you end up with a population of users that are desperate to switch away.
At that point, you are one viable competitor away from being screwed. Worse, as users defect, telemetry will tell you that usage of the features associated with switching is declining, so you’ll double down on stripping out the functionality that you should be investing in if you want to defend your market share.
See also: Firefox.
I don't think that a good case has been made for this. Telemetry does bring benefits -- it's cheaper than the old way, and captures useful things that couldn't be captured without it. But it also comes with some costs that are far from insignificant, especially for users.
I don't see any real reason to think that on the whole, telemetry has been beneficial for users. I think a stronger case can be made that it's beneficial for software makers.
In practice things rarely get that far, thanks mostly due to the complete farce that is "data driven design". Such an approach to design would naturally lead to the axing of every unpopular feature if it were followed mechanistically, but in reality "data driven design" actually refers to the practice of mining to support whatever you (the designer/programmer/manager/etc) have already decided to do, and the data is ignored for anything else.
It actually works like this:
I form a subjective judgement against some feature which I dislike (maybe because it's a pain to maintain or just because it doesn't suit my preferences for any reason.)
Then, I look in the telemetry and start making charts of things until I can find some combination of the data that seems to support my prejudice against the feature. I bury all the presentations of the data which failed to support or even refuted me, I pretend I never saw them.
Then I write up my report claiming that the feature should be removed for objective scientific reasons, and my fancy charts and graphs prove it! It's totally not just my subjective prejudices driving the design, no Siree...
Besides the privacy/consent concerns, this is why telemetry is bad for users. It devalues dialogue between the users and developers and replaces it with pseudo-scientific arguments based on nothing but the developer's preferences. That's what it's really for, an excuse to never talk with the users.
I also wonder how much of a chicken-egg effect there is on the trend of developers hiding niche/power-user features that they would prefer to sunset deep into settings menus and then that setting seeing negligible use.
(e.g. is the 1% feature a feature that is pivotal to getting new users?) (e.g. is the 1% feature a feature that is key for a specific audience within my total addressable market?)
But you have know from somewhere that the 1% is, in fact, 1%...makes sense?
Only 1% of mobile phone users need 911 regularly. Lets remove that feature.0
If this is the type of PM you are accustomed to, fire them. Because that's a dumb argument.
> If this is the type of PM you are accustomed to, fire them.
People's expectations about how projects treat 911 are born from those hard requirements. If it didn't have those requirements, you would find that every PM is the kind of PM that would let the service suffer. Maybe not scrap 911 altogether, but certainly deliver a less robust and reliable 911 service than they do now.
One time there was this new mode, for like 1-2 years, of 3v3 online. it was probably the most fun mode madden has ever had. it just needed a few tweaks. sensible stuff. But it would've been amazing!
but no. because it was difficult to get to in the menu system, fewer people used it and few people knew about it. so of course telemetry is going to say few people used it. but it had a TON of potential! a product owner made the wrong decision to cut that feature, and guaranteed they used misleading telemetry to justify it.
It's good for my privacy
If you want my data you can fucking pay me for it.
Otherwise, GTFO. I'm here to use your product, not give you data that has nothing to do with your product without my permission.
So it's about the user as well as about the software. If it weren't, the only telemetry that would be of any actual value to you would be crash reports.
If you want something from me, pay up.
My experience with shipping software is that if 50 people encounter a bug, maybe 1 of them will click the "feedback" button to report it. If you don't have a feedback button, it's less than that. If you don't have a technical audience, it's even less again.
So if you want to ship high quality software, then automatic error reporting is essential.
If you want to ship high quality software, listen to your customers and make it clear their words really matter.
No, telemetry does not count; that only reinforces I'm speaking to /dev/null.
But, especially these days, you pretty much have to regard all software as being a bit hostile.
Microsoft has people who hunt for evidence of zero-days in their telemetry, for example.
Exactly. But that doesn't mean it's not important. If you think something is important, but it's not being used, it's something to look into.
I think you’re correct in that the telemetry is to show managers, but once it’s in their central log aggregator, there’s another team that then uses that and other aggregated logs from other teams to make business decisions
Master Data Management pipelines and systems make this processing quite easy.
Third party companies like Accurint will pay for data.
Very rarely is it just how much time are you spending in various features.
The real solution here is to remove all optional code from the critical path - this not only includes telemetry, but also things like in-app tutorials, automatic updates, crash reporting... If some code is not implementing a core function of the application, it should be allowed to crash and the application continue working. This is a core principle of good software engineering and it needs to be talked about more.
After quite a bit of digging into the problem we eventually found that the iOS app was using a single error handler for all HTTP requests, including the one being made to query Status Page for any active incidents so they could be shown to users - our Status Page account had been erroneously disabled, which meant the requests were returning a 401, which the app interpreted as the user needing to re-authenticate. I'm not sure I've ever been embarrassed than having to report upstream that the cause of an incident was our solution for flagging incidents to customers.
Part of the bug here would seem to be this returning a 400-class status code: it should pretty obviously be a 500-. The most suitable standard code is 503 Service Unavailable.
401:"not authorized" seems exactly correct for status page saying "I know who you are, and what you want, but you can't have it because you chose to disable your account."
401 is only permitted if an Authorization header is missing or insufficient, and the server “MUST send a WWW-Authenticate header field containing at least one challenge applicable to the target resource” (https://www.rfc-editor.org/rfc/rfc9110#name-401-unauthorized). Presuming an account-specific status page endpoint, this cannot be satisfied; nothing the client can send will work, therefore it’s far more reasonably treated as a server error than a client error: the server has been instructed not to service these requests. “Service Unavailable” is clearly suitable. (Refusing the connection would also be reasonable, and have certain technical advantages for machine use.)
I maintain some small open source projects (which I wrote for my own benefit), and judging from GitHub issues and stars, get some use.
But I have absolutely no idea who's using the software, or which subsets of features they care about.
I'd happily put work in to improve things users cared about, but have no idea how to get that info (I don't know who they are so can't ask), or using telemetry (which I'm loathe to add due to its perception).
What I've seen some software (including some enterprise software) do -- and what I do with my own products -- is generate a human-readable text file that the user can email to the company. Then the user can be 100% certain of what data is and is not being reported. It also gives the user the opportunity to redact any data they are not comfortable sharing.
I think that is wrong, it increases awareness of something that I would not care otherwise, and should not by default
There should be more trust. Telemetry should be auditable, and companies should be more transparent with how they're using it and storing for how long
Users should be able to see everything being sent
Allow to reduce telemetry levels, all, erros, settings...
Allow users to reset user identifiers, reset at every installation, reset periodically
Use a trustable third party telemetry service
This is, really, the heart of the problem. There is very little trust. And the software industry, as a whole, is the reason. The amount of abuse we as an industry have engaged in, and continue to engage in, is remarkable.
Repairing that trust is something that will take years of good behavior. We as an industry haven't even begun the first steps down that path, though.
> Use a trustable third party telemetry service
Does one exist? And if so, how would you or I -- let alone an ordinary user -- know it was trustable?
I foolishly set up my microsoft account, github, and linkedin using the same email address, so now it is just a matter of time before Microsoft crosses the telemetry streams, and I get insta-cancelled for naming something “kil the pooppy crepers!”
Yes, Microsoft also created a social media profile for a kindergartner without any sort of notification or parental consent screen. I don’t see why they get a free pass to ignore all the “protect the children” laws.
It should be fine to collect "telemetry" as long as the data itself isn't personal and that it also cannot be tied in any way to a particular user.
At least, this is how I've interpreted the GDPR rules.
On the other hand if you tie telemetry to a user ID, fingerprint, unique device identifier, etc, then all the data is personal, and since it's not obvious and necessary, you also cannot argue its a legitimate interest (note that most GDPR popups arguing legitimate interest are absolutely not valid).
The answer is always "No" to telemetry in every case.
I can't believe how much BS we have to hear about consensual sex these days to the point people are suggesting contract before making love, and yet the virtual signalers in the valley can't still see the problem with forcing people to be spied upon even when told to piss off.
Maybe too many new web framework turn over also burnt out their common sense?
You could also show the same person, 10 minutes prior, telling their locksmith they should say "main-key" instead of "master-key" because that's inappropriate, otherwise they won't call them for more gigs.
And somehow the DJ is always playing "What is Love?" when it happens.
I would unironically require this simply because the legal liability risks are too bloody high otherwise.
Not that I would get myself into such a situation in the first place. In this day and age there are many other avenues of entertaining oneself without the ridiculous liability risks.
If they try to hide it? Huge red flag.
Or use a much better program that doesn't spy on you, (FOSS Software)
When has that ever been an option?
[0] https://medium.com/incerto/the-most-intolerant-wins-the-dict...
The telemetry is collected at time T, and your decision that you care must come before time T if it's to have any effect on the outcomes. After the telemetry is collected, and decisions are taken based on that telemetry, it's too late to opt in.
This reflect a common problem with humans that what they don't like is regret, but what we offer them as a fix is informed consent -- it turns out what they wanted to be informed about was really whether they would in future regret it, which we cannot know.
Now, as a democracy we "fix" this by just deciding that even if you refuse to participate in the process we don't get to claim that your interests don't matter. That feels like a reasonable price to pay for very important decisions like "Should we build a freeway across this land occupied by the people from this weird religious sect?" but it feels way out of proportion for something like, "Should I add more animated Emoji reactions to version X, or should I work on enabling the text-to-speech feature in group chats as well as one-to-one ?"
Laws like GDPR can help organisations stick to collecting only what they actually needed to know and not just "Everything because why not" - GDPR is why not. Did you just collect the full name of every user? You'd better have a damn good explanation of why you did that. Whether user has <4GB, 4GB-8GB or 8GB+ of RAM? Very easy to justify.
I believe this is a better choice for developers and users.
Your test harness should cover edge cases. Invest in that, versus the externalized cost of collecting telemetry.
Your users are not your labrats. Quit pushing your testing costs on to them.