It is impossible to get proper usage feedback from your programs without being swayed by the vocal minority community.
We always find posts online on how crappy software is, but how can software improve if the majority of people actually using the software don't give feedback at all?
I’m going to reply strictly to how you’re framing your response.
I believe that “telemetry” Was only “bastardized” because companies have taken Privacy Policy language to the extreme. I’ve read every single Terms of Service and Privacy Policy I’ve ever had to agree to. The language in them has always been dense but since the age of the ad driven internet, the scope creep has been intense.
Most people will not read these documents, so they will assume that they are all the same and that all of them are equal to the worst version you hear of. That’s why “telemetry’s” meaning has changed. You cannot blame the general populace for this. The root cause is how bad acting companies have added language to Privacy Policies such that:
* just about all data == telemetry
* telemetry can be used for “other purposes”
* privacy policies can change without notice
* you cannot opt-out
These are the worst case scenarios. Do not blame people for assuming the worst. Blame those who have changed the rules of the game such that assuming the worst is the default.
Ah, so blame lawyers, product managers, anyone who releases free as in beer software, the open source contributors making all that software everyone else is stealing, the customer’s unrealistic expectation that someone else should pay for things, engineers for lacking the affect to care about any of this or to empathize even one iota with the real economy, product designers for being outwardly countercultural while authoring the literal dark patterns that get the telemetry in the first place...
No. I hope this burns to read.
Software never improves because incompetence is the norm. Not because we didn't have a magical data collection unicorn available.
Competent software companies ran user panels, had decent quality control, didn't steamroll their communities, didn't market loudly over user dissent and certainly didn't shut down their issue tracking to even their top tier partners.
That was Microsoft 10 years ago. That is Microsoft today. But you know, Telemetry solves all these problems doesn't it? No.
The real answer to your question: ask and listen. People will gladly tell you. Do not just take the data otherwise you end up with a set of poorly selected metrics which do not represent user opinion and a lot of pissed off customers who don't want to or can't tell you due to legislation and data protection.
Edit: to back up my point, Microsoft closed down Connect with over 30 issues open from me and our account manager left to go and work for a competitor because he was fed up of dealing with that kind of shit and couldn't even get basic issues from a Gold partner actually escalated to anyone. We had a ticket open for 7 years against clickonce where IE9 broke it completely for about 15,000 users.
As for community steamrolling, this is a repeat of this one again: https://github.com/dotnet/sdk/issues/6145
Edit 2: I have removed some irrelevant stuff. This story goes on forever. I have so many anecdotes from dealing with MSFT pre and post OSS glory that I concentrate all my effort on staying as far away as possible.
And likewise simply listening to a vocal minority via "ask and listen" is not a silver bullet.
So, you're both right, you need both to make informed decisions.
I've built and supported software with 80k end users and did that effectively single-handedly.
I'm sure a large corp can do the same if it sacrifices a bit of bottom line...
This sounds great in theory - harder to do in practice. Often what ends up happening is the only people who will share their time with you are the ones who want something specifically changed for them. Thus my point, it's effective, but it's not a silver bullet.
> I've built and supported software with 80k end users and did that effectively single-handedly.
And plenty of businesses have used Google Analytics, Mixpanel, etc. combined with the aforementioned technique.
TL;DR - The two strategies are not mutually exclusive.
Sure, but lots of things are hard. That doesn't mean we should all be happy about software phoning home without the consent of the user.
And therein lies the root of "telemetry" — the SV bubble's lack of interest, lack of effort, and fear of interfacing with the wetware.
Telemetry is easy. Talking to people is hard. Too bad.
The user spent an hour fiddling with settings. Is that because they love the new settings toolbox? Or is it because they were very frustrated with it and couldn't find what they were looking for?
For example, I have an XBox One controller. It used to pair fine via Bluetool. It still pairs fine with my Mac. Other stuff still pairs fine with my PC. But it just won't work after a Windows update.
What is telemetry going to tell them that they don't already know from the forum? Maybe the scope, but it's fuzzy. Some users might give up after one or two tries. Some users might be using the "Add hardware" box several times in a row for different reasons. Telemetry isn't a magic insights thing. It's difficult to get right, and to draw the right conclusions.
One thing's for sure, telemetry's cheaper than QA-ing updates properly.
You could argue that telemetry should then only exist in your beta channel or testing builds, and some developers do that. It's silly to argue everything can be caught by your QA team, that is simply not true for online services. In the past projects I've worked on have had long-standing bugs that took weeks of ongoing effort between both our paid QA staff and customers to finally identify reproduction steps, at which point we were able to examine telemetry for those reproductions and fix the problem.
"This is happening for 100% of users with the B2C3D4 controller and is likely a driver bug, but has happened only twice on the C3D5 device, both for the same user - likely a hardware failure"
What will happen if they won't? Are you going to switch to different Windows or Office?
Microsoft have a team of people who look at crash reports, and categorise the results (see for example https://devblogs.microsoft.com/oldnewthing/20050412-47/?p=35... , just a quick thing I found).
Having the ability to track the crashes of millions of machines, to find patterns in which drivers are crashing which applications, seems like an impossible thing to replace.
Being unable to opt out and the default being opt-in is what is unacceptable.
Nevertheless, consent is still paramount. Removing consent on the basis that most users are incapable of being informed is a poor excuse.
Most people aren't really stupid, rather bad software make them look stupid and bad tech support shifts the blame to the users.
There are dozens of ways to get feedback from users, but most of them require the company to pay for them. Companies are as bad as your average Joe in this fashion; why pay when they can simply pretend that privacy and data protection laws don't exist and just take?
I said it elsewhere in this topic: My purchase of your software does not give you the right to exfiltrate data from my system. You're welcome to ask for it, or to pay for it, but in no way is it yours to just take.
Additionally, mandatory telemetry was in no part of the purchase process (well, neither were ads). Instead, it's in a completely separate clickwrap 'agreement' (that's subject to change without warning) that's only made clear when you're install the software.
One thing you're omitting: windows xp home was only $199, just like with windows 7. vista seems to be an outlier in terms of prices.
Also, all the prices you've listed are for the full version (ie. not upgrade). The upgrade prices are much lower, and are in line with the current price for windows 10, which does not have separate pricing for upgrade vs full. You can still interpret this as a price drop, but most people get their computers through OEMs, and so aren't paying retail prices. I suspect people who build their own PCs also tend to not buy legitimate licenses. Also, AFAIK the checks for the "upgrade" version aren't particularly rigorous. You could install a pirated copy first, leave it activated, then do a clean install, and it wouldn't complain about the licensing.
"Surveillance" has already been thoroughly scorched by the reaction to the US government's broad violation of the Fourth Amendment. Talk to anyone in the defense industry. Now, rather than internalizing that many people don't want to be spied on, product managers are deciding to double-down on surveillance but use euphemisms. Don't whine when people respond by scorching the new word as well.
They could call it "logging" next year and we'll start tarring and feathering the word logging. The issue is not the word, but the behavior it represents.
Gathering detailed usage behavior of applications must be made optional to the user, as doing so without opt-out is decidedly hostile to the user's privacy.
You say it is impossible to get feedback from programs without a vocal minority being dominant. This is untrue on the surface since providing the option to disable telemetry removes a minority of users, probably a set highly correlated with the vocal minority you are concerned about. So if they have something to let you know, they'll probably contact you directly—the old fashioned way.
As others have pointed out, there's also no compelling evidence that software is better since the advent of widespread telemetry. Telemetry so often lacks context. You don't know what the user was trying to do; only what they did. Just because a feature is used a lot, that doesn't mean it's a good feature. It's merely what users have found in your software that approximately does what they intended. What's unseen, what can't be seen, is intent. You can't (yet, thankfully) measure the reluctance or happiness of the user as they pressed the button.
Even when working ideally, and observing willing users, telemetry has a nasty habit of navigating products to local maxima at the expense more quickly finding significantly better options.
Which almost nobody does, unless they are either completely unsatisfied with the software or completely in love with the software (just like TripAdvisor feedback)
Good point. Better build in a surveillance engine to spy on your users. /s
Data mining is about using the customer as product, not improving yours.
Hire people specialised in Q&A, send someone over to big customers to observe how people use your product and ask/pay a customer to interview a random selection of the people using the software.
Or just stalk your customers. It doesn't require human interaction, it's cheap, and probably not illegal enough to actually get fined. Who cares about the actual opinions of your customers when you can just interpret some carefully selected dashboards, right?
Because they're not asked.
There's a reason that gathering feedback from IVR and web sites is a multi-million dollar industry. It asks people, and they respond.
There's a sports quote about "You miss 100% of the shots you don't take." The tech industry has to learn that it misses out on 100% of the feedback it doesn't ask for.
This seems like an issue with how the product presents that choice or ability to users.
For example about half of the people who make it to the end of my video courses write in and give me feedback based on a 7+ question form I ask them to fill out at the end of the course. I ask very specific questions that could likely be answered in 1 minute or less each. Out of thousands of submissions, a huge majority are positive.
I don't ask for feedback or anything early on, and do my best to avoid giving someone extra "work" to do. I present it in the form of "hey, I see you made it to the end of the course, your feedback is helpful so that my next course is even better aligned with what you want...".
Folks are happy to provide feedback in that case.
I'm one of those people who typically turn telemetry off when I can because in a lot of cases it's not clear on how it'll be used unless I read a 100 page TOS or I simply don't trust the company is telling the truth on how it'll be used. I shouldn't feel like I need to diagnose my network traffic with WireShark just to double check a company isn't harvesting usage stats about an app I'm using.
Telemetry isn’t about better user feedback, it’s about cheaper user feedback, even at the price of quality and ethics.
Additionally, What's to stop Microsoft from turning their Telemetry data into sales or marketing data?
Software, especially Microsoft software became much worse in the past 5 years or so despite their heavy push on telemetry. Therefore I don't think telemetry is a magic bullet that will make software better.
- Uncontrollable automatic updates on Windows 10
- Forced reboots for updates even if the computer is in the middle of a long-running task
- Ruining of Windows search to the point it is basically unusable
- Removal of Paint and deprecation of the Snipping Tool
You know, just to name a few.
Oh God don't get me started. Few days ago I was in the middle of a videoconference with some important people, when suddenly my screen went blue, flashed a Windows spinner and the word "Restarting", and boom. It just rebooted. With no warning, despite me being on a videocall, on Microsoft Teams of all things! And within the "active hours". How this behavior is acceptable is beyond me.
Why? Because I set my dual-GPU gaming rig to sleep every night, and Windows feels the need to wake it up about an hour later, fail to do updates, and then leave the machine on the rest of the night, even with auto-sleep set for 30 minutes.
I noticed the problem a few weeks later, and by then my electricity bill had increased by 20 bucks.
Corporate-forced updates, however...
Old Microsoft software had a simple toggle switch for this.
By default, windows 10 lets Microsoft engineers remotely log into your box and browse your filesystem. They say they only use it for diagnostic purposes, but I don’t see how that could be true unless they’re in violation of US law, which compels them to give the same access to law enforcement.
I’m not sure if you can opt out of that (or whether the opt out would survive a warrant).
I switched away from windows over this sort of thing. There were dozens of other objectionable things they were caught doing, and efforts to build windows 10 “decrappifiers” made it clear they were adding new telemetry every month, and laundering the data through sock puppet domains.
I'd like to see a reputable source for that claim.
The GP comment seemed to imply that Microsoft engineers could log in remotely without your knowledge or consent.
This is correct, they AFAIK need a password/acceptance from the user, that's the proviso, but the original comment didn't say "without anyone knowing" (and as it's closed source none of us knows for sure). Their quoted claim is true it's just of very limited value.
This whole thread is going nowhere.
The first question should've been "yes, but can they do it without a password or user-acceptance". The answer is "we don't know" AFAIAA.
> Full: All data necessary to identify and help to fix problems, plus data from the Security, Basic, and Enhanced levels.
In the Windows 8 days, they claimed that engineers couldn’t silently pull individual files from machines without managerial approval. I can’t find the source. It was some old news article with an interview with a Microsoft manager.
Anyway, “All data necessary to identify and help to fix problems” pretty clearly implies they can pull whatever they want as they debug. I don’t see how they could implement that without exposing customers to warrant requests.
This page outlines everything additional they recieve on the Full setting.
> In the Windows 8 days, they claimed that engineers couldn’t silently pull individual files from machines without managerial approval. I can’t find the source. It was some old news article with an interview with a Microsoft manager.
I recall reading something similar, but for Windows 10. AFAIK it said that engineers diagnosing a difficult problem can select a group of machines to receive raw telemetry from, after getting permission from managers + microsoft's privacy team. I have a feeling it was for insider builds only though.
for example, i have seen videos of ms word and ms visual studio, on old pentium, load instantly with a splashscreen flashing by. i was truly impressed indeed.
For example, Word 6 on my 386DX with 4 MB RAM took some time to launch and I had ample time to admire the art that went into the splashscreen. On 486DX with 16 MB RAM, it did flash by.
The calculator used to be fast and load instantly, now it's one of those UWP monsters that even asks you to rate it in the Microsoft store...
I don't recall hearing about updates bricking machines or causing data loss at scale back in the Windows 7 days but it seems like that is now a relatively common occurrence, amplified by the fact that you can no longer hide/defer updates on consumer versions of Windows. I think the firing of their QA team and delegating the work to unpaid "insiders" and telemetry might have something to do with this.
The new Settings UI is absolutely disgusting both in looks and information density and is a clear downgrade from the previous version.
I can go on and on. I would sympathize if they were pushing the boundaries of software engineering but what we're talking about isn't groundbreaking - these are problems that were mostly solved a decade ago and Microsoft intentionally backtracked on their progress by the looks of it.
This could also be explained by user expectations for software rising but quality of Microsoft code remaining constant. In the past users may have written off such events as 'just the way computers work sometimes' but perhaps now users realize that computers needn't be so unreliable.
I disagree. Evidence that supports MS code quality dropping includes a significant amount of users hanging on to Windows 7 with their cold dead hands even post years of MS marketing, arm twisting, GWX updates, and EOLing Windows 7, with users paying for ESUs via Ask Woody vendors and/or that 0patch tool.
I myself moved to Windows 8.1 and from there am hem-hawing on whether to use KDE Neon or Linux mint XFCE and just leaving behind Windows except for the air-gapped Windows 7 VM I will no doubt need for things like Anime Studio. I will not allow Windows 10 (outside work devices) on my home network.
(Maybe for Centaurus aka Courier Jr...but I'll put it on the guest wifi and make a bunch of throwaway accounts for it. )
Of course what constitutes an improvement or regression may vary by person
How long does it take after boot for disk I/O to stop if you've got a 5400 rpm hard drive? It's maybe a few minutes after login on 7, and I've never seen it stop on 10.
Why does the calculator take seconds to start? Why does it ignore keypresses when it has focus?
How many clicks does it take to set my IP on an isolated network with no DHCPd? How many different contol interfaces will I see on the way?
On Old Edge (I haven't tried Chrome based Edge), why does the stop button sometimes not stop until the page finishes loading over several seconds? Why do the back and forward, and url navigation interactions queue up in that case?
>How long does it take after boot for disk I/O to stop if you've got a 5400 rpm hard drive? It's maybe a few minutes after login on 7, and I've never seen it stop on 10.
Unsure what you are saying. Windows is the only OS to write to disk after boot? I don't see anything hitting the disk after a few seconds.
>Why does the calculator take seconds to start?
Instant
>Why does it ignore keypresses when it has focus?
unable to reproduce
>How many clicks does it take to set my IP on an isolated network with no DHCPd
4
>On Old Edge I don't use that particular piece of software so I can't tell you.
>Instant
It's "instant" to show up, but you got to wait one to two seconds for the splash screen to disappear, whereas the windows 7 showed up instantly and was usable immediately.
And I would say most software today is incredibly buggy. Almost every major piece of software I use now from large, well known companies is just rife with bugs.
You're right, it has been bastardized.
We used to call it what it actually is: spyware.
Usage data is a better name for what gets returned in most telemetry.
I would much rather genuine telemetry is supported so it can be reported in a way that doesn't allow a "you agree to this" hook to be used for both innocent telemetry and problematic telemetry.
Throwing your toys out the pram at any usage data going back is harmful not helpful. It will mean that the bad actors will win because they'll be the only ones who have the data to improve their products.
Or everything will go saas so you'll get desktop "software" that's nothing more than a shell making HTTP calls back to a backend so all the usage gets tracked there and it'll be slow as shit for the priviledge.
I don't know where you heard that but that is absolutely untrue. There have been anti=spyware apps dating back to the late 90s and before, and they essentially AV scanners with a slightly different focus
The New Oxford American Dictionary defines it as "software that enables a user to obtain covert information about another's computer activities by transmitting data covertly from their hard drive"
Without consent / explicit opt-in, this describes most telemetry perfectly.
There are tons of other products in this world that don’t rely on surveilling on their users’ every single move to solve these problems and improve the product.
Besides, there is a huge assumption that the data gathered will be used only to improve the value of the product for user. Considering the fact that the two sides of the market, the buyer and the seller, is in adverserial dynamics when deciding the price point, it is irrational for the seller to not use this information to actually increase their profitability. They might as well, and indeed do, use telemetry to cut their costs without moving their price, for example in the form of cutting support from existing but under-demanded features, allocating resources that could bug fix an existing feature to features of the next product, to engage in extractive behavior such as upselling new products etc. All of this shortchanges the end user.
Some problems get reported for years without any fix in sight.
Like, stop the telemetry nonsense and read the darned forums, starting with your own?
There are
1. well-behaved programs that do what I tell them to do,
2. programs that surreptitiously send my data to an external party without being told to.
I thought we were calling the second category of behaviour "telemetry" (or "spying" when we're not being polite). If not, what is the correct terminology?
Those users participate after providing consent, and are paid for the feedback they offer.
Your comment could be seen to imply that software developers are entitled to behaviour data from users and organizations without their consent (or even awareness).
The best way to understand and improve a product if you lack the ability to gather telemetry or user feedback is to use it yourself and identify areas for improvement.
You won't hear about it because nobody will respect that setting in any meaningful way, so everyone will disrespect it.
Needless to say, authors weren't thrilled about it
This is how it was done before telemetry was possible. Good software was made back then. All our software today is built with or on top of software that was developed without any telemetry.
Today it's much more common to see companies / product managers using telemetry, instead of their brains, and making bad decisions as a result. There are always confounding factors, and they usually dominate. Collecting numbers is easy, collecting the right numbers is hard, product teams don't have time for that. Telemetry ends up mostly being used for excuses for bad decisions.
Is there a particular company size when you suddenly can't collect telemetry because of the privacy implications? Why aren't they allowed to compete using the same tools as everyone else?
Microsoft's telemetry practices SHOULD DEFINITELY be more heavily scrutinized than other companies.
Your website uses Fathom which focuses on privacy but doesn't have any third party auditing of that claim (searched their site for "audit", no results). Why do you trust them to do the right thing with my browsing data when I visit your website? What are you using the data for, to improve the site and create content more people are interested in? Why can't Microsoft do that do to improve their applications?
I don't say all of this to downplay the importance of knowing that things are collecting telemetry and shipping it off somewhere but we can't just have a blanket statement of "you can't do it once you're big enough to abuse it" be our guiding policy either.
The problem is on the business side of software.
I'm curious how many people here decrying telemetry use Google Analytics or similar on their websites or their employer's websites and think nothing of it. Or have ads on the websites. Or literally just about any 3rd-party Javascript. Or literally just about any 3rd-party resource of any kind.
I'm seeing a lot of pearl clutching.
As it is now, they don't deserve it and the dark patterns being deployed ought to be criminal.
It is a word that has been cleaned up and packaged to minimize discussion and confrontation.
There are plenty of words like this.
Even "justice" or "freedom" are ambiguous and allow multiple meanings. Two people can talk about justice without actually meaning the same thing.
One wonders how we ever came this far without looking over the shoulders of users.
If they want to understand how people use their products they can perform usability research. If they want me to participate in such research, they can offer to pay me for my time. Snooping on me with embedded spyware is not acceptable.
1. Let's collect information on every URL, including every distinct pornhub URL, a user visits.
2. Let's collect information on how many times a user browses pornhub.
3. Let's collect information about how and when a pornhub site crashes our browser.
4. Let's collect information about how and when our browser crashes - without submitting the website that crashed it, just the internal info such as call stacks.
Only 3 and 4 are are about fixing bugs, and while 3 would maybe make it easier to reproduce issues by knowing the exact website that triggers faulty behavior, you end up with a lot of information about your individual users that could be abused.
1 and 2 are about potentially making your product better by being able to tell what websites (or features, etc) users are actually using and focusing on improving those features, or figuring out why they do not use the other great features (e.g. bad UX?). However, this comes at the great expense for user privacy.
What really makes me mad about microsoft is how little they are telling you what kind of telemetry they actually collect at what configurable level. I do not know if they do 1 or 2 or 3 or 4 or any combination thereof by reading their privacy notice. The privacy notice says (or used to say?) that they can even transmit your files around for telemetry/bug fixing purposes if they feel like it. And they are very unclear about how the data is processed and retained and how long such a data retention is.
I'd be happy to contribute some telemetry depending on what it is. But they are refusing me configure let alone tell what they collect, why they collect data and if they got proper processes in place before they add certain types of data collection, so in this scenario I'd like to opt out completely. But I cannot because not an option. And I genuinely hope they get slapped with enormous GDPR fines for it.
As a counterpoint, Firefox - while not perfect
- There is an opt-out/opt-in for telemetry/crash reporting
- They have a privacy document describing what they collect and when: https://www.mozilla.org/en-US/privacy/firefox/
- They also openly describe their process, which makes me more confident in them talking this seriously: https://wiki.mozilla.org/Firefox/Data_Collection
- They show me what telemetry exactly they collect: about:telemetry (While it's certainly hard to figure what all that data means, it being there in the first place means that third party experts can easily access it and evaluate it)
This telemetry is mandatory. Users are not permitted to opt-out.
This means that the average Joe must understand what information is being sent and the submission must be opt-out rather than opt-in.
It's possible to do telemetry in a way that does not violate this law, but that means you're not allowed to do more than basic aggregates that can just as easily be collected on the backend. A collection of installed software can easily act as a globally unique identifier because every PC installation is different, so even training a "recommended for you" system that just finds other software that people commonly install along with a certain package must already be opt-in.
Because of the requirement that combining data with other databases (server logs, for example) must not allow for unique identification of a single person in a data set, you really should only be using opt-in with informed consent. "We may use data as described in our privacy policy" is not informed consent, "we will send a list of all software you install, remove, change or update, when you do so and in what way you did it" comes close. Microsoft is severely lacking in this aspect.
The developers add insult to injury by also lying about the theoretical ability to opt out.
https://www.apple.com/privacy/docs/Differential_Privacy_Over...