Aggregating request statistics server-side unless you're only generating those requests to spy on what I'm doing on my computer is more like the not-spying you're talking about.
You’re making a distinction that puts you in the privileged judge position of evaluating if a service is making requests just so you can spy vs what the app author’s might believe is a critical design feature in how they want the product to operate.
I agree, but once you cross the borders out to the internet, I'd say you need to stop seeing that as "Me sitting at my computer at home", because you're actually "on someone else's property" at that point essentially. And I say this as someone who care greatly about preserving personal privacy.
Watching people move their mouse and click stuff on “your webpage” is fucking spying. It’s in my browser. On my machine. Not running on your hardware.
Tracking what I do on my own computer doesn’t stop being spying because the program I’m doing stuff in can make network requests. WTF.
Well, I was mainly talking about network requests, which are quite literally served by "my hardware" when your client reaches out to my servers, and they agree to serve your client. I do agree that it sucks that browser viewports now also are considered "mine" from the perspective of servers, but you do have a choice to execute that code or not, you can always say no.
I don't think it's as much "this attitude took over", people saying that the internet is the wild west and warning you "browse at your own peril" has been around for as long as I can remember.
The attitude that’s changed is that in the 90s and 00s a program that sent information about what you’re doing that wasn’t necessary and expected for how it operates would have been instantly, popularly, and unequivocally labeled spyware by a programmer crowd. Now it’s normal and you get a bunch of folks claiming it’s ok.
Are some companies spying on you the way you say? Yea, probably. Most of us just want data to know what's working and what's not.
Telemetry would be more akin to looking through your customers windows each day and noting down how often they use the tools you sold them. Not cool.
Sometimes HN drives me crazy. From this thread you’d think telemetry is screen recording your every move and facial expression and sending it to the government. I’ve worked at places that had telemetry and it’s more along the granularity of “how many people clicked the secondary button on the third tab?” This is a far cry from “spying on users”.
Yes, admittedly, the first time you do these things, they're difficult, hard and you have lots to learn. But as you do this more often, build up a knowledge base and learn about your users, you'll gain knowledge and experience you can reuse, and it'll no longer take you weeks or months of investigations to answer "Where should this button go?", you'll base it on what you already know.
This is why telemetry happens, its faster, easier and more resilient to organizational turmoil.
I don't disagree with that, I was mainly talking about trying to deliver an experience that makes sense, is intuitive and as helpful and useful as possible, even in exchange for it taking longer time.
Of course this isn't applicable in every case, sometimes you need different tradeoffs, that's OK too. But that some favor quality over shorter implementation time shouldn't drive people crazy, it's just making different tradeoffs.
I think in terms of corporate teams this is the issue a lot of times, people just are not on the team long enough to build that knowledge. Between the constant reorgs, these days layoffs and other churn the no one puts in the years required to gain the implicit knowledge. So orgs reach for the "tenure independent knowledge base.
Usage data is the ground truth.
Soliciting user feedback is invasive, and it's only possible for some questions.
The HN response to this is "too bad" but it's a thought-terminating response.
I haven't been in a single metrics discussion where we didn't talk about what we're actually measuring, if it reflects what we want to measure, and how to counterbalance metrics sufficiently so we don't build yet another growthhacking disaster.
Doesn't mean that metrics are perfect - they are in fact aggravatingly imprecise - but the ground truth is usually somewhat better than "you clicked it, musta liked it!"
One: Building a good UX involves guesswork and experiments. You don't know what will be best for most users until you try something. You will often be wrong, and you rarely find the global maximum on the first try.
This applies to major features but also the most trivial UI details like whether users understand that this label can be clicked or that this button exists.
Two: Like all software, you're in a constant battle to avoid encumbering the system with things you don't actually need, like leaving around UI components that people don't use. Yet you don't want to become so terse with the UI that people find it confusing.
Three: I ran a popular cryptocurrency-related service where people constantly complained about there being no 2FA. I built it and polished a UX flow to both hint at the feature and make it easy to set up. A few months later I saw that only a few people enabled it.
Was it broken? No. It just turns out that people didn't really want to use 2FA.
The point being that you can be super wrong about usage patterns even after talking to users.
Finally: It's easy to think about companies we don't like and telemetry that's too snitchy. I don't want Microslop phoning home each app I open.
But if we only focus on the worst cases, we miss out on the more reasonable cases where thoughtful developers collect minimal data in an earnest effort to make the UX better for everyone.
That's because you don't understand your users. If you did, you wouldn't need to spy on them.
> you rarely find the global maximum on the first try
One never finds the "global maximum" with telemetry, at best a local sort-of maximum. To find what's best, you need understanding, which you never get from telemetry. Telemetry tells you what was done, not why or what was in the people's mind when it was done.
For what, precisely? As far as I know, you can use it to know "how much is X used" but not more than that, and it's not a "ground truth" for anything besides that.
What the ground truth usage data is completely ignorant of is that Netflix's copy is a crappy blurry transfer, and so I got dvds instead.
It's not that it has no value at all, it's just that it's stupid to know one thing (how to collect usage data) and think that is all you need to know and that that obviates all other sources of understanding.
If you wish to collect money from human customers, you have to be some minimum level of human yourself. Talking to your customers is not some icky hardship to be avoided and replaced with nice bash script.
Now, they could've done a better job by increasing the quality, but that's a further (and costly) optimisation.
Sure.
> Usage data is the ground truth.
Absolutely not. That's how you get "we buried this feature and nobody used it, so clearly nobody wants it".
> What I’m saying is that only relying on user interviews without supplementing them
I also took your "spend the weeks to months of expensive and time consuming work [...] Or you can look at the analytics" as a "either this or that proposition", where if we're making that choice, I'd go with qualitative data rather than quantitative, regardless of time taken. But probably it comes down to what tradeoffs we're willing to accept.
I think the root of the complaints here is prioritization. The things they care about are prioritized. Qualitative feedback is likely already telling PMs that something is wrong and really should be fixed, but other feedback has more data supporting it.
Precision isn't accuracy and all that.
It literally is. The network itself is always listening: https://en.wikipedia.org/wiki/Room_641A
The mere act of making a network connection leaks my physical location, the time I'm using my computer, and the fact that I use a particular piece of software. Given enough telemetry endpoints creates a fingerprint unique to me, because it is very unlikely that any other person at the same physical location uses the exact same set of software that I do, almost all of which want to phone home all the goddamn time. It's the metadata that's important here, so payload contents (including encryption) don't even matter.
the other crowd that pretends otherwise are larping or only have some generic open source project that only a handful of people use or they only update it every 6 years
Probably because there is no "truth" here, only subjective opinion, there is no "winning", only "learning" and "sharing".
I could ramble the same about how "people relying on data never shipped an enjoyable thing to people who ended up loving, only care about shipping as fast as possible" and yadda yadda, or I can actually make my points for why I believe what I believe. I do know what I prefer to read, so that's what I try to contribute back.
Have you asked those users or did you conclude this from (likely unrelated) "telemetry" you have gathered?
"Sure, you can spend the weeks to months of expensive and time consuming work it takes to get a fuzzy, half accurate and biased picture of what your users workflows look like through user interviews and surveys. Or you can look at the analytics, which tell you everything you need to know immediately, always up to date, with perfect precision." -> your analytics will never show what you didn't measure - it will only show what you already worked on - at best, it's some kind of validator mechanism - not a driver for feature exploration.
This kind of monitoring need to go through the documented data exposure - and it's a sufficient argument for a company to stop using github immediately if they take security seriously.
But I'd add that if you take security seriously you are not on Github anyway.
They'd probably call the issue you see a "craft" issue. Some PM is likely raising it. What happens is that leaders in big companies want perspectives based on data. You can go in with issues like yours but if you don't have clear data that shows significant numbers of users leaving, or users piling in, then you might as well not show up. People care about craft primarily will really struggle in these large organizations. That's not a good thing but how it is.
In large organizations, you'll see a lot of A/B testing or experimentation. Some of the worst decisions from a craft perspective are ones where they only look for "did this cause some kind of negative impact on numbers?" situation. If your feature is neutral (on abandons, uninstalls, or whatever negative outcome), then it can get shipped which overrides any qualitative question around "should we ship this in this state?". Doesn't matter too much according to these folks because it's not making things worse (in terms of numbers.)
There is probably more to explore in modern "product management" that's at the root of many of these problems. HN tends to focus on engineering but within large companies there is now a bifurcation and development of a field that forgets lots of PM was already invented.
How do the metrics you speak of capture these subtle, delayed effects?
I can guarantee you that the class of problem you describe has been discussed at the individual contributor level, so is known to some extent. Getting it from recognition to action is the problem. It is a huge lift to get some of these small things through the gauntlet to execution. Meanwhile, as you say, competitors with taste and attention to detail are building a better product.
This is very much a problem of large organizations. Those same PMs at a small company. If Google Maps was an independent company, the impediments would be fewer and priorities more aligned with building the best Google Maps.
> it’s more along the granularity of “how many people clicked the secondary button on the third tab?”
You don't see the contradiction here?
You can tell the difference between those who build businesses and those who simply use them.
1) Metrics lead to wrong conclusion. There is software which has extremely rarely used features, I need it once or twice a year only, but the ability is why I use the software to begin with. If metrics get too much attention such things are removed as being unimportant ...
2) a lot of the tracking happening is way too intrusive and intransparent. There are valid use cases, however some large corporations especially, in the last had cases where they collected way too much, including private information, without really giving information about it. That overshadows good cases.
If the questions you have can be answered by simple telemetry you are likely asking the wrong questions. E.g. a confused user will click all the buttons, while one thst efficiently uses your software to solve a very specific problem may always ever press the ssme ones.
The actually interesting questions are all about how your software empowers users to deal with the things they have to deal with. Ideally with as little buttons as possible. And if once a year they need that other button it will be there.
It is very easy to draw the wrong conclusions from telemetry.
Analytics do not tell you everything you need to know immediately. The analytics may say that no one is using a given feature, but they don't necessarily tell you why. Maybe they don't use it because they're not aware of it, marketing is presenting it wrong, or sales isn't selling against it. Maybe they've tried to use it and it doesn't work for them and they never tried it again. Maybe the call to action to bring them to it doesn't work or directs them wrong. Maybe it gets used by 1% of the users who happen to be power users. You might look at that 1% and conclude that it's not getting enough use to warrant supporting it or keeping it around.
This is just nonsense tbh. Surveys and customer outreach solve completely different problems from analytics.
Survey data is still real data that can be used for "analytics".
Some people also hate telemetry. It feels invasive. I have a guess about what direction the percentage of consumers who hate telemetry is moving toward.
While you’re comparing different information sources, you might even want to consider telemetry, too.
And the difference between what they do and what they want is equally shocking. If what they want isn’t in your app, they can’t do it and it won’t show up in your data.
Quantitative data doesn’t tell you what your users want or care about. It tells you only what they are doing. You can get similar data without spying on your users.
I don’t necessarily think all data gathering is equivalent to spying, but if it’s not entirely opt-in, I think it is effectively spying no matter what you’re collecting, varying only along a dimension of invasiveness.
Excellent point.
> but if it’s not entirely opt-in, I think it is effectively spying no matter what you’re collecting, varying only along a dimension of invasiveness.
Every web page visit is logged on the http server, and that's been the default since the mid 1990's. Is that spying?
Logging every page visited is not a technical requirement of serving the requested resource.
How will you know which page is having problems being served or is having performance problems?
Logging the requested resource is not a technical requirement of serving that resource.
If you buy something at the supermarket, the supermarket keeps a record of the transaction - it's part of the process.
However if you try and link that to entities and build a pattern behaviour across multiple websites then I think you stray into spying.
Also if the tin of beans I bought at the supermarket records audio at home and uploads to the cloud - that's spying.
We... we are talking about a CLI tool. A CLI tool that directly uses the API. A tool which already identifies itself with a User-Agent[0].
A tool which obviously knows who is using it. What information are you gathering by running telemetry on my machine that couldn't.. just. be. a. database. query?
Reading the justification the main thing they seem to want to know is if gh is being driven by a human or an agent... Which, F off with your creepy nonsense.
Please don't just use generic "but ma analytics!" when this obviously doesn't apply here?
[0]: https://github.com/cli/cli/blob/3ad29588b8bf9f2390be652f46ee...
And you know what happens when you reach out to talk to your customers like human beings instead of spying on them like animals? They like you more and they raise issues that your telemetry would never even think to measure.
It's called user research and client relationship management.
In the OSS world this is not a huge deal. You get some community that’s underserved by the product (ie software package) and they fork, modify, or build something else. If it turned out to be valuable, then you get the old solution complemented or replaced. In the business world this is an existential threat to the business - you want to make sure your users aren’t better served by a competitor who’s focusing on your blindspot.
Everything went to crap in the metric-based era that followed.
Yes, vendors can, do, and should talk to users, but then a lot of users don't like receiving cold messages from vendors (and some users go so far as to say that cold messages should _never_ be sent).
So, the alternative is to collect some soft telemetry to get usage metrics. As long as a company is upfront about it and provides an opt-out mechanism, I don't see a problem with it. Software projects (and the businesses around them) die if they don't make the right decisions.
As an open source author and maintainer, I very rarely hear from my users unless I put in the legwork to reach out to them so I completely identify with this.
Yeah, sure. How long is that policy gonna last? How does a user even know that that checkbox does anything?
Once you’ve decided to break a social contract it’s not like you can slap a bandaid on it and it’s all okay now.
> I hope you try to build a business sometimes and open up your perspectives that maybe just maybe you don't have all the answers.
People were building successful businesses long before the Internet.
People in this case are likely extrapolating based on how user data is harvested in the industry at large. So there is bound to be (very likely) some characterization that is unfair to you.
Given modern data aggregation, really data vacuuming, and that software is opaque, it can be really hard to trust anyone with any aggregation of data. They say that they pseudonymize properly. The proof? Trust them bro. Then read yet another news article about how some data aggregation was either sloppily leaked or just a front for selling data.
A natural response to opaque practices by people you don’t trust is a hardline no.
However, the plural of "anecdote" is not "data". People are unreliable narrators, and you can only ask them so many questions in a limited time amid their busy lives. Also, there are trends which appear sooner in automated analytics by days, weeks, or even months than they would appear in data gathered by the most ambitious interview schedule.
There is a third, middle-ground option as well: surveys. They don't require as much time commitment from the user or the company as a sit-down interview. A larger number of people are willing to engage with them than are willing to schedule a call.
In my experience, all three are indispensable tools.
Talking to users when you have hundreds of customers does no more than give you an idea of what those specific people need. If you have hundreds of users or more, then data is the only thing that reliably tells you these things.
Having the data doesn't mean you will act on it. And doesn't mean microsofts interests are aligned with the users.
Analytics driven development easily leads to bad outcomes. 1. Important, but less frequently used feature gets moved to a hidden spot leading to even less usage leading to eventual removal. 2. Poorly functioning features not getting the improvement they need because few use them due to how poorly they function.
I have seen these patterns a lot in software where decisions are based on analytics, and I usually stop using that sofware when I find a replacement.
More like flying based on your knowledge as a pilot and not by the whims of your passengers.
For many CLIs and developer tooling, principled decisions need to reign. Accepting the unquantifiability of usage in a principled product is often difficult for those that are not the target demographic, but for developer tools specifically (be they programming languages, CLIs, APIs, SDKs, etc), cohesion and common sense are usually enough. It also seems real hard for product teams to accept the value of the status quo with these existing, heavily used tools.
Flying based on the whims of your passengers would be user testing/interviewing, which is a complementary, and IMO necessary, strategy alongside analytics.
Ad slots will be filled whether or not you click. If you never click, you'll tend to match with either very low quality ads or ads that pay per impression (display ads).
Email registration is highly valuable for a business, so analytics won't be used to decide whether to show the modal but rather test different versions of it.
Cookies are too valuable to not push on users, because without them only the previously mentioned low quality ads can be shown. High quality and display ads match on interest or demographic labels.
The business decision to keep vertical videos is highly likely to be affected by analytics, and of course the choice of which videos to show is based on recommendation models trained on interaction logs.
The priority isn't making your experience better, though that is often an incidental result -- it's driving the business.
If you have too much emphasis on (invasive) analytics you might end up flying empty i.e. without customers.
This would be _absolutely insane_ telemetry to request from a user for any other piece of software, but it would be fantastically useful in identifying where people get frustrated and why.
That said, I do not trust Microsoft with any telemetry, I am not invested in helping them improve their product, and I am happy not to rely on the GitHub CLI.
Why is it that startups and commercial software developers seem to be the only ones obsessed with telemetry? Why do they need it to "optimize user journeys" but open source projects do just fine while flying blind?
whereas, commercial software has a disconnect between who are the users and developers are
Unthinkingly leaning on metrics is likely to help you build a faster, stronger horse, while at the same time avoiding building a car, a bus or a tractor.