The design of transparent telemetry | Hacker News Reader

The design of transparent telemetry

research.swtch.com

IMO the only part of this article that is general enough to discuss is the bit about opt-out telemetry; everything else is just slapping technical solutions and attempts at preserving privacy on top of an opt-out system.

On that front, I find it confusing why the option of “we ask the user when they install what they want” is not explored. It’s given as an example in the article, and then there’s a whole section about how they want to give people sufficient notice on their website and how it would be bad to report stats before people could realize it’s turned on. Why not just tell them when they install and let them change it then and there?

> why the option of “we ask the user when they install what they want” is not explored.

Because Go == Google, and when has Google "asked the user" before slurping data?

If they are going to do it, it should only ever be on an OPT-IN basis.

> If they are going to do it, it should only ever be on an OPT-IN basis.

They made their case against it, and I can even understand the reasoning behind it. Doesn't mean I have to agree though.

But still there's programs like Git as an existing example of this kind of "ask the user" thing. Last time I checked:

* When you run `git init` without having configured default branch name it warns you with a big wall of text.

* When you run `git commit` without having configured a user, it warns you with a big wall of text.

* When you run `git pull` without having configured the strategy, it warns you with a big wall of text.

As much as I want to give them the benefit of doubt, this really comes off as wanting to silently enable it by default, because they know that if they actually ask in a similar way that `git` does, the answer will be "no".

As much as I like and appreciate git, it's hardly a good example for something with great UX.

Exactly, it's an example of a CLI that already messed up its interface, and has actually changed commands (checkout -> switch, has added flags for submodules, etc), and overall has improved since past versions, and still tries to improve.

And AFAIK it didn't add telemetry to send used commands to any server, even if that would have been by far the easiest way, given 90% of devs use it and complain about how bad it is.

It's an arbitrary percentage, but it gets the point across. The amount of devs using git is probably at least an order or magnitude higher than those using Go.

Go, on the other hand, already has decent-ish tooling (at least better Git), but it wants to add opt-out telemetry anyway.

This is very uninteresting data to Google. They are interested in personal data that you can use to optimize targeted ads, not compiler cache miss counts from your local development workflow.

The reasoning for not making this opt-in are in the blog post and I find them reasonable.

Their reasons are centered on the low returns of opt-in (due to people regularly not changing defaults). They obviously fail to see that people might check their settings and do not want to opt in. And the reason for that is given right after that observation: people have been burned too often by companies and systems that slurp up ever more data way beyond what is necessary for the alleged purpose.

Opt-Out is simply not acceptable.

> This is very uninteresting data to Google.

It's not about how interesting the data is to them, it's about the hubris of believing they are entitled to every last bit of our data.

Let's not forget the https://en.wikipedia.org/wiki/Boiling_frog

If you think this data is not interesting to Google, I think you're not using your imagination enough. Even "compiler cache miss counts from your local development workflow" could be used as a signal to optimize ads for development tooling. Or hardware.

In general, when looking at telemetry data, you can't stop at asking "what possible use would the org collecting it have for it right now, in their current business model". You need to also ask what use they could potentially have for it in the future, especially combined with other data they have access to. A year from now, or five years from now, or after a pivot, or after partnering with (or buying) a company in a different industry, or after leaking (or selling) it to third parties.

When you ask these questions, I think you'll also conclude that enabling telemetry by default and pinky swears they're not doing anything naughy with it is not good enough. You may trust the people doing it now, but as experience shows, business priorities change, people in charge change, but the collected data remains.

A set of static, anonymous metrics, updated _once_ a year from a tiny fraction of developers is not worth the effort to even ingest into some targeting system.

There's many ways to get valuable data for ad targeting, this is not one of them.

If they want to turn bad, they can just use the dependency proxy.golang.org / sum.golang.org that already exist for a long time. Or gather data on things running in Google Cloud. All of these would be unlikely, and bigger levers than gathering some data from a few Go developers.

There are many other applications for data than just ad targeting.

The pushback here may seem paranoid for the specifics of this particular case, but that's because it comes from a more generic and IMHO perfectly justified position of treating all telemetry as risky by default, and all attempts at making it opt-out as wrong, dangerous, and likely malicious.

The arguments "for" are all based on assertions about this particular case at this particular point in time. The data to be collected is benign. Collecting it will translate to meaningful improvements being delivered faster. The people behind the project are all nice and reputable. And so on. In contrast, the arguments "against" cover the possible future evolution of the telemetry itself, as well as the project and the people behind it. The set of metrics and reporting frequency can change. The goals behind it can change. The nice people might turn naughty, or be replaced by scoundrels.

The arguments "against" include, in particular, building the precedent and infrastructure. From that point of view, you saying:

> set of static, anonymous metrics, updated _once_ a year from a tiny fraction of developers is not worth the effort to even ingest into some targeting system

sounds to me like me saying "this browser extension, which code I audited myself, is only reading the account numbers and names from my banking page - it's really not worth anyone's effort to do anything naughty with it, so there's no reason I should be afraid of giving it full read and write permissions for the banking page, and no reason I shouldn't leave the 'auto-update' checkbox on".

> I find it confusing why the option of “we ask the user when they install what they want” is not explored.

I don't. It's because they know that most people don't want this, and will say no if asked, but won't realize they need to opt out if it's just silently enabled by default.

Most users don’t want to send telemetry. But most users also want a system built by people who understand how users use it.

You can achieve 99% of that by using your own product and talking to your customers, and everybody managed to do just that and to make useful products long before telemetry became ubiquitous.

I would argue that using your own product and talking to your customers is the only way to make good and useful products. Telemetry may have a good use case (although I struggle to find one), but it isn't a substitute for dogfooding and direct customer interaction.

The true use case here is to get you used to the idea of telemetry in your tool chain, then move the goal post further down the road. In a few years, it'll be encouraged to put telemetry in your library ("Why?" "We've already had it in the tool chain this isn't any different!") then eventually, have every application sending telemetry data to Google's servers.

If you think this is ridiculous, remember they built Chrome with telemetry embedded .. why? Not for improving services. To build better targeted advertising.

Remember Android OS? Yes.

But there's still push back and the primary group of people resisting this at large are developers and the more technically inclined.

People don’t know how many cache misses they had, or what IO exceptions and retries led to their poor experience. They know they had a poor experience.

No one will afford to make interviews on the scale of making good statistics.

Knowing when I can sunset support for a specific OS or hardware (e.g screen resolution) would require a huge amount of expensive surveying.

The telemetry that tells me 20% of my users use 4K screens now literally costs single digit dollars per month. It costs the same as one or a handful of “talking to users”. More importantly, you don’t need to waste precious time when talking to users with asking them what driver version or screen resolution they have. Because of course users are spoken to. Telemetry isn’t instead of talking to users. It’s so that a) you can focus your user conversion on human focused things and b) so that you cover a wider group than you could ever afford talking to.

It’s better to be up front with the fact that software does anonymous telemetry and prudent that it actually is good (anonymous, and not using excessive resources).

The final link in the post is relevant to mention. Pointing to the Github discussion related to this idea: https://github.com/golang/go/discussions/58409

Locking the conversation and marking many disagreeing comments as spam or off-topic doesn’t look good for Golang.

I know that managing a community online is difficult especially when the community is against a decision, but I think Google employees should communicate better.

The most straightforward path for them is to announce that they have changed their mind about telemetry. Or everyone should use Rust instead, obviously.

I'm guessing your last comment is a joke, but honestly, that's sort of the consequence for at least some of my projects. I like Go, and I have a prototype of something written in Go. Now that I understand the domain better, I want to rewrite it; the initial plan was to do the rewrite in Go as well, but honestly, I don't want to endorse Google's telemetry and push it onto my users. So the rewrite is going to be Rust instead.

This is not adding telemetry to your code, it is very minimal telemetry within the go toolchain. They expect a user will send a week's worth of counters to the server once a year on average. It does not push telemetry on your users

If mort96 is writing libraries it does

How?

The telemetry is still only present in the toolchain. A library would be imported and built, and could trigger the telemetry collection, sure, but in what way would that be considered "part of their code".

If another developer is using their library, then it would ostensibly be a go developer who themselves has chosen to opt in or out of the telemetry, so they wouldn't be pushing it onto their users either.

If there are a wide variety of well-maintained libraries available in Go, people are more likely to pick Go, or even feel compelled to choose it, and possibly be subject to telemetry.

If the bulk of developers cease to create or maintain Go libraries, fewer people will use Go and be subject to its telemetry.

I'm sorry, but that's an absurd argument.

Your argument is that one user publishing a library in a sea of libraries which enriches the ecosystem is equivalent to directly pushing telemetry onto their users.... Because people are enticed by golang as a whole?

One library isn't going to be the deciding factor in someone choosing a language, and they'll likely have to deal with telemetry well before they get to a random library.

And that still doesn't answer the question on how that possibly means telemetry gets injected into their code.

OK, I was imprecise, writing a library in this hypothetical future Go would not add telemetry to developers' code, but it would contribute to inducing those developers to use the hypothetical telemetry-enabled Go toolchain. One library is probably not the deciding factor, but an ecosystem of libraries is just many thousands of instances of "one library".

This:

> but it would contribute to inducing those developers to use the hypothetical telemetry-enabled Go toolchain.

does not follow from this:

> One library is probably not the deciding factor, but an ecosystem of libraries is just many thousands of instances of "one library".

An ecosystem becomes enticing due to a number of factors. Availability of libraries is one thing, but so is the language, tooling, and general popularity. Those are all interdependent as a thriving ecosystem will generally mean better tooling and more focus being put on the language development, which boosts the language's popularity.

A user is going to choose a language based on all these factors, and while a large number of libraries would be one of the factors in favor, telemetry would be one against. It's up to the developer choosing the language to weigh these factors, not the individual library developers. Nor does writing a library for a particular language "push telemetry onto their users" unless the library is actually pushing telemetry directly onto their users. The language or tooling doing it does not equate to the library itself doing it.

The more Go grows, the lower the probability that any one installation will send data, per the proposal.

https://research.swtch.com/telemetry-intro#summary > sampling bullet point

My project is open source. That means I'm asking my users, explicitly or implicitly, to download the source code, download Go's tooling, and compile my code. That exposes them to Google's telemetry.

If you are building a library, then you are likely targeting Go developers who have already made their decision. If you are building a binary you can publish those and no one has to compile them

It's a CLI program with an associated library. That means anyone who wants to use it on a system I don't happen to publish binaries for, or who prefer to compile things from source rather than using pre-compiled binaries, or anyone who wants to use a commit I haven't published binaries for, or who uses a source package from their distro (Gentoo users, Arch users who use a source package from the AUR), anyone who wants to modify it to add a feature or fix a bug, all potential contributors, etc etc etc. would be exposed to Google's telemetry because of my decision to use Go.

And then there's everyone who wants to make use of the library and chooses whatever language makes that easier. I've certainly used Python for some projects just because a certain library is most easily used from Python.

Have you read their proposal and blog posts? A Go installation will send a report ~1 per year on average

This is not Google telemetry or analytics, this is independent Go telemetry and is very different

I suggest everyone read about Transparent Telemetry before overreacting. It's honestly quite reasonable and will benefit everyone

I'm not sure if you're aware, but the official Go tooling is developed and managed by Google; the decision makers and core developers are Google employees. Go is deeply died to Google.

I'm not comfortable asking my users to send telemetry information to Google. The exact details aren't very important. People can make their own decisions about whether they're fine with it or not, but I'm not going to push people towards accepting it.

If that's the way you feel, then stop using Go. No need to make misleading statements because you have some gear to grind every time telemetry comes up

What have I said that's misleading? I don't want to expose my users to Google's telemetry, and that is affecting my choice of language; what part of that do you have a problem with?

What the... ?

Telling someone to stop using an OSS programming language, because they like the language without spyware is a pretty shitty take. :(

Telemetry can be disabled trivially.

I know it can be disabled. Whether an environment variable is "trivial" or not is certainly up for discussion. We're talking about, "never ever run the command on a system without the invonment variable set globally or in a context where the environment doesn't apply". (Did you know `sudo` strips most env vars? `sudo -u www-data go build` would send telemetry even if you added the env var to your profile)

It's trivial to disable telemetry for any given invocation, but making sure that no invocation at all, across decades, is far from trivial.

There was a Nobel Prize in Economics awarded specifically for recognizing that things "disabled trivially" generally won't be, and this is a fully generic method of denying people their agency. See [0].

Opt-out, in most cases, is malicious, and you only think about introducing it when you're trying to make a change that leaves your users worse off.

[0] - https://en.wikipedia.org/wiki/Richard_Thaler#Nobel_Prize

That’s true, but there is another side to this same coin — people won’t do the right thing if it’s opt-in. Organ donation’s percentages are the two ends of the spectrum simply depending on opt-in/out.

I’m of course not comparing telemetry to organ donation, but I do feel that the problem is often overblown and is not that big of a deal. There is not much evil in trying to find out how many people used your toolings and from which OS approximately. (I dislike the Go language as much as it gets, so I am as unbiased as it gets)

You're right, of course, and your take is how "nudge theory", "trivial inconveniences" and low-key behavioral interventions are typically presented. I chose to use a phrasing biased towards the other end of the spectrum to highlight the "dark side" people often forgot about.

Now when I first learned about all this, I loved the organ donation and retirements plans examples - this all seemed like a brilliant hack to leverage human nature for greater good! However, my view on this flipped almost 180° over time. Two major reasons for this are:

1. Software industry. I actually deleted the long rant, because I have a specific, highly negative opinion of the business side of the industry - but we could debate that one all night, and it's not the main thing I want to say.

2. I actually brought the organ donation and retirement plan arguments up with family and friends, and was surprised by how much pushback I received in response. The specifics of each counter varied due to individual beliefs on those particular issues and politics in general, however the overall shape of the counterargument was always the same:

"I agree that having more people ${opting to donate organs, having a retirement plan} is good for everyone, and I understand there's a pool of people who don't make that choice simply because they never thought of it, don't care either way, or find opting in too difficult. Personally, ${I'm already signed up, I would consider signing up}.

However,

if they would do a trick like this to me, I would be angry. I have a right to be informed and make a choice on my own. This is denying me that choice, it's patronizing. If this was done to me, I would opt-out out of spite - in fact, just hearing now that people do this makes me want to not participate. After all, if it was all good-intentioned, they'd be up-front about it. Nudging with opt-outs smells like a scam."

Hell, a close family member said plainly to me that, hearing about this idea, they're now very skeptical about organ donation, and will no longer consider signing themselves up.

I tried to respond to people saying this. I tried to find a flaw in the overall spirit of their counter, tried to find a counter-counter of my own. But after many conversations, I instead found that I agree. On paper, opt-outs sound like super useful trick for doing good. In practice, they're predominantly used for bad, to exploit and abuse people. People know it, people feel it, and people also (even if irrationally) feel patronized by such attempts. As a result, even if you're 100% honest and well-intentioned with your opt-outs, whatever good you achieve this way comes at a cost of burning trust in the whole market/problem space.

The organ donation example now haunts me a little. Sure, signing people up by default may save lives in the short term - until enough people realize they haven't been given the respect and consideration that comes with decisions this big. Then those people will opt out, saying they've been used, and then others will follow, and then more people will die because of the bad reputation of the medical system wrt. organ donations.

> people won’t do the right thing if it’s opt-in

Yes they will. They'll leave it off, which is how it should be. ;)

> This is not adding telemetry to your code

Yet ....

It could easily be only a matter of time before we see feature creep and another "proposal", this time to insert it into the binaries.

This is a valid forward-looking concern that I think needs to be much more clearly addressed. It's one thing to have default-enabled telemetry in the toolchain itself, but once that window is open it's going to be much more difficult to close it and stem the tide of telemetry features creeping in further elsewhere in the Golang ecosystem where they'd be much more of a privacy concern.

Without a strong commitment from the Golang maintainers that this is where it would stop (have they communicated anything to that effect?), we have no reason to believe it will stop here, and all the more reason to be concerned that this is just the beginning and it will be pushed much further in the future.

From the issue on GitHub, Russ believes it would be "clearly inappropriate" to push telemetry into user code, so yes, they have communicated their opinion on this

> To be clear, I am only suggesting that the instrumentation be added to the Go command-line tools written and distributed by the Go team, such as the go command, the Go compiler, gopls, and govulncheck. I am not suggesting that instrumentation be added by the Go compiler to all Go programs in the world: that’s clearly inappropriate.

> Russ believes it would be "clearly inappropriate" to push telemetry into user code, so yes, they have communicated their opinion on this

Erm, not quite....

Somebody explicitly asked (on one of the moderator hidden posts, I might add) "@rsc Can you guarantee that this will always be the case?"[1]

The answer was wishy-washy to put it politely:

"I can't guarantee we'll all still be here tomorrow. But for as long as Go exists and is developed, I expect that interested open source contributors like yourselves will notice and flag changes that move backward. That seems to me a more useful check on the future evolution of Go than anything I promise."

Saying "you'll have to notice and flag it" is not exactly reassuring. Especially given the way the Go team have been reacting to the "flagging" on the telemetry discussion itself !!

[1] https://github.com/golang/go/discussions/58409#discussioncom... [2] https://github.com/golang/go/discussions/58409#discussioncom...

Saying he cannot guarantee in no way invalidates the team's stance and beliefs on the matter.

Citing toxic users does not help your argument

That's pretty common. It's what you say to your kid when they want to go to the amusement park this weekend, but you don't want to disappoint them right that second so you say something to push it off until you figure it out.

As always, slippery slope is a logical fallacy.

https://en.wikipedia.org/wiki/Boiling_frog

> Locking the conversation and marking many disagreeing comments as spam or off-topic doesn’t look good for Golang.

Yes, I noticed that.

Particularly given the discussion at hand (telemetry), employing heavy handed moderation tactics is not a good look.

They likely locked it because people are

1. Overreacting without reading the proposal and blog posts and making inaccurate conclusions

2. Violating the Go Community Code of Conduct (https://github.com/golang/go/discussions/58409#discussioncom...)

> 1. Overreacting without reading the proposal and blog posts and making inaccurate conclusions

As was pointed out multiple times during the ensuing "discussion":

    1. The proposal and blog posts were heavily biased, if not one-sided (highlighting only "pros", no "cons", pushing towards the only option being "opt-out" etc. etc.)

   2. Posts in opposition to the proposal were being moderated in a heavy handed fashion through the Github hide function etc.

   3. Valid answers were not being provided to those in opposition to the proposal, instead the answers were largely brushed aside with dismissive waffle.

Thus the only conclusion it was locked is because the discussion was not going in the desired direction, i.e. supportive of the proposal.

> Posts in opposition to the proposal were being moderated in a heavy handed fashion

> brushed aside with dismissive waffle

> Thus the only conclusion ...

It's hard to take your response seriously when you only offer

> heavily biased, if not one-sided

opinions.

If you want to make an argument that can convince people, it is best to not use polarizing language.

Most of the hidden comments are of very low quality or do not add to the discussion. Others are off topic. Many dissenting comments remain. So this "heavy handed moderation" you describe is not really accurate

Russ Cox gave his reasons in the Google Group.

https://groups.google.com/g/golang-dev/c/73vJrjQTU1M/m/twcLx...

> * Can we trust the server or Google to not track IP addresses?

Well, no. I think we all know the answer to that. Telemetry and tracking is in their culture. Their culture is embedded in advertising - why trust anything an advertising company wants to do?

FWIW, people had started to post obscene ASCII drawings, were using various curse words, and there were many, many repetitive comments across the ~400 comments posted.

From what I understand of the conversation there, the core Go team said they were going to take some time to digest the feedback, including [0]:

> The goal here is a productive conversation that aims at better understanding of different positions. Many comments here have contributed to that, and I am grateful for them. To be extra clear, the people who have been discussing opt-in vs opt-out respectfully and with reasoned arguments are most welcome here and have been an important part of the signal, not the noise. Thank you to them in particular.

> Much of the moderation is being done by volunteer contributors working valiantly to keep the conversation on track, polite, useful, and non-repetitive. I appreciate their efforts.

> This discussion has in fact scaled somewhat beyond what GitHub discussions can reasonably manage (I just spent a while clicking every "load more" link on the page to make ^F work again), which is causing even more repetition, so I will probably lock the discussion at the end of the day and take some time to think about the feedback we've gathered so far.

[0] https://github.com/golang/go/discussions/58409#discussioncom...

> people had started to post obscene ASCII drawings, were using various curse words, and there were many, many repetitive comments across the ~400 comments posted

Sounds like an appropriate and commensurate response to this terrible idea.

Incredibly disrespectful.

At first glance I'm not a big fan of this proposal, but I'm probably less opposed to it per se, and for different reasons, than the typical commenter.

But the behavior of "rsc" and "mvdan" there is just enormously decrementing my opinion of the Golang team.

mvdan is not on the Go team, he works on CUE primarily, he has been a volunteer for a long time.

Can you point out specific comments that you find problematic? I have looked and am not sure which ones you are referring to

Recent and related:

Transparent telemetry for open-source projects - https://news.ycombinator.com/item?id=34707583 - Feb 2023 (297 comments)

also:

Google's Go may add telemetry that's on by default - https://news.ycombinator.com/item?id=34744948 - Feb 2023 (37 comments)

> The server would necessarily observe the source IP address in the TCP session uploading the report, but the server would not record that address with the data, a fact that can be confirmed by inspecting the reporting server source code (the server would be open source like the rest of Go) or by reference to a stated privacy policy like the one for the Go module mirror, depending on whether you lean more toward trusting software engineers or lawyers. A company could also run their own HTTP proxy to shield individual system’s IP addresses and arrange for employee systems to set GOTELEMETRY to the address of that proxy. It may also make sense to allow Go module proxies to proxy uploads, so that the existing GOPROXY setting also works for redirecting the upload and shielding the system’s IP address.

I wonder if Apple’s Private Relay [1] mechanism might be a safer path that doesn’t rely on operators actually running the open source version verbatim nor on the privacy policy.

Disclaimer: I work at Cloudflare.

[1] https://blog.cloudflare.com/icloud-private-relay/

It's a good idea as long as they actually have some good motivation for capturing data. If it's just "we're curious" then I don't think that is good enough.

Many of the questions they hope to answer are here

https://research.swtch.com/telemetry-uses

Ah great link. Seems like a very reasonable list to me.

https://en.wikipedia.org/wiki/Boiling_frog

Opt-out sucks.

Depends, it is great for organ donation.

Reply on news.ycombinator.com