Firefox/Normandy/PreferenceRollout
wiki.mozilla.org
wiki.mozilla.org
https://bugzilla.mozilla.org/show_bug.cgi?id=1548973#c57
> Update: We have rolled out a partial fix for this issue. We generated a new intermediate certificate with the same name/key but an updated validity window and pushed it out to users via Normandy (this should be most users). Users who have Normandy on should see their add-ons start working over the next few hours. We are continuing to work on packaging up the new certificate for users who have Normandy disabled.
Further context: https://news.ycombinator.com/item?id=19823701
Tentatively re-enabled, it's now 2 hours since the announcement but I haven't received the preference flip study yet. A couple more hours before my fox's lastUpdateTime rolls around and my extensions get disabled, patiently waiting to see if I'll have to take manual action!
Edit: Funny, my app.normandy.run_interval_seconds was set to 24h for some reason, the new default seems to be 6h. I wonder how many people are also in that situation.
Targeting can be based on many criteria, including:
Firefox version
channel (release, beta, nightly)
a percentage of users
country
Firefox locale
installed add-ons
profile age
any preference value
many keys in Telemetry
I'm not sure I want any of those shared, especially the last several. Yet this was enabled by default and the only way to disable Normandy is through about:config? Does disabling that also stop whatever allowed the targeting in the first place?
It just seems like there was a real lack of informed consent regarding this feature and it only came to light when the team used this as a shortcut for fixing the add-on disaster.
What you experience is a synced remote interface, you don't own your software anymore.
Granted, the secret lies in providing such an excellent and subtle service that the majority of users actually either endorse or not even notice the dependency.
Which means changes should be subtle and consistent.
There should be exactly zero change to it UNLESS I said so, because I am admin of my machines - not someone from Mozilla, nor Microsoft, nor Apple. Is it that hard to understand?
And stop talking about "owning" anything, you clearly do not understand ownership rights - your idea of it does not apply to whole world AND this is not USA.
I'm too old for this "word plays". It seems like there is a need to pay someone, to not listen to mozilla and finally strip firefox from it's newly introduced "features".
The auto-update mechanism exists almost exclusively for the benefit of Windows users. None of the Linux distros use it.
Like for example you could make it so the script to be run when a browser encounters a pdf file is put directly in the settings. Thus changing that setting changes what the browser does—you can add, remove or change an in-browser pdf viewer that way.
It’s not what Normandy currently does, but it is something it’s at least theoretically capable of. And hitting that extreme would, in my opinion, made it reasonable to start calling it SaaS.
Typically, a user has to click the installer or use package manager or something to get the update, it’s their decision they make locally. If a setting is in place where this decision can be made somewhere else, it’s now a remote thing, a remote update. Quite useful thing in some cases and one of the things that make SaaS possible in the first place.
It may come across as a nasty dark pattern if a user must opt out of auto-updates (which is loosely what Normandy is, if you extend “auto-update” to also mean tacit enrollment in an experiment).
Instead it should be off by default, with a clear and unambiguous description of the functionality on a menu page if a user wants to opt in.
For example, I am a very privacy-focused browser user, relying on many special settings & extensions. Despite frequently reading about browser updates, I had no idea Normandy functioned like this, nor that it was enabled by default, until reading into this current Firefox certificate bug.
Upon learning what Normandy is and that it is opt-out by default with unclear connections to the traditional privacy dropdown menus to disable experiments or usage stat sharing, I feel that Mozilla violated my trust in a profound and deeply upsetting way.
But if speaking about the issue. Why Mozilla even need so much control over extensions? Seems like they built an infrastructure to be able to disable any installed extension remotely. That's just evil.
As opposed to what? How do you test for all that with manual updates, as an individual or a small organization? Even for a huge organization which reviews most of the updates it installs, critical updates should be applied immediately, to avoid the lag.
If it's opt-out I guess it's not "consent"? I recall opting into it, but I can't be sure they didn't change it.
The reason for Normandy to exist is to allow the developers to check if some change to that defaults is production ready yet. For example, you can start enabling by default hardware video acceleration for some people and compare the number of browser crashes they experience compared to the general public and use that knowledge to know when this feature is stable enough to be enabled for all.
[1] https://drewdevault.com/2017/12/16/Firefox-is-on-a-slippery-...
On wrecking Normandy, you can actually see all enabled recipes [2] and nothing seems smoky. It even seems that the hotfix (id=721) was used to unbreak Office 365, supporting the positive uses of this system. But I strongly agree that there should be more approachable list of them.
[1] https://blog.mozilla.org/futurereleases/2018/01/24/update-on... (HN discussion: https://news.ycombinator.com/item?id=16229927)
It gives the developers a great power—they can change applications behaviour outside typical venues for this. Most users expect an update to change things, but if my browser would start to behave erratically and I knew I haven’t updated it for a while, it would have send me on a wild goose chase for other things that I might have done that make my Firefox crash or whatever. It wouldn’t have occurred to me that some change to my settings was pushed without my knowledge.
We can only hope FF devs will use that power responsibly. It can be a nice feature or a complete nightmare. And I for one would very much welcome some kind of pushed notification about that (‘hey, there was a problem with this and that, so we changed that and this; here’s how you can revert the change if needed’).
I'm sure the users who experience said crashes without doing anything in their browsers are happy about this :-P
EDIT: In addition, think hard about the update strategy of applications that need to support that moving web.
> In addition, think hard about the update strategy of applications that need to support that moving web.
If you want the web and accept that the web is moving and you can't do anything about that, then it should be recognized that the web browser is hard time catching up. I actually hate most auto update things, but it had been too hard to use any web browser without such facilities. I use Firefox because at least I can see---and have a non-zero probably to alter---what happens to this moving platform.
Yes. I have friends that use it. They rarely run it on the entire web; they use it by default on unknown sites but they allow sites they use regularly to run unaudited JS so they can get things done. (The whole point of using NoScript instead of the config option to disable JavaScript is so that you can run some scripts.) This seems equivalent to not running random applications but letting trusted vendors ship you automatic updates.
> Heard of application sandboxing?
Yes. I even briefly worked on it in grad school before I left to join a company that was selling sandboxing for corporate desktops. But you were worried about "HW, electricity bills and connectivity," none of which application sandboxing helps: sandboxing helps which APIs you call but not how much resources you use, and application-level controls don't affect what an individual website uses. Regular ulimit-style limits work for limiting CPU usage. That was why I asked whether you have resource limits on requests from individual websites.
> Do you actually understand WHY we run web via HTML, JS and CSS and not as actual compiled applications randomly downloaded from internet?
I do—but again, that's about the API surface available to applications, not about the CPU or bandwidth resources available to applications, which is what you said you care about.
I am 100% sure of what I'm talking about. That's why I'm asking whether you are. I'm not being manipulative, just trying to figure out if you have a coherent threat model or not.
That would be equivalent only if you were running applications from trusted vendors each in a sandboxed VM that can't access anything on your system and only provide narrow functionality, not platform-like stuff.
"HW and electricity bills" are just examples, to tell you they're messing with someone elses stuff. Is it OK for someone from SAMSUNG to come to your kitchen and mess with TV settings? Or from IKEA to take your sheets and maybe give you something completely else? Also clerk would have keys, because using security cams they took picture of your keys while you were shopping and made "backups for you". You wondering what happened they would calmly tell you that it was OPT-OUT and you agreed to it by entering the shop. For me this is equivalently illegal.
I'm using (linux) distribution with pretty nice way to deliver software updates. (At least from my point of view) There is no need to have redundant malware entry channel into system.
I very well do understand needs to support moving web.
Auto update is something completely different than remote messing with users settings. I cannot help myself this reminds me of "windows support" scammers from last years.
That concern just doesn't parse. If you don't trust Mozilla to give you working software, why are you running their software in the first place?
There are circumstances and concerns where you might want to limit things like software and preference updates. And both of those are and remain under your control. But the default is to allow updates, for obvious reasons.
Edit: I'm not implying anything about the capabilities of this preference-updating system; the subject was software updates and who is trusted to deploy them.
You have completely misunderstood what Normandy does.
I cannot understand how anyone could think otherwise of Normandy.
I don’t think it is. That way, you have two authorities checking things over instead of just one.
I trust my mother; it does not mean I trust you, and even less the entire crowd here on HN.
I've stopped using Windows years ago because I didn't trust Microsoft anymore. I'm having minor trust issues with Canonical and Mozilla. I still trust Debian and Arch.
Trust is personal. Hard to get and easy to lose.
Only the trustee changes, so I still can't parse your concern. You don't trust Mozilla to RECONFIGURE (!) or REPLACE (!!) your software unbidden. You trust Canonical or Red Hat or whoever. And that changes what?
This has some limits -- it largely applies to systemwide, not user settings, user dotfiles are generally outside of scope (though are also typically unmolested), and some packages, including Firefox, manage to introduce their own configuration management schemas and misbehaviours.
But yes, the general idea that software updates do treat user and local configuration as sacrosanct is well-established, decades-old practice, in at least some parts. And goes a long way to establishing and sustaining user trust in those systems and their update processes.
Mozilla face numerous challenges, including a large multiplatform largely non-expert userbase operating in an overtly hostile environment. The whole browser extension regime is a very unsatisfactory compromise itself. There remain lessons which might be applied from Debian.
I know when my Firefox updates, so I can expect changes.
My Firefox is vetted by my Linux distribution.
This is the first time I have heard about Normandy, and I certainly would not turn it on, because I prefer my work machine to be a laggard than an early adopter.
What is the point of people like me using Firefox Nightly? Do your tests on me. Don't do stupid shit with people who choose Firefox Stable. Who came up with this idea anyway?
How is a lack of data for Mozilla my problem? Why does it mean that can inject default preference changes?
It would mean they can't roll out things like hardware acceleration or Stylo or Webrender as quickly despite their numerous manifest benefits.
By not opting in, users would indicated that experiment-avoidance is a feature that gives them more value than the fast rollout of those other features.
A better middle-ground is to let things get tested by those who opt-in to it (using Nightly and Beta versions), and slowly trickle changes down. That way none of the published versions are so different that Mozilla needs more staff to handle the different versions. And of course, stable users have far fewer issues than nightly/beta users. Certificate expirations throw a wrench in the whole system, but even if you made FF stable never update, you'd still have a problem because the cert expired.
» It would mean they can't roll out things like hardware acceleration or Stylo or Webrender as quickly despite their numerous manifest benefits.
You said:
» Eventually Nightly would be completely different from stable, especially with the switch to Rust. So then Mozilla would have to maintain 2 completely different versions of FF - that's a lot more work!
As unofficially mandated by the new owners of the Internet - the Google Chrome team - the time between two "major" versions of Firefox is six (to eight) weeks. Yes, we can wait six to eight weeks for new features or twelve to eighteen weeks from trunk to stable (x2 of six to eight).
What we should do is enourage a wider swath of the population to adopt Firefox developer edition and Firefox nightly.
» A better middle-ground is to let things get tested by those who opt-in to it (using Nightly and Beta versions), and slowly trickle changes down. That way none of the published versions are so different that Mozilla needs more staff to handle the different versions.
Thank you. This is exactly what I want. I am not saying things should never change. I'm just saying don't experiment in stable. We are already hemorrhaging market share as is and this nonsense doesn't help.
Mozilla seems to be culturally broken these days, and learned absolutely nothing from the previous studies crap.
People might say no, and that would likely very much anger a PM somewhere.
Of course it begins with good intentions, and promises to leave explicit privacy options alone, but new devs show up with different opinion and the old devs are gone, and suddenly privacy options are getting toggled any old time.
Beyond even that, we all know that the realities of privacy are never ever cut and dry. Leaky details can expose people in peculiar ways. Fingerprinting preferences and hardware facts for forensic purposes has taught us that much. Viewport size, OS, connection speed, graphics capabilities, hardware acceleration profiles. Even stylometry, choice of words, manner of speech can give people away. In that sense, exposing any user choices might prove to compromise identity to some degree.
1. https://bugzilla.mozilla.org/show_bug.cgi?id=873709
2. https://limi.net/checkboxes-that-kill/ (seriously breathless persuasive writing about how urgent it is to hide the toggle for javascript, among other things, but make no mistake, the high value target was the javascript checkbox)
It's kind of cool that this worked without a browser restart. My extensions just reappeared while I was watching some netflix.
If Mozilla started playing around with the rendering preferences, perhaps enabling OpenGL hardware acceleration, that would definitely not have been OK. I don't need my browser deciding to suddenly surprise me by crashing my system.
Maybe if we put this in terms of user stories...
As a user, I don't want to be "targeted".
As the person deploying updates, I don't want the deployment updating without an update having been deployed.
Then all my plugins came back, and I disabled data collection once again.
The downside is that the process of updating the software becomes a bit fragmented, which is probably confusing users now.
However, the current problem people are criticizing is not about Mozilla's choices for default preferences. The specific changes they ship with the browser or update with Normandy are not (currently) particularly interesting. The problem is that a new way to remotely control the browser was added unannounced that bypassed existing update methods.
If you want to change other people's property, you get permissio9n first. If someone doesn't want to give you permission and you change their property anyway, we usually call that something like "trespassing", "vandalism". It doesn't matter if you think it's an important change or if you don't understand (or even know) their reason for not granting permission; their computer is their property, and they don't have to justify why they want to use it in any particular way.
If the _only_ way to prevent having a remote entity modify your settings unannounced (even if not for malicious purposes) is to enter about:config and change app.normandy.enable to false, that seems like a situation where the absolute best case, most charitable interpretation is to call it an incredibly deceptive dark pattern from Mozilla.