Google Chrome experiment crashes browser tabs, impacts companies worldwide
zdnet.com
zdnet.com
> "Do you see the impact you created for thousands of us without any warning or explanation? We are not your test subjects," said an angry sysadmin. "We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'?"
Did you, dear sysadmin, pay anything for Google Chrome? No? Are you in any contractual relation to Google that covers your use of Chrome? No? Well, there you have it. You are not a customer, hence you aren't really in any position to demand anything. You basically agreed to take whatever Google shoves down your throat for free, and if that includes "experiments", then that's what it is.
If your multi-million dollar programs move so much money around, maybe take some of that to either invest in the necessary software - including the browser - so you are a paying customer and may actually demand anything, or pay some people to be up to date on the intricacies of Google Chrome and test them under your environment or disable them if undesired. The experiments are not exactly a secret program and have existed for a while. Firefox has a similar thing going. The Firefox one can definitely be disabled. The Chrome one unfortunately not (at least I don't know of a way at the moment, there might be one, maybe it's also only available for the Enterprise subscription that I have never personally heard of any business to be using, but for which money is actually paid, hence there might be leverage in that case to force Google to offer an opt-out).
Or ultimately you could also just compile yourself a Chromium from scratch and update it regularly. I've done it, it's not that hard, and that gets you the ultimate level of control over that nice free piece of software that you depend your business upon.
If these "experiments" didn't exist, the alternative would be that the new stable version would come out and it would break everyone, so we're back to square 1.
> We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your 'experiment'? -- Angry sysadmin
Well, do they pay Google for testing in their environment?
Chrome is a free-to-use product. Their rollout strategy is good. Not that they experiment in prod - the flag was in beta for 5 months. And then they turned that on in prod for %1 users, still no reports.
Well what better could they do?
I think you've misunderstood what they did
> the flag was in beta for 5 months. And then they turned that on in prod for %1 users, still no reports.
It was in beta, had no reports, and then they turned it on for a subset of users, and created a flood of problems.
> Well what better could they do?
Release it in a new version that doesn't auto-enable for small subsets of people, but is enabled for everyone if they've deployed that version. It's how software releases used to work. Enterprise environments can then test before it gets rolled, and save this problem from occurring.
Their rollout strategy was bad. They tested it on 1% of beta users for a month. They should have ramped it to 100% in beta before thinking about prod. Maybe 1% for a week, 25% for a week, 75% for a week then 100% for a week.
Then think about enabling it in prod. Testing on a subset of the subset of people that run beta is not enough to validate the functionality.
Even then, they probably should have done a ramp in prod as well. There are certain configurations that seem unlikely to be tested in beta. VDI is one that comes to mind. Headless operations also seem less likely to be running the beta build.
Your claims would make sense if Google hadn't destroyed the market for paid browsers and are now trying to kill what's left of the free market too (Safari & Firefox).
Who can compete with a product which receives tens of millions of dollars of investment per year and yet is given away for $0? US competition law is a joke, all regulators are asleep at the wheel.
I've been at conferences where senior Google staff went to great lengths to present the benefits of switching to Chrome.
> The experiments are not exactly a secret program and have existed for a while
I've been responsible for managing Chrome in a professional environment for several years and looked in detail at many Chrome management settings. I was not aware of the experiments. They don't seem to be mentioned in the Chrome Enterprise documentation or policies.
> Both can be disabled
How can we disable them? This is on my to-do list for today, so would genuinely appreciate your help!
Chrome (and chromium) calls the feature "field trials" and there doesn't really seem to be a way to opt out. I'm seriously baffled by the audacity of Google here.
Seems as if the only way to actually get rid of them is to modify the Chromium codebase and compile it yourself.
It might be worth to investigate whether there is some kind of host name that could be nullified on the network, of the server that the trials are loaded from.
Sorry for that misinformation, I clearly have to update my original rant.
Join the club!
> It might be worth to investigate whether there is some kind of host name that could be nullified on the network, of the server that the trials are loaded from.
It looks like they're downloaded from https://clients4.google.com/chrome-variations/seed
Blocking the entire hostname would cause problems with other features of Chrome. But I think blocking the particular URL would need SSL interception which isn't very appealing.
But the options to disable it on official prebuilt Chrome appear to be grim indeed. SSL interception might not even work at all, as I guess Google uses certificate pinning.
SSL interception is very appealing in a corporate environment. Thanks for sharing this, I've raised it with the guys who run the proxies here to investigate if we can / should drop this.
Any enterprise that didn't have mitigations in place cannot legitimately complain about what is probably the most reliable piece software they use (adjusted for some complexity metric).
The frustration here is that Google are operating outside the way we're all used to for software updates. If we update the software, we know to expect problems. So we can test it and (crucially) roll back.
If Google make changes outside of that then it becomes far more difficult to manage.
I am sure that those multi million dollar enterprises affected by this issue have other softwares so reliable that this was the only time they lost millions on software issue /s.
https://bugs.chromium.org/p/chromium/issues/detail?id=102483...
It is well worth a read! There are lots of complaints from what seems to be business users of the free version, but also a few by customers of the Chrome Enterprise subscription with managed installations, and they also appear to be affected by this issue and are demanding an option to disable the trials. From that I conclude that it can be considered safe that the trials also affect paying customers, and it does indeed seem as if they also have no option to disable them, at least currently (my personal guess is this is going to change, for sure at least for the paying customers, hopefully for everyone).
If that doesn't count, there there is no scenario where you can say you've paid for it.
Yes, if you have a G Suite or G Suite for Education account, or if your company purchased Chrome Enterprise Support.
However, Chromium being open-source, there is always the possibility of taking matters into your own hand here as an alternative to the above. That doesn't come free of charge as well, of course, the money just ends up somewhere else than Google.
The other thing is the way that was described by the article: "I am a sysadmin from a multi zillion enterprise, your idiotic experiment broke our setup. Disable this ASAP and never do something similar ever again".
The idea that because you did not pay anything for a product translates into 'you have no rights' really is getting old. Even if you don't pay you have plenty of rights and the provider of the software should not - and in many places can not - walk away claiming that because you did not pay you have no rights. It may not be the rights that you want, and it may very well be that certain behavior even if you don't like it is allowed but it just simply isn't true that you have no rights.
And that contractual bit runs both ways: companies have tried to argue time and again that free users of websites and software products were bound by EULAs, that means you can't now turn around and suddenly claim that there is no contractual relationship. Yes, there is no paper contract. But you are in a relationship and if Google or any other provider claims that they have rights by the definition of the word 'consideration' so do you.
I am not doubting that such rights exist. But I am doubting that you have any legal right to demand Google to not enable a generally non-destructive and well-intentioned feature in some randomly chosen installations of the software that just by bad luck happens to have a bug which makes the software non-functional (but not actively harmful) in your particular terminal server environment. There clearly was no malicious intent behind this particular issue, and Google is not obliged to test every single feature on every possible combinations of systems, including every kind of remote terminal solution imaginable.
And, having established that the legal system does not help you here, I additionally doubt that you have any other leverage over Google to make them not do such feature enablements in randomly chosen installations if you don't pay for the software and if you don't happen to be a company of huge size (which might allow you to threaten to switch your hundreds of thousands of users to a different browser). But if you paid for it, Google just might be interested in keeping that cashflow flowing, and thus might be inclined to put the additional effort in to create an option for you to disable these random feature enablements, and/or to disable them outright for the paid Enterprise installations.
So don't go around now claiming straw man arguments. Clearly, and you seem to agree, you do have rights, and whether or not that particular right is yours remains to be seen. I can see several ways in which it just might be so that you have that right based on your expectations of performance and that a near-monopolist like Google is walking on very brittle eggs the day they start using their position to perform unsanctioned experiments on the population at large, especially when those experiments can't be opted out of.
> straw man (n):
> an intentionally misrepresented proposition that is set up because it is easier to defeat than an opponent's real argument.
So, no, it isn’t a straw man because he’s not manipulating/misrepresenting your argument. Whether an argument is about defending malicious behavior or not has no bearing on whether or not it’s a straw man.
If your business is so locked into using Chrome, and specifically Chrome, and you aren't paying money to Google to ensure it keeps working as you need it, then you have only yourself to blame.
(And yes, being unable to disable the experiments thing, and not giving any warning before-hand, is a mistake on Google and the Chrome teams part, and hurts Chrome's enterprise-ready image, but that doesn't refute the point above).
If an essential piece of your business relies on something you have no control over, then you only have yourself to blame when that reliability fails and you can't fix it. Cost of doing business, at worst.
The problem with Chrome is that developers push each other to use Chrome, so they only develop for Chrome, and the stuff they make might only work in Chrome.
Google cheated their way into the #1 browser position by bundling Chrome with other installers (similar to the spyware of old), advertising for it on its other properties and misleading users into thinking that their current browser was worse.
Now that they're #1 they're hard at work implementing anti-features such as disabling an API which was used for ad filtering, trying to keep users always logged in, tracking visited URLs, etc.
Chrome is Google's trojan horse, they're not investing millions into it out of the goodness of their hearts. They're doing it because it gives them something priceless: control over the window people have to the internet.
Once the above is clear, it becomes easy to understand why Chrome auto-updates and is nearly impossible to stop from doing so. With these experiments Google has a backdoor installed into most computers in the world.
And you're saying people should not be upset about the backdoor, because they didn't pay for the trojan horse.
It's not the people that should pay. It's Google for spying on them and for running "tests" on their computers for free.
This was not a new version of Chrome or a software update. None of our software was updated at all (and we spent a long time checking!). But apparently Google have an ability to change a setting and globally affect the behaviour of all Chrome browsers by enabling experimental features.
We carefully manage our software updates and patching so that we can test it and roll back if it impacts the business. Google had been good at understanding "enterprise" requirements - disabling automatic updates, setting policies etc. But this shows that they're really focused on consumers and business users will always be an afterthought.
My own experience with ChromeOs is that I switch to beta, report bugs via interfaces provided (by Google) and then the chromium team acts surprised when the regression hits stable.
If you ignore signals don't ask for them.
Edit- clarity
It's not about businesses versus personal users, people are for Google an afterthought, they are mere information generators.
Uhm, isn't that the entire auto-update feature, that google ships changes without you even being aware?
What kind of site does that? I'm on ESR too (from Debian's repos) and never seen anything like it
Sure there are, especially in enterprise, where IE is still a thing.
There aren't good alternatives, but...
>"The experiment/flag has been on in beta for ~5 months," said David Bienvenu, a Google Chrome engineer. "It was turned on for stable (e.g., M77, M78) via an experiment that was pushed to released Chrome Tuesday morning."
>"Prior to that, it had been on for about 1% of M77 and M78 users for a month with no reports of issues, unfortunately," he added.
The bug affected Citrix-style environments, where perhaps 10-20 users are running Chrome on the same machine. If 1% of users are running the experiment the chance of two of those 10 users running the experiment is tiny, and when they experience the bug it's unlikely to get reported back to Google.
Ref. quote from article:
"Chrome engineers operate a system called Finch that lets them push updated Chrome settings to active installs, such as enabling or disabling experimental flags."
We were affected and our version hadn't changed (in fact we weren't quite on the latest version - we were still testing it). We have updates disabled and are very much aware of how to manage this.
Google changed a feature flag that was automatically picked up by existing copies of Chrome and changed their behaviour.
I think the difference here is to do with who's responsible for keeping things "working".
In a business environment there is a whole IT department who go to a lot of trouble to guarantee that their colleagues can continue working. But when Google take that control there's a clash, and it hinders the IT team's capabilities.
I've seen this in a Swiss government agency that has Firefox installed exactly for this purpose. Their main browser is the built-in one, but if it can't or must not be used (e.g. because of a zero-day threat), they can quickly roll out a change to make Firefox the default browser until the problem/threat is over.
...why not just make firefox the default browser full-time?
Another example is a recent issue where a Chrome update broke WebView based Android apps and stopped them from being able to make certain types of network requests. It was fixed in 2 weeks, but that 2 week period was full of unhappy customers and lost revenue.
I'm hoping the upcoming Chromium based Edge from Microsoft will allow IT Admins to control when a browser update is rolled out and give them more control over the update process.
Chrome auto-updates can be disabled for networks behind a firewall by blocking the update server address, but that's a very crude way, and doesn't allow for updating a test machine to see how the new version works, or updating to the latest minus 1 version.
Of course, Mozilla needs to make sure they never enable extensions like the way they did for the Mr Robot one many years ago...
Where is all the freaking out at Chrome? My theory is that is was mostly Chrome users (the majority) complaining last time when they had no personal experience.
I hope this changes when Edge goes to Chromiun but from a UX perspective, too much stuff doesn’t work on Edge.
"The attack is only possible through a carefully crafted browser ..."
Google thought this feature was polished, finished, and ready for release. What used to happen was it would be pushed out in an update, and then everyone on a server would have seen the crash. This "experiment" is just slow releasing.
The quote from this article is clearly from the later case, even if I am sure that other people reported it more humbly.
Enterprise IT management should realize this.
Not sure where you've worked!
[1] https://github.com/brave/brave-browser/issues/4283 [2] https://github.com/brave/brave-browser/wiki/Deviations-from-...