Hacker News Reader

Comment by arp242 | Hacker News Reader

> How is a web developer supposed to test the website when he and user might have different browser behaviour?

So that is always going to be the case with a change like this, simply because people use older versions, use different browsers, etc.

If this change breaks popular-site.com then it will continue work fine for 95% of people, and it breaks for "only" 5%, one of whom will (hopefully) report it. This allows the Firefox people to test the waters to make sure it's not going to horrible break things and break things for too many people.

View on HN

Remember: The people who would supposed to benefit from a rollout like this are the website operators, not Firefox. The change is happening; I doubt Mozilla is going to reverse course because a couple websites look a little wonky (totally reasonable).

How would they report it? "Hey, your website looks weird. Yeah I'm running Firefox 138 stable and I'm in the 5% experimental group who received the default h1 styling change"?

What is more likely to happen is: A website might get a report, the dev goes to reproduce it, and they're in the control group, so they can't; and because it was only one or two reports, it gets closed.

Firefox also has a built-in "broken website" reporter in the Help menu, which sends a report to Mozilla. The WebCompat team analyzes those reports and can decide to report a Gecko bug if it has a bug, contact the website if they have a bug, or apply a sitepatch. We're also monitoring the broken website reports for regressions from this change in particular.

I understand that a rollout makes reproduction harder, and not everyone will be aware of the change. It's a tradeoff when deciding whether to do a rollout or ride the trains normally.

> I doubt Mozilla is going to reverse course because a couple websites look a little wonky (totally reasonable).

This is totally possible. Many changes to the web platform we abandoned or changed significantly because a couple of websites look a little wonky.

> simply because people use older versions, use different browsers, etc.

Yes, but in this case asking the user to update to the latest version won't fix the problem either. I agree it's a terrible idea.

You would direct the user to submit a bug to Firefox if the latest version doesn't fix the issue. That's how they learn if this change causes issues.

You should first investigate the cause of the bug before spamming Firefox developers with potentially invalid reports. And it is difficult to investigate when your browser and user's browser behave different.

It's a beta version. Betas don't roll out evenly. Asking the user to update to a beta version has never been a good idea.

They are doing it for the stable release as well:

> The plan is to roll out to 5% of users on the Firefox 138 stable release, ramp up to 50% of users, then ship on all platforms in Firefox 140.

Yeah that makes no sense. Versions are meaningless if they don't represent standard defaults. But then, evergreen software makes versions something like an implementation detail anyway.

Usually we can use feature flags. Browser makers also have beta versions or their browsers available for devs.

Yay, "our users are our beta-testers" hard at work.

How about devs evaluate a change by doing some testing, instead of using users as guinea pigs?

I agree with parent, this is an awful approach.

They're not testing the browser, they're testing the Web. Testing in the real world with real users is the only way to adequately probe the effects of potentially breaking changes like this.

This is not new either. Many such changes have been reverted after discovering that they broke more things than expected. One example is the Object.groupBy static method, which was initially Array.prototype.group.

> They're not testing the browser, they're testing the Web.

That sounds like a pretty lame excuse.

So if a self-driving car manufacturer does testing, they are not testing the car, they are testing the environment? Sounds like a pretty neat trick, maybe marketing should adopt this attitude.

Some testing has already happened before starting the rollout.

The change has been shipping in Firefox Nightly for a year. I have analyzed impact of affected pages in the HTTP Archive dataset (about 12,000,000 pages), twice:

https://github.com/whatwg/html/issues/7867#issuecomment-1977... https://github.com/whatwg/html/issues/7867#issuecomment-2595...

The next step is either rollout or ship directly.

You can't test against the entire web, especially internationally. Do you know what the most popular Vietnamese sites are? I don't either. Do that for >200 languages. Never mind of course there's tons of non-public stuff.

You can test on special testing versions. Browser instances behaving inconsistently with the same version only add more problems to developers.

The scale is far too big. You cannot take into account the millions of permutations of websites and browser extensions when testing changes like this.

> How about devs evaluate a change by doing some testing, instead of using users as guinea pigs?

Presumably they have done as much of that evaluation as is reasonable already.

Second that - I assume that they did asnmuch as they could and this is just defense in depth one additional step.

That is not defence in depth, that is causing annoyance to web developers.

In my experience, that's probably not a good presumption anymore.

It feels to me like usability testing and quality assurance have both joined security and accessibility on the back burner now.

Yeah why didn't the developer simply test every website on the internet.

> Yay, "our users are our beta-testers" hard at work.

That is kind of part of the bargain when you download a version of the browser explicitly labeled as a beta.

> How about devs evaluate a change by doing some testing, instead of using users as guinea pigs?

"Who would pay for it ?" Sincerely, Microsoft™

Even Microsoft managed to figure that out and if Teams or Outlook get new look, there is an option to use it or switch back to the old one.

Reply on news.ycombinator.com