Replace the Text Encoding menu with a single override item
bugzilla.mozilla.org
bugzilla.mozilla.org
1. Very few people use this feature but a few vocal users who are particularly good at navigating bugtracking bureaucracy do use it and care deeply.
2. Of all Firefox users, a small percentage speak languages for which there are many non-ascii letters and a relatively large amount of pre-Unicode content online, and of those users a small percentage read niche (eg non-facebook and old) sites with broken character encoding.
Both situations could probably lead to few uses of the menu in telemetry. Perhaps we should care more about users in the second category as they are perhaps more correlated with other things like location whereas the first set may be more uniformly distributed across the population. I think other points are also being disregarded:
- ordinary users may find the feature cryptic and confusing. Or at least undiscoverable
- dealing with character encoding and supporting the feature is painful (though the will still deal with it under the hood). Mozilla have limited resources and they (and Firefox users) want the browser to be successful and sustainable, and this means focusing resources on things that will make the browser easier to adopt and less confusing rather than chasing the short-lived gains from appeasing a small and stagnant pool of ‘power users’.
To begin with, near nobody uses Firefox today. Very succesful, and sustainable.
I have to pour heavy sarcasm here. It's honestly justly needed.
Firefox went from 50%+ to sub 1% under the current leadership.
215 million MAU is still a lot of people: https://data.firefox.com/dashboard/user-activity
This feature is in no way "painful". It's a legitimate facility for dealing with non-Unicode, legacy content, and AFAICT it's not clear that it can be reliably replaced, even by a plugin. Any argument that the menu entry is "confusing" for most users can be addressed by hiding it behind an advanced preference setting, and I think this was done already.
To be clear, no such extension appears to exist yet, but it should be a matter of time before one is created.
The hardest part was doing all the UI bits, because I had to learn how to make a UI that would look halfway decent in the popup menu. But hey, it works!
Ask yourself: How often do you use the emergency number, how small is the percentage of the population that does in a day? Or even in a life? In the light of this, would you want it discontinued?
This is a pretty familiar pattern for controversial telemetry-based browser decisions, first making the feature more obscure or hidden, then later noting that few people are using it when justifying change/removal. How many users just abandoned a site they used to use or switched browsers is not captured in the telemetry (though options to switch to that still offer this feature are pretty thin on the ground).
To be clear I don't think this was a "blind" telemetry decision; the developer seems to have done a pretty thoughtful and thorough analysis about it. But you see the pattern of such changes over the years and you start to imagine there's a thumb on the scale...
In the cases where you'd use the remaining menu item in Firefox, Chrome offers no recourse (apart from extensions).
The collapsing the submenu of the View menu into a single item shipped in Firefox 91. The submenu was removed from the hamburger menu in Firefox 89.
This order of changes was unfortunate. If I had gotten the single-item design done sooner, perhaps the rewritten hamburger menu would have kept the single item instead of removing the entry point altogether.
People only dealing with languages that use the Latin script probably don't use it at all. But, speaking from my own experience with Russian, this feature is useful. Now that unicode is the default, text encoding problems are very rare, but they do sometimes still happen. Tools like this[1] were created out of necessity, not curiosity.
Dealing with character encodings is indeed painful, but they are not dropping support for all of these legacy encodings. If they did, I am sure it would simplify things a lot in terms of maintenance, but they are too widespread in existing content to do that. Firefox will still use a legacy encoding if it detects it. All they are doing is removing the ability to tell Firefox that a website uses a certain encoding. The idea that this option is a maintenance burden is kind of hard for me to believe.
> Additional non-user-facing benefit:
The back end code for supporting the non-Automatic menu items got in the way of implementing bug 673087. Based on that experience, chances are that fixing bug 1701828 would be much easier if we first implemented the change proposed here.
First off, what needs to be acknowledged is that any website that requires you to override the charset is broken and the greatest part of your ire should be directed at the people who write and operate such websites for expecting that everybody else should be responsible for cleaning up their messes. Let's also acknowledge that avoiding this breakage has been possible for (checks notes) 22 years.
Since this is very legacy stuff, at what point can we say that it is no longer the end consumer's job to deal with legacy cruft? The motivation for why a user-level override was ever necessary in the first place--the fact that once upon a time computers couldn't simultaneously represent content in different locales--is quite ancient history. I don't think an answer of "never" is compelling: why then aren't we complaining that browsers don't let us select EBCDIC or UTF-7 as an encoding (both of which were once supported, but have been dropped)?
At the same time, this change does somehow still feel premature; dropping the ability to override charset encoding does feel very "WTF?" to me. But... if that's the case, then what can be done to hasten the moment where we live in a world where dropping this feature doesn't feel like that? (This makes me glad I work in compilers, where no one attacks you for deciding that some inputs are just too wrong to attempt to deal with it.)
Also, it's wrong to say that there's "no harmful effect whatsoever"--the ability to do charset overrides requires code to support it, and that code could force awkward compromises in (say) your HTTP layer. All features have costs, even if they're invisible.
There's a whole world of patents that make video a particularly problematic space for browsers, but just the basic philosophy of continuing to work with old static content forever isn't that strongly held (and really, isn't that strongly held in much of software: consider opening really old Word or WordPerfect documents just as an example).
Especially in this heated sort of discussion, I think we need to know more than 'feels premature' or overripe or whatever. What about the data they have? The developer's research is linked in this discussion somewhere.
And of course we are deep in a bubble. Almost no end user knows what character encoding is, and few have any hope of fixing the problem manually. In fact, calling the menu item 'Repair Character Encoding' (or whatever they chose) is probably poor UI - you need something that end users will understand, more like 'Repair Gibberish Text'.
It used to be that even minor web platform changes wouldn't go through if they would break 0.1% of webpages, but we don't live in that world anymore.
And I'm glad. I'm sick of using software that has 2 trillion toolbars, menus and settings. I want something which sets the defaults to be pretty much what I want and makes the settings all the common things I might need to change. Telemetry driven design lets designers build what the majority like rather than the most vocal bug tracker using power users who want exactly the opposite of the average user.
I disagree. Maintaining features requires engineering overhead. It sucks to have to maintain features that barely anybody uses when you have a backlog of features that are in demand by a large number of your users.
No one is suggesting the feature isn't ever useful. The issue is that they've reached a point where they have to choose what stays and what goes, and the least frequently used features are obviously at the top of the list for what to cut.
You can't make everyone happy in these scenarios, just as you can't deliver everything.
Some features are used a lot because some other features are inefficient. Sometimes a feature is very useful in some rare situations (compare with seatbelts).
Removing something like the encoding menu is a choice, not a necessity.
I'm sure it doesn't help that Chrome killed its own encoding menu years ago.
Not that I would ever justify removing something like this - it's clear that for a percentage of the userbase, this is necessary for them to browse. But I can see a product manager trying to cut "fat" if they see that 0.01% of people use it.
What better data should they use?
> you can see how that's going for Mozilla
HN commenters (not necesarily readers) react this way to most changes in everything. Mozilla doesn't stand out IME.
For one, they could take comments on their bug tracker to heart instead of locking the discussion as soon as there are enough users unhappy with a change.
Alright 15 minutes have passed, here we go again. Back, reload, view page source, bookmarks,...
Sure, but you're missing the bigger picture, which is that each feature has all sorts of other costs (maintenance, security risk, codebase complexity, etc). As mentioned in the bug, this feature got in the way of implementing another feature.
Telemetry helps guide the balance between those costs and the value the feature brings, and you can disagree with how that balance in each decision, but it still is a fact that codebase health is important and if you don't prune, you'll very quickly end up with a huge messy project, no matter how careful you are along the way.
Of course, the other option would've been to rewrite/refactor the feature, but again, is an engineers time better spent working on something that helps 50 people or 50K people?
But nobody has infinite engineering resources. Maintaining everything comes at a cost, and eventually you have to make some hard decisions about which features to abandon so you can reallocate engineers to features people actually use.
Frustrating for the small number of people who use that specific feature, but you can't please everyone.
Irrelevant in this case. There are far more effective ways of paying down technical debt and enhancing future maintainability than removing a small, self-contained feature that occasionally addresses a critical user need. This is Firefox we're talking about, they've got a humongous legacy codebase to play with.
I bet the maintenance burden of the menu is extremely low.
> Hamidiye turecki kr±¿ownik pancernopok³adowy z pocz±tku XX wieku, wodowany w 1903 roku, zbudowany w brytyjskiej stoczni Armstronga. Wyporno¶æ normalna okrêtu wynosi³a 3904 tony, a d³ugo¶æ siêga³a 112 metrów. Napêd stanowi³y maszyny parowe o mocy 12 000 KM, pozwalaj±ce na osi±ganie maksymalnej prêdko¶ci 22 wêz³y. Artyleria g³ówna sk³ada³a siê z dwóch pojedynczych dzia³ kalibru 152 mm i o¶miu dzia³ kalibru 120 mm. S³u¿y³ w marynarce Imperium Osmañskiego podczas wojen ba³kañskich oraz I wojny ¶wiatowej, a nastêpnie w marynarce Republiki Turcji do 1947 roku.
Personally, I think this change is ridiculous. It has been a few versions already since it was rolled out, but people are still complaining in the issue tracker. I know one Russian guy who has resorted to using an extension which replaces arbitrary strings in order to correct common mojibake sequences in order to deal with the regression in functionality brought about by this change.
If you feel strongly about this like I do, I strongly encourage you to comment in the Bugzilla thread. Yes, the web should be using Unicode these days, but if it isn’t, that is not your fault as a user, and making the experience miserable for the end user is not justifiable.
As a Polish person who has been seeing this sort of mis-encoded Polish text for over two decades now, my gut instinct is to immediately reach for the encoding menu. That menu is gone now.
We live in the era of almost omnipresent UTF-8, but it simply feels wrong to remove backwards compatibility with older documents on the Polish web that are mis-encoded like that - and there are still some of them out there.
It's open-sourced on GitHub in case you're curious. It just overrides the Content-Type header on pages where you've turned it on, setting the `charset` parameter to whatever you select.
What is the compatibility worth if people have to manually figure out how to switch it on?
Feedback & bug reports welcome!
"Set screen coordinates during HTML5 drag event"
> The current HTML5 spec describes that all DragEvent properties should be available during all the events - according to editor Ian Hickson.
>> Note though that it doesn't specify what the properties should be set to, just that they should be set and we currently set them to 0.
Goal was apparently to bring FF to feature parity with Chrome. Sad to see it's not working as well as intended. (Even sadder to see his person being attacked by some commenters below.)
And because of this post, I just checked if chromium supported encoding menu. Yep, they removed it long ago. Basically, when you need to read some very old Japanese site where their encoding is pretty non standard, you are screwed.
I think your example of emergency services is a bit hyperbolic. This is a feature that is really not often used, whose omission is not fatal, which often requires several tries to get right, and which is increasingly less useful thanks to gradual changes on the Web. Much more widely-used features like FTP and Flash have been deprecated; people howl and yell every time, but yet things still seem to work.
> This is a feature that is really not often used, whose omission is not fatal, which often requires several tries to get right, and which is increasingly less useful thanks to gradual changes on the Web.
How often do you visit small indie websites in non-English languages? Because in my experience, as soon as you do, this is not a rare occurrence.
Another example is Google search remove the ability to perform exact text search such as "some phrase I found important to search". Maybe based on statistic, it was 20% of user using that, 80% of users does not rely on this feature. The logic is saying messing up 20% of user is not a problem. We are serving the 80% of users pretty great.
The current paradigm of UX design for providing "good default" to serve 80% of users and removing customization to screw 20% of advanced user makes me pretty helpless for using modern app. That's why I prefer cli nowadays for the flexibility of those software.
If you need to read them frequently, yes. Otherwise, if this is just one-off, maybe user can download the page source, extract text content, paste into text editor and select encoding? Do you have an example CJK site I may try?
As it happens, the primary reason why the single menu item remains instead of an override being totally gone is usage in Japan. (Many people in this thread comment as if Firefox had done what Chrome did: complete removal of override as you note.) The remaining menu item is pretty accurate for its primary use case, which is dealing with Japanese legacy sites that misdeclare their encoding. (There is one exception: If the page declares UTF-8 but is actually ISO-2022-JP, then you don't have recourse.)
If the encoding is broken, it cannot be fixed by Henri Sivonen auto-detection code, but he can claim he "simplified the menus" as a bullet point to project managers.
This is the gradual but inevitable backslide of Firefox into Chrome.
Firefox already shows page-specific info in the left side of the address bar. There's usually a shield icon and a lock icon, and permission requests appear there too. Why not add an "encoding icon" that shows whenever the encoding repair feature is active? Such a marker would also be a great place to put a "override with custom encoding" menu.
When you claim that something isn’t working for you (and it’s completely plausible that you have encountered a case where chardetng doesn’t work for you), it would be polite to post the actual failure and not something that’s not an actual example of failure.
Having an automated way to solve this seems pretty good to me. Especially if the solution works pretty much 100% of time (especially for non-ASCII languages, like Japanese or Chinese) - see paragraph Document-length Accuracy in https://hsivonen.fi/chardetng/
When a website breaks because 3rd party cookies go way of dodo or the page relies on ActiveX, shoudl Firefox provide an option to run it?
I understand that some people have different opinion, but any page past 2005 should have encoding in order and older sites should fix their sites.
> older sites should fix their sites.
A lot of the reason they are considered “older sites” in the first place is because they are not heavily maintained. The thing to remember about the web is this: not all websites are “software” that is maintained. Some of them are static works similar to a movie, with static content.
> The single-byte Latin scripts are semi legible, so that is enough for me.
Did you see the example that I gave of Polish text? I am sorry, but if you feel like reading multiple paragraphs this way is bearable, then we will just be stuck in disagreement.
I did see the text. If it bothered me that much, I would write a proxy that passed HTML through iconv tool before serving it to user (with a dictionary to remember setting for each site).
But more likely, I would stop visiting such site or complain to the owner. Situation doesn't get better by covering up errors.
If there's more elaborate UI surface in order to address Latin-script cases where autodetection fails (I estimate 1% failure rate for Polish), it risks getting the whole thing removed completely even for Japanese users (as happened with the hamburger menu entry point in Firefox when the hamburger menu was rewritten, as happened in Chrome previously, and as happened with Safari for iOS in the sense of the menu never making it to iOS).
I'm, of course, glad that the guesser can work more effectively. But isn't this a bit different from something like FTP (which can introduce nasty security bugs?)? And you still end up with "change encoding of page" with the guesser, so it's like... it's not like you are removing some lower-level functionality.
It's unfortunate because yeah people mention Flash getting removed in a similar tone, but at least with Flash there are things like ruffle that can pick up the slack. I don't really want to have to install a plugin for the couple of times I land on the page that miss this encoding, but when I do hit the page I'll be frustrated.
Of course the original sin here was browsers not throwing warnings or something on every page without the encoding set, and having users become the people who complain to webmasters about it like... 15 years ago or whatever. But meanwhile we had a really _fine_ place for the encoding picker. Maybe first click is "Guess", second reveals the menu... or something.
I don't want to hold too much malice towards the devs here cuz getting yelled at by all the internet sucks. But I hope there can be some resolution here that doesn't lead to massive burnout. I will be very happy if there is some backup solution inserted (even if it's behind a conf flag), and I would see this as an overall positive to FF if that happens (listening to feedback!)
Regardless of your opinion of the change in question, these two statements are completely different from what the headline implies.
I'm not sure why it doesn't follow HN guidelines to use the page title:
"Replace the Text Encoding menu with a single override item Repair Text Encoding"
> Also, removing the encoding override is completely separate from that,
It is not separate at all. If the detector cannot do it, and the ability for the user to do it is removed, these issues are intrinsically linked. Sure, there are the other issues you mentioned around keeping it around, but you can’t act like these issues can be considered separately.
This got in the way of implementing various improvements to <meta charset> and interop improvements to XHR.
I really hope the tech industry will see the light at some point and stop this trend.
It's pretty bare-bones and probably has some bugs (after all, I hacked it up in a few hours...) but it does what it says on the tin: it lets you set the text encoding for any webpage manually, just like the good ol' Encoding menu.
But perhaps most importantly, I have never agreed with any mindset of that encourages punishing some subset of users in order to get someone to change their ways, even more so if that isn’t even the users themselves.
So you wouldn't bother to try to get the webmaster to fix the issue, you would just resolve it on your own, letting everyone else be miserable?
> But perhaps most importantly, I have never agreed with any mindset of that encourages punishing some subset of users in order to get someone to change their ways, even more so if that isn’t even the users themselves.
It sounds like you don't care if people are punished, as long as it isn't you.
I mean that I would not be able to. Huge swaths of the web are static content that was just put there and then just abandoned. The web will remain this way unless everything is moved onto SPA social networks, and I hope that we agree that that would not a good thing.
> It sounds like you don't care if people are punished, as long as it isn't you.
If the added benefit of causing people more inconvenience is that they push for problems to be fixed, I don’t think it’s fair to cause that inconvenience with the intent of using that to solve the problem. I think it’s a manipulative tactic. That is my idea of what “punishment” we’re talking about. What kind of punishment are you thinking of if you think I’m fine with punishment, and who is getting punished by it?
We are talking about the same punishment.
If fixing the pages would reduce the amount of times people would have to manually fix page encodings locally, just working around the problem leaves the punishment in place for more people.
You express that you are being manipulated to prompt webmasters to solve the underlying problem, even though many more people would benefit from not having to manually work around it.
And again, I will reiterate that this does not solve that problem anyway because vast swaths of the web are static content that was put there and never touched again, so no one will fix it. Unless we want everything to live on content farms, it is essential to be able to access content like this. You are viewing the web as something that is actively maintained, similar to a software project. That is true for a certain category of website, such as the one that we are on right now. It is not true for the kind of website which is most likely to have borked text encoding.
My anecdata: once having only a smartphone with me I needed information from a txt file in cp1251 encoding. The server was configured correctly for .html, but not for .txt Mobile browsers available on the smartphone didn't allow me to override text encoding. It was very frustrating experience and it would be enough for me to switch to another browser in future except I don't know a mobile browser which allows to select text encoding and not much worse than mobile Firefox. And I don't blame the maintainer of that site - he does (or did) this in his spare time and had more important tasks than to check text encodings for every single URL on the site so users of crippled web browsers would have no problems.
I like comparison of this feature to emergency services - you rarely need it but when you need it you need it very much.
Probably a great many because ASCII is a subset of UTF-8, and afaict is common for English when the site doesn't use more sophisticated typography such as curly quotes, em dashes, etc.
As it happens, part of what made the menu back end cause other changes to take more effort than they should have taken on their face were the security aspects of the menu in Firefox.
Try https://hsivonen.com/test/moz/never-show-user-supplied-conte... in Firefox 90 (which had the menu) and in another browser that has a menu or an extension that adds a menu.