Chrome 97 is coming today with controversial Keyboard API feature
neowin.net
neowin.net
Edit: And the same applies to Edge, they shouldn't be let off the hook.
Then people here will say that google docs doesn't work well on firefox because google is sabotaging firefox.
I don't really use Google Docs, what does it do today with alternate layouts? My understanding was that (unless the user rebound them) DVORAK defaults for stuff like copy/paste were still ctrl-c, ctrl-v in most native apps.
Yeah that's Google sabotaging others as is usual. Why are you pretending it's not?
I want my keyboard settings to work the same way in documents and in the vast majority of the applications I run. I want to be able to query pure-text representations of state in applications, not just in documents. With some minor exceptions, I want standardized controls across most of my OS, including in web apps.
In regards to privacy, I don't want applications to fingerprint me. I have capabilities that I'd like to turn on for some apps, but I don't want them all to be on by default for every app. The system shouldn't be "you're either static content, or you can do anything you want." All of my privacy/security concerns about documents also apply to apps; most applications I use don't need access to things like fullscreen, NFC, mouse control, etc. Some do, and I want them to work, but I shouldn't need to grant access to every application across the board for that.
I think the division between apps and documents is largely fictional; it's a spectrum, not a hard line, and most "apps" are still fundamentally closer to documents than they like to admit, even in their flashy interactive forms with animations and user-state, and could benefit from the same kinds of standardization and sandboxing that documents enjoy. I don't think we should be separating out a category of websites (or native programs for that matter) where we say that anything goes just because they're "apps", I think that masks the problem that sandboxing is a UX/architecture issue everywhere across the entire spectrum.
Given the critics understand this, do they believe the required knowledge to write a native app is too obscure at this point to be considered harmful?
I say this because the measurable difference between a desktop keylogger and this is having to run the downloaded program manually. Heck I bet a lot of keyboard manufacturer’s “drivers” do this today.
On PCs, there was no fast/commercialized enough internet to worry much about spyware. What good is fingerprinting if you can't send the result to a server. What good is tracking every individual if nobody will pay you to send ads? Notice in the same way how good old IE6 with ActiveX was horribly insecure, but it didn't matter as much until the Internet itself became an ad-infested environment.
Desktop software could theoretically do a lot of harm in the past, but there was no point apart from childish damage. Viruses were more a kind of graffiti. Ransomware did not exist, there was no way to pay the ransom.
I don't like how e.g. Wayland pulls up high walls between software. Applications work best when they can easily share data, including robotic automatization, keylogging and screen recording. But it seems that innocent time is simply gone. Even major software companies shamelessly piss in the well by deploying trackers and malware together with their software. So I hate to say it but the wayland devs are probably doing it right.
No, but user expectations about safety between native apps and web apps are different.
I will say that, personally, I think that native apps place far too little emphasis on security/fingerprinting, and I think that we're kind of ignoring a serious problem with the lack of default sandboxing for native apps. I don't know that it makes sense for every native app on my computer to have this kind of capability. I think most of the native apps on my computer shouldn't be able to access the Internet at all if I'm being honest. But that's just my personal opinion, other people have different approaches/philosophies about software -- and disregarding my own preferences, I think it makes a lot of sense to worry more about fingerprinting in an environment like the web where people are running a lot more untrusted code than they normally would.
The web has kind of been a boon for native security, it's really good that users have a quick way to run code that doesn't involve download software and giving it access to their filesystem. It significantly lowers the number of opportunities for attacks. But that only works if the alternative to native apps is both:
A) useful enough that users can commonly use it as an alternative to downloading software and running it on their filesystem
B) sandboxed enough that it's not just equivalent security-wise to downloading software and running it on a native filesystem.
Figuring out how to strike a balance between those two concerns is why thinking about security/fingerprinting on the web is hard.
Going off of physical locations of keys is pretty bad UX practice for shortcuts as far as I know; we pretty much always want keys to be remappable either in software or at the OS level. Games spring to mind (WASD), but games definitely want remappable keys, and it's not clear to me that this (to use a Chrome term that I kind of like) is encouraging developers to fall into a "pit of success" in regards to gaming and user controls.
This has popped up occasionally for me where gamepads are concerned -- it's nice to be able to provide a sensible default when someone plugs in a controller. But in that situation I care more about just having a default layout then I care about knowing the specific key that was pressed. In that scenario what I actually want is a gamepad "type" string that I can just map to a list of defaults I've pre-configured for that specific gamepad type. In fact, the more I think about it, getting a key code is almost the opposite of what I want in that situation since different gamepads have different norms about what locations should do what and are shaped differently, there isn't a universal map. It's explicitly the wrong decision for me to say that the top button on the right hand side of a gamepad will always do one thing, because that's going to breaking norms on different controllers.
And it seems like keyboards are kind of the same? I sort of see the value of a keyboard being able to signal a common layout that has different default shortcuts, but I'm having a really hard time thinking beyond that why I would want to as a dev override normal shortcuts. Okay, sure, it would be nice to query that a keyboard is in AZERTY layout and suggest different shortcuts. Beyond that?
Enough people have asked for it that it must have a use, but even from a gamedev perspective this seems like a strange API to me. The privacy angle is one thing, but I also just have an instinct that this is probably not a behavior that users want from applications. They probably want Internet applications to respect their shortcuts instead of overriding them. And they probably want games to avoid forcing controls based on physical locations of buttons.
----
From the Github spec:
> E.g., If a game supports the standard WASD keys (to move up/left/down/right), then the instructions for the game need to be able to tell the user which keys to press. On a US-English keyboard, they are 'W', 'A', 'S', 'D', but for French (AZERTY) layout the user would be told to use 'Z', 'Q', 'S', 'D'. [...]
> Some applications make use of keyboard shortcuts that are based on the position of the key rather than the symbol that is generated. [...]
> E.g., a drawing app may have a number of drawing modes arranged from left to right on the screen and may wish to have keyboard shortcuts that correspond to the screen position.
I don't know, maybe I'm being unimaginative. But I've dealt with at least the first problem here, and I don't think that keymaps are necessarily the best way to solve it. Also, if we're talking about games we should be coming up with a solution that maps to controllers as well, so I don't really get why we're going down this specific route. And the second two use-cases seem like user accessibility bugs, not features. I guess enough devs want it that Chrome is implementing it, but I feel like there's something I'm missing.
As a user, I want a user-controlled (or at least OS-controlled) layer between myself and app shortcuts/keycodes; it's super useful and I customize that layer all the time, even for simple things like rebinding caps-lock. And as a dev I'd rather not have to try and intuit what setup a user has based on physical locations. Heck, maybe someone toggles into a different layout to game and they use IJKL to move around when they're playing web games. Maybe their W key is broken and it's bad for me to try and force them to use that specific physical button. I don't want to have to think about any of that as a dev, just have the OS signal to me what keys they want me to think they pressed. And if they can signal to me a common string for the overall layout that allows me to give them a sensible default, then great, I'm happy to do something like that. But this feature doesn't seem to make any of that easier, it just gives me a granular map for individual keys and then makes it my problem to interpret.
:shrug: I don't know, maybe there's some nonsense with native behavior that we're trying to match or something. I customize my keyboard layout a lot, but I've never jumped to DVORAK/AZERTY, maybe there's a lot of weirdness with shortcuts there that native apps expect to just paper over, where they bypass the layout? But I'm not really seeing that online?
Curious how you solved the WASD problem not if not by using a keymap in some way?
> we should be coming up with a solution that maps to controllers as well
The gamepad API already normalizes axes/buttons.
> As a user, I want a user-controlled (or at least OS-controlled) layer between myself and app shortcuts/keycodes
That'd be great but there is no reason this couldn't be pursued as a separate layer on top of this instead of something that has to block it. I doubt it'd get the same support though as it'd require a lot more "thou shalt do it this way" rather than "you can call this if you need" and such changes are much harder to get support for.
Short answer, I just give the users an option to change their keyboard layout and use arrow keys/mouse as a backup for menu traversal in case they set their bindings weird and get stuck. For apps where that's annoying to prompt, I still provide the option but default to assuming QWERTY layout unless I see some signal (language options) that tells me I should more explicitly ask.
Longer answer, I don't have hard stats and this is mostly just based on my intuition, but my feeling is:
- QWERTY tends to be a pretty safe default for most US users, DVORAK tends to be either uncommon or the people using it already have custom maps set up, which I don't want to mess with.
- Other layouts tend to be associated with other languages, so using `navigator.language` seems to be a decent proxy? At the least, if `navigator.language` says something other than "en-us", at that point I might decide to surface a prompt to pick a keyboard layout.
- Users would rather be able to select their controls from a list (ie, have some keybinding "templates" or common schemes that players can toggle on) and then customize them, rather than have to build their control-scheme from scratch.
----
I do think it would be easier to have better defaults if I could look up the base layout from a list of common values and default them to a layout that's likely close to what they want. Something like `navigator.keyboard.getLayout` would be useful to me, similar to `id/mapping` on a gamepad. There are privacy risks there too, but I see the use case at least. But that's not really the same thing as wanting a map of every key, a map of every key is less useful to me than a string that says "DVORAK".
It's also important to understand that displaying the key on the screen isn't actually the problem here; I don't really understand what the Github repo is talking about in that sense. Without the new API, the prompt you display on the screen will always be correct 100% of the time: if you tell the user to press the "W" key, they'll press the key they associate with "W". So the problem is not "how do I tell the user which key to press", it's "which key should the user press?"
And in that situation all you're really looking for is the ability to set sensible defaults; I can't think of an example where I've wanted to be prescriptive about "the user must use their left hand to navigate, and it must be in the location of WASD". So I don't really know what I'd even use a full keymap for, the most I want is a signal for which pre-defined default keymap I should load up. I guess I can use `getLayoutMap` to do that, but it's clumsy and inflexible and requires extra coding: it's kind of an inferior solution to just giving me a layout string that I match against a table.
Matching against a table based on ID/layout is also what I already do for controller defaults, so it's annoying to me to have a separate system somewhere else.
----
I mentioned "pits of success" above. There are a few traps and good practices when working with keybindings (opinion me). Some bad outcomes:
- Not having rebindable keys or alternate layouts that the user can set.
- Not respecting user-rebound keys (my caps lock is escape, I want that button to register as escape in everyone's web app).
- Showing inaccurate information onscreen, telling the user to press "W" when you mean something else. Note that this is impossible to do without `getLayoutMap`, and very easy to do with `getLayoutMap`.
Some positive results:
- Having better default keybindings, as long as they can still be changed
As far as I can tell, giving devs the option to ignore user keymaps and to use physical key positions for bindings makes it easier for them to fall into the first 3 traps, and only marginally helps with the second positive result.
----
Anyway, second TLDR, have some common layouts in your key rebinding options that people can quick-select (you probably should have this for WASD/arrow-key toggling anyway, not everyone on QWERTY layouts likes WASD). If you're worried about defaults, use `navigator.language` as a signal for whether you need to be more proactive about asking for the keyboard layout up-front.
I also don't really understand what the Github repo is talking about when it says that displaying messages is hard, because as far as I can tell messages about what key to press are the only part of this that are currently impossible to break, and I'm a little annoyed Chrome has now made it possible to break them.
Again, maybe there's something I'm missing, I assume devs have some reason for asking for this. I just can't personally think of what it would be, it seems like really bad practice to be prescriptive about what keyboard controls people are allowed to use, and I don't know what else this feature is good for.
----
> That'd be great but there is no reason this couldn't be pursued as a separate layer on top of this instead of something that has to block it.
I think my perspective is that I already do have this, it's my keyboard layout. Why are we taking away something that already exists and then reintroducing it as another separate layer that's optional? As a user, I want my keyboard customization to work in all apps, I don't want the developer to be able to override my preferences just because they don't feel like using them.
It doesn't make sense to me that we'd get rid of a user-defined map that's currently working today, and then reintroduce it later.
The point of this is to fix that assumption for the cases it is wrong as well as simplify that process. Rather than have the app assume things about language, country defaults, and non-customization to try to select a hardcoded keymap this API just lets the app reference the actual keymap directly without requiring the user to do anything. Either way though both methods are using a keymap even if you're trying to hide that's what you're really doing. This new way is however easier for app developers, requires less action from users, and more accurate than trying to heuristic your way to the keymap. Similarly:
> Users would rather be able to select their controls from a list (ie, have some keybinding "templates" or common schemes that players can toggle on) and then customize them, rather than have to build their control-scheme from scratch.
It's nice if this list can show the letter of the key, particularly once the user has left the config menu where you can reasonably show a full picture of the input device i.e. in use e.g. https://i.imgur.com/ZwVjubs.jpg
I still argue the entire set of rebinding points are an orthogonal issue. While rebinding is a useful feature it is neither gated by nor gates exposing the current keymap and fixing either individually will not provide the full advantages of having each.
> I think my perspective is that I already do have this, it's my keyboard layout.
As someone who actually uses a non-language default keymap I'll say it becomes immediately apparent I don't want my keyboard layout to have anything to do with the keymap (besides notification of what the key to press is) in a lot of cases. Of course there are many such as ctrl+c or cmd+q I want to stay the same letter regardless of location as well but exposing the keymap doesn't inherently force anyone to override that even if it allows it. It does however let the apps function fully for the first case.
I think there's a really tangible difference between saying that AZERTY layouts prefer to use ZQSD for layout and saying that everyone is going to use the four keys on the left-hand side of the screen.
It's sort of the same situation as controllers -- using physical maps to standardize controllers is just flat-out wrong, because different controllers have different conventions about what buttons in the same physical locations should commonly do.
It's fine to say that there are common layouts that have defaults, but I think it's a much larger step to say that shortcuts/keybindings should be defined by physical location rather than convention. I don't completely see how that makes thing simpler for me as a developer (I don't want to intuit keyboard layouts by iterating over keys), and certainly I don't want it as a user.
----
> It's nice if this list can show the letter of the key, particularly once the user has left the config menu where you can reasonably show a full picture of the input device i.e. in use e.g. https://i.imgur.com/ZwVjubs.jpg
I have some other objections to your points below this (allowing applications to override user preferences is in fact a loss in user functionality, even if developers aren't forced to ignore those preferences), but I want to really focus in on this specific example, because you're not the first person I've seen bringing this up as a problem that needs to be fixed. And honestly, I really don't understand at all what people are talking about with this "fixing keymap messages" thing. I don't think I've ever had a problem with this[0].
Under the current scheme, before Chrome introduced this feature, it was impossible for a developer to mess this up. If you told the user to press "Z", and they pressed the key that they normally press to get "Z", no matter what their layout was or what their keyboard looked like your app would register a "Z" press. Whatever key you told them to press, if they pressed it you got that key. Perfect, impossible to break.
But if you now are basing your prompts on a keymap that bypasses the user's keyboard layout and uses the physical position, and still labels those keys based on a QWERTY layout, then it is now possible to break the message and to tell the user to press the "Z" key, and then to have the "Z" key not work because what you're listening to is the physical location of the "Z" key on a QWERTY layout rather than the key that they use in their layout for typing a "Z".
So I don't really understand what people are talking about with this objection; Chrome isn't fixing anything about user prompts because nothing needed to be fixed about matching user prompts to keybindings, it worked 100% of the time. The new feature actually breaks that and makes it possible to now implement user prompts that are inaccurate.
I believe people who say they have struggled to implement this in the past, they wouldn't lie to me, but... for the life of me, I can't imagine how they're implementing keybindings then; I'm not sure what I could do architecturally to make inaccurate user prompts about keycodes into an issue for a webapp.
----
[0]: Okay, that's not quite true. I did have a problem with this once on a controller that mapped keys based on their physical location. The only time this has been a difficult problem for me to solve was when I was trying to figure out controller layouts and I realized that the letter painted on the controller button wouldn't consistently match up with the keycode across controllers.
To the second part: say you have a positional input set in the game, e.g. "WASD" except which letters it actually is don't matter (because of either preference mapping or keyboard layout) just that they are positional inputs mapped to movement (up/left/down/right for your average WASD user), in such a case you also have positional support keys e.g. in an FPS commonly "action", "grenade", "strafe", "crouch", and so on. A sane default will usually have these all clustered together and laid out in a way the user can press common combinations easily (worth noting sane defaults aren't universal, what is good positioning for a particular binding in one game may be poor positioning in another that plays differently or has different mechanics) and a good game will allow these inputs to be remapped (be it for personal preference or accessibility). For the default mappings if you say "press F to pay respects" without an idea where "F" is because you don't have the keymap for the user either one of two things is going to happen when it's not the same as the developer's keymap:
1) You're using scancodes and the user actually needs to press "Q" or another random letter but you don't know how to tell them that because you don't have the keymap
2) You're using keycodes and the default mapping is completely garbage for the user. This will commonly lead to... 2a) The user goes to configure a custom positional mapping but because of 1) when they try to configure a custom mapping for the "Q" on their keyboard it shows "F" in the mapping (though works as expected) making custom mappings a PITA to configure.
The way keycodes help the developer in these instances are:
1) You can display the actual letter for the default mappings, regardless if the user's keymap matches yours/your assumptions 2) You can display the actual letter for custom rebinds both in config and gameplay 3) If the user changes from DVORAK to QWERTY neither the developer or the user need to change anything for it to keep the positions/display the right letters
.
Now can this be misused to break things? Absolutely, a bad developer could misunderstand when to use keycodes vs when to use scancodes and in turn recreate a the problem class from 1). At the same time on native apps I can't say I've ever run into that vs the deluge of games and apps with positional bindings (typically content creation) that assume I'm QWERTY and look for the keycode. As for Office desktop as an example the character bindings still follow the character, e.g. I press ctrl+v to paste (which would be ctrl+o on a QWERTY layout) even though it has the ability to query the keymap for positional shortcuts.
Generally the only reliable thing is if Games never use the physical key layout and just provide the tools to manually rebind the keys if necessary. At least for Colemak, I generally find rebinding games to use the WARS keys not too much effort from WASD. (Though admittedly, I also tend to favor/default to gamepads on the PC these days, a small part because I don't have to rebind keys to play.)