It took me quite a while to figure out that this was because I had set my keyboard for the logon screen to Cyrillic. I confirmed by overwriting sethc.exe with cmd.exe and enabling high contract - the text I typed in the command prompt window that opened was in Cyrillic. (Not my own trick, but a very useful one for recovering access to Windows without directly editing the registry files!)
That's quite a downside, I would say! Turns out all those Microsoft warnings that say "do not modify the registry unless you know what you are doing" had a point.
I had deleted the keyboard and the Canadian English language, with the system set to to US English. Now this keyboard suddenly came back, with no way in the UI to delete it. I had to first install the Canadian language. Then I was able to remove the keyboard.