So this is what's happening, but it's still messed up. The menus appear, but don't disappear, so pretty soon I have 20 menus all over the screen. Also no mouse pointer until I click. Also, the menu entries don't do anything (yes, I'm holding until the entry is selected, then releasing).

Anyway I tried it in vnc4server, where it does work, though you have to start it with LANG=C. But now I can't find any programs that are not working.. I'll have to use it for a while.