Warpd: A modal keyboard-driven virtual pointer
github.com
github.com
Rather than try to simulate the moving the mouse itself, Shortcat [https://shortcat.app/] indexes the user interface (buttons, text fields, links, menus, etc) and enables fast fuzzy search of the interface. Type a word, abbreviations, or hints and hit Enter to click or action the element. Works almost everywhere on macOS, including browsers, Electron apps, and even iOS apps!
The goal is to minimise cognitive overhead to achieve a particular intent, so being able to type a word to hit a button, or active a deep menu item when you don't know the shortcut is quick and easy.
I'm currently working on a modal option which enables staying within Shortcat to navigate an interface, as well as chords for simulating scrolling and arrow keys.
Shortcat relies on using the Accessibility API to index UI elements however, and is dependent on how well an app or website has implemented it. One of the goals is to help improve accessibility implementations by exposing more people to its implementations and pushing for developers to fix broken or incorrectly implemented accessibility tagging.
Shortcat is macOS only for now as I haven't been able to investigate how viable doing this on Windows or Linux would be, especially on Linux considering all the different toolkits that exist.
Thank you for your hard work!
Thanks! It’s easily my longest running project at a decade
I did charge for it a couple of years ago, however I rebuilt the whole thing from scratch after a long hiatus and hadn't had bothered to reimplement licensing because the existing options all kinda suck, and figured I'd focus the time on features and usability first. I think with the modal mode in the next release will bring it much closer to a 1.0 release.
edit: nvm me, found the option in settings (on activation show shortcuts immediately).
Quick question, I've been playing around with Shortcat for a while. When I press the activation hot-key it takes about 4 seconds for the yellow two-letter denoated highlights to show up, despite the app's text stating "found n elements in ~0.20s". Is there a config option to instantly show the yellow highlights?
I see you found the setting for that. It was a deliberate default initially as the intended way to use Shortcat is to activate Shortcat and type what you want without waiting to see hints, as this is generally faster and less mental overhead IMO, especially for fast typists and well-structured interfaces.
However, some people prefer minimal keystrokes and I get that. I'm trying to figure out the right set of defaults to make it friendly to new users while nudging people to how Shortcat is designed to be used and will be tweaking it as I go.
Given that Linux doesn't have anything like an accessibility API, I think the only option is training ML models.
I love Shortcat's approach in general, indexing the UI. However, the reliance on the Accessibility API is actually a significant downside in the real world in my experience since so many apps don't properly implement it. I feel like Warpd is a good complement to this, you could use Hint or Grid mode as a fallback when the indexing approach fails.
I wish I could use shortcat or Warpd, but unfortunately I'm on windows. Curious if anyone has any good tool recommendations for windows? Currently, I'm using:
1. Vimium for Chrome (so good, wish I could just use it across the OS).
2. Hunt and Peck: https://github.com/zsims/hunt-and-peck has been my favorite for OS-level use, a simple version of shortcat for windows. But, it's not maintained and not as slick as some of these newer tools.
I would probably need to pay someone to build that particular version though cause the last time I built anything for Windows was like 15+ years ago
It does basically cut out the mouse, and had a several-days learning curve for me, but after that it's pretty great. Here are some cool features, off the top of my head:
* Python-scriptable, though I haven't figured out how to use this yet.
* Bind javascript bookmarklets to a keyboard shortcut (use :bind with the jseval command)
* Toggle not only javascript, but image loading and a whole slew of other features, with a keyboard macro.
* Vertical tabs.
* All config is adjustable via commands.
* Keyboard macros like "pop tab into a new window", "clone tab", "close all other tabs", etc.
* Text selection using the keyboard.
* Quite similar keyboard dynamics to vim.
It has a built-in ad blocker, and you should run :adblock-update when you first use it.
Another browser which is similar, but which I haven't gotten into as much, is Luakit.
I think this paradigm along with more app developers putting all the important functions in menus is a strong contender for Maximum Intuitive Productivity
Thank you for Shortcat, I used it a long time ago and loved it. Excited to giv it another go!
Alt is Meta on modern keyboards. You can use ESC to emulate Meta in some applications. But this doesn't work here.
It should say A-W-x (alt+super+x) Super is Windows key these days.
https://en.wikipedia.org/wiki/Space-cadet_keyboard#/media/Fi...
- X
- Wayland + Gnome Shell
- Wayland + wlroots-based compositor
- Wayland + sway or any other non-wlroots based compositor
Is Wayland really that much better that this is worth it? Why can't Wayland be aware of the compositor like X?
I don't use it at the moment because of compatibility issues with some software that I use, but that's not Wayland's fault.
I mean, that's the whole biggest criticism of wayland, 15 years in and it still doesn't work (well?) with the biggest GPU brand. The criticism has always been about whether it is worth it to fragment linux land and throw away a few decades worth of work on X.
It's not just throwing away a decade's worth of work on X, it's making everyone redo that work per display manager (thankfully wlroots exists as a kind of Wayland shared library).
Once the the need is understood well enough, a common extension will be written, and Mutter, KWin and sway/wlroots will implement it.
This is better for security than X11 free-for-all.
Like how they standardized nice simple things like taking a screenshot?
Full disclosure, I am a Wayland skeptic. I don't think your focus on X input security is as justified as you probably think.
I know in the Windows world, one of the UAC features was that a less privileged process can't send events to an elevated window.
In X11, I think last I checked most distros disable the XTEST extension by default out of security concerns. Skimming the warpd code, they are using XTEST for the X backend.
As I think of the keylogger problem, it's not really privilege escalation, is it? If you're running as the same user as all the other clients, you could ptrace(2) them and intercept their event loops. I guess there are some container-based app deployment solutions now where you could run stuff at different security levels, so maybe it's more of a legit issue now...
And most likely it won't be injection of mouse and cursor events, but something higher-level, like focus switching requests.
This one looks like a small feature in a compositor and not an external tool, really.
I guess it would take 100-200 LoC to implement in GNOME Mutter.
Excuse me for being blunt. I don't know if you understand how shitty of a design you advocate. Solid designs do not require modifying core components to write application level features the original authors did not envision.
Moving cursor around is a compositor's domain, not some arbitrary application's that decided to fiddle with the user's input.
An API should not be so preachy about which programs can theoretically be written. It should provide broad mechanisms.
It is very frustrating to work with people who think like you do, that 3 or 4 unrelated projects have to carve up narrow exceptions to how the platform works for every single use case, nominally because of theoretical harm of this exploit no one will write, but actually more based on your ego perception that you know better than every other developer on the planet.
So Wayland has this long list of impossible applications which are doable everywhere else. It's a prima donna.
https://github.com/rvaiya/warpd/tree/master/src/platform/way...
If there is a distinction, and there might be, "completely separate" is exaggerating.
Edit: I see that only wlroot based implementations are supported so far, and of those there are some things broken in wayfire. Perhaps this is what you're referring to?
A few days later I wanted to add custom macros and had to read the docs. Skimming the warp section made me realize it's basically just recursive space positioning. I tried the mode again, and soon realized how incredibly useful this is.
It's a bit difficult for me to keep the state in my mind when going down the tree because there's no visual indicators. It's a keyboard firmware after all.
In fact, historically AHK is actually a fork of AutoIt.
I have been using something called keynav[1], for getting a similar grid mode. I would never guess how intuitive it is.
It doesn't replace the mouse, but it's helpful for that occasional click in the middle of heavy keyboarding.
In the 1980's, there was a thread of animosity directed at GUIs and mice as productivity-killers and providing accessibility to novices that robbed power-users of expressivity and automation as features shifted towards UIs over text mode applications. I think we can agree that with necessary and sufficient software engineering and UX, CLI-UI-API parity is achievable offering an easier learning curve, varying levels of user astuteness, mental models, and expressivity to accomplish a task by having different MVC "views" or "presentations" to interact with software or systems of any sort.
I just wish I had an easy way to do that from the numpad. That way, to move the mouse to an arbitrary location I need it to be, I could type 19432 enter and know that corresponded to the coordinates to refresh the page I am reading, that way I could use the mouse less and less as I started to memorize the 80% case of where I need the mouse to go and just bang it out on the keyboard.
I don't understand why more people don't adopt it. Is it because it's so different from a normal mouse ?
On my thinkpad, I use the trackpoint and trackpad equally.
Sadly every non-IBM/Lenovo trackpoint I've ever used is awful (although significantly improved by putting a Thinkpad cover bit on the joystick, if you're stuck with one).
That said, having played through the SC1 campaign with a trackpoint... even the best ones are not as good as a real mouse.
But why someone would intentionally make things more difficult for themselves as a preference, I don't get. It would be like walking around in crutches when you have two perfectly healthy legs.
As a more common example, I only launch applications by opening a prompt (e.g. Spotlight on Mac) and typing the first couple letters of the program I’m starting. This is much faster than navigating using the mouse to the applications folder/menu/dock/taskbar etc. and clicking an icon.
I agree keyboard-based navigation is not faster for everything. Luckily, tools like this don’t prevent you from also using a mouse!
There are times when a mouse is good, there are times when I don't want to take my hands off the keyboard and mouse for something.
No-one would do that, that would be crazy. People intentionally make things easier for themselves as a preference, and different people find different things easy or hard.
So it's less like crutches and more like rollerblades.