Neverclick: Desktop application for performing mouse actions with your keyboard

I wonder if blender works on wayland, because it has a feature that makes the mouse cursor "loop" across the window if you are dragging something.

There is mouseless that seems to work on wayland

https://mouseless.click/

This is why I'm still on X11, the lack of deep level accessibility/automation is still a big hurdle when it comes to Wayland.

Actual problem on Wayland is that it often doesn't work when you are trying to click somewhere in the menu - for the mouse clicking app to read keyboard input, it must take focus away from the app where the click should go. And some toolkits just close menus on focus loss.

I personally use wl-kbptr, and the problem above is the only one that can't be solved without explicit support from the compositor.

What about wp_pointer_warp_v1? That should be supported in some compositors. But not sure how to actually "send" the click event.

I often try a session with Labwc/LxQt instead of my Openbox/LxQt, and while there is progress, still not fully usable, so still no Wayland for me. But with X11, I can at least add xvfb, xdotool, and other utils, and AI has no problem writing (and testing) a GUI for me.

May I interest you in this:

https://github.com/uukelele/waydotool

I think you can get the source directly from github, probably even a pre-compilled installer from the releases page in the github project.

It seems to implement the Avy (https://github.com/abo-abo/avy) model of input. It can't be improved very much, because when dealing with dynamic data, you can't make things repeatable. The idea of "muscle memory" for things that change is kind of fallacious. The best you can do is ensure some predictability in the subset of keys you're using for element selection.

Neverclick dev here. It's not random, a lot of thought went into how the key combinations are generated and assigned.

When Neverclick generates the key combinations, it prioritizes keys in the center of the keyboard. It then sorts the key combinations from most to least convenient (so for example JJ and FF would be at the top of the list). It then assigns the most convenient key combinations to UI elements closest to the center of the screen because that's where people are most likely to want to perform a mouse action.

As the other commenter already said, with dynamic data, you can't really make things repeatable. However, it's still really cozy to use and way better than if it were randomly generated and assigned (which is how it used to work before it was updated to the new system).

What is needed is "Mouseless Friday" for UI designers/developers.

Hello, I'm the developer behind Neverclick. Neverclick doesn't use accessibility apis. It uses local computer vision so it works in every app.

I've had a poor experience with accessibility apis, they're clunky, slow, and unpredictable, and as you said many apps aren't built with accessibility in mind. With computer vision you don't have to worry about that.

This might sound strange, but the cv system in Neverclick actually runs significantly faster than UI Automation (which is the accessibility api on windows) in nearly all of my personal tests. I can't believe that accessibility apis are so poorly optimized that raw pixel analysis is faster lol. Also, I have users that use Neverclick on 10 year old hardware and they tell me that the cv runs instantly whereas UI Automation is super laggy for them.

Also, accessibility apis typically don't give you useful bounding boxes for text in text editors, this is another advantage of the cv approach.

I really wish the title of the post was "Desktop application for performing mouse actions with your keyboard using computer vision" so that it's more obvious that it uses cv, although it's my fault since the current title is how I have it in the readme and they probably just copied that.

The commit history is a mirror of the real codebase, I wanted to show people that I've been working on this for a long time and that it's not vibe coded ai slop. In retrospect it seems that it was a mistake to do this.

Neverclick is digitally signed with Microsoft Trusted Signing (now known as Azure Artifact Signing). I had to send them my address and ID to get my identity verified so that I'd get approved for it.

Is there anything else I can do to put your mind at ease?

My reading is that it uses CV to identify potential click targets on the screen.

It uses CV to analyze the pixels on your screen to detect clickable UI elements.

Presumably it captures a screenshot and then uses CV on the resulting image to "see" the application and identify the various things that you might want to click.

The commits are a mirror of the real codebase. My intention was to show people that I've been working on this for a long time and that it's not ai slop abandonware. But after reading some of the comments here I can see how it has raised more questions than it has answered.

Yeah after I read this I looked at the "code". It's a lot of placeholder.cpp with int main(){return};

This is so very sketchy.

I am guessing wp_pointer_warp_v1 in Wayland, but not sure if all compositors support that. And that is just for moving the cursor. Can we "send" a wl_pointer click like that?

Most software in this space relies on accessibility apis, which means it only works in apps that have accessibility support. Neverclick uses computer vision. There are several advantages to using computer vision, the biggest one is that it works in every app. It also produces really good bounding boxes around text so it's great for editing text. And counterintuitively, it runs significantly faster than the traditional accessibility api approach. The hints show up instantaneously, even on old hardware.

The Mario Kart youtuber?

I mean, it can make mouse movements and clicks but you can't be suggesting it can do on-screen hints, which makes it a way better experience.

thanks for the heads up, I have a keychron gathering dust at home, so I'll test this later on. favved the thread anyway, since I'm interested, but will wait for the macOS version.