Mouse Pointer as a Mere Mortal | Hacker News Reader

Mouse Pointer as a Mere Mortal

unsung.aresluna.org

95 pointszdw48 comments

I would not be happy with mouse pointer hijacking. Seems to belong in the same territory as scroll-hijacking but worse. The example case here could have been served by simply highlighting the area of interest in the UI with a red circle or a flashing pointer, whatever does the trick -- even though that may be distracting too.

There are a lot of interactions on a PC where user inputs land in the wrong place.

Claude Code and Codex in their various avatars allow us to type the next prompt while the agent is still working and responding on the earlier one. But this constantly runs into a permission prompt from running session -- either interrupting or worse entering a response to the permission prompt unintentionally.

Even during normal prompting slash commands interfere annoyingly with normal use of the slash key (i use a slash to indicate a list of two or more choices sometimes when i write).

Permission popups and confirmation dialogs that appear unexpectedly and swallow our keystrokes, spacebar and enter key hits mid sentence have always annoyed me.

Laggy devices, and resource hungry sluggish UIs compound this problem.

From the counter argument:

> as a matter of fact, there is something really interesting about a mouse pointer feeling less like a deity floating above it all, and more like a regular in-game actor.

My counter-counter argument would be a general principle for UX designers: Are you designing a game or a tool? If you're designing a tool, don't put cutscenes in your software.

I think games are special, because their explicit purpose is to deliver an experience and often also tell a story. Within that context, I'm fine with having control restricted or yanked away if it's in service of something meaningful in the game.

The same is not true for tools (even in-game tools actually), where I want to complete some kind of task in the most efficient way possible - and often only I know the context of that task.

Unfortunately, that stuff has already seeped into UX design in a lot of forms, in particular as random "new feature" popups that usually appear at the worst possible moment and cannot be shown again. In situations like this, I'd value predictability much more than the coolness factor of a game-like UI.

> often only I know the context of that task

This doesn’t account for cases where the system is semi-automatic, such as a tiling window manager (you don’t set all the dimensions manually). If the automatic part is to stay at all, it should make certain assumptions for the user.

Even in the context of game development, if you don't provide a "skip cutscene" interaction, some players get very upset. The desire to tell a story can be at odds with the player's desire for control, even there.

Software moving the mouse cursor is only acceptable when the window is full-screen. If the user makes an application go full-screen, they are opting out of the normal desktop UI conventions. It's expected that full-screen software completely takes over the UI, and there are legitimate uses for moving the mouse cursor in full-screen software, e.g. centering an invisible cursor every frame in a first-person shooter game so endless view rotation is possible. But if it's windowed then it should be impossible.

> But if it's windowed then it should be impossible.

I worked on several apps for the visually impaired that automatically move the mouse cursor to different UI elements in the front-most application, regardless of the window state. It’s a good reminder that “impossible” often just means “I haven’t accounted for that use case yet.”

If it's part of the OS's standard accessibility framework then it's acceptable. The important point is that applications shouldn't be able to arbitrarily move the mouse in situations when it's unexpected.

> The important point is that applications shouldn't be able to arbitrarily move the mouse in situations when it's unexpected.

That is quite a different statement from "It should be impossible." What should be impossible is for the OS to prevent this type of usage when it is clearly useful. Beyond accessibility, I use these features to automate testing of native macOS GUI apps.

Coming from Linux, the accessibility framework is just another series of programs. My main a11y program is a tiny little binary that uses the keyboard to move the mouse around at will; I certainly don't want the system to try and restrict that.

You are arguing for uniformity. It does make a lot of sense: the global UI makes a considerable effort to build a single perfect UI, but that can only work if the apps actually make use of it.

But why shouldn’t the global UI itself make use of mouse warping?

Blender (3D modeling & animation software) implements this cool thing when rotating/resizing objects: if the mouse cursor moves out of the window it reappears on the other side (enabling resizing/rotating ad infinitum).

I think a better way to implement that feature would be a mechanism for programs to temporarily enable off-screen mouse cursors. This should also track the position where the cursor would be if it had been clipped to the screen boundary as normal, and immediately return the cursor to that position when the off-screen mode ends. Note that the OS returns the cursor, not the application, so applications can't abuse this mechanism for repositioning the cursor.

I don’t find that better. Why would it be? Now you don’t see the correlation between the movement of the cursor and its in-app effect.

It's better because it's the minimum change to mouse cursor behavior that allows the feature to work. You don't need to see the cursor while it's off-screen because the point is manipulate the 3D object, and you can look at the 3D object instead. The same is true for things like controls in an audio DAW which might also benefit from off-screen mouse movement.

If there's really a case where you need to see the exact position of the cursor while it's off-screen, you could display it wrapped around only while it's actually off-screen. But this would potentially confuse new users, so it should be optional and disabled by default.

> You don't need to see the cursor while it's off-screen because the point is manipulate the 3D object, and you can look at the 3D object instead.

Disagreed. Seeing the cursor at all times gives you some point of reference, and once you release the tool, you know where your cursor is.

> If there's really a case where you need to see the exact position of the cursor while it's off-screen, you could display it wrapped around only while it's actually off-screen.

I don’t understand what this means. If it’s not off-screen then it’s automatically also not wrapped around.

> But this would potentially confuse new users, so it should be optional and disabled by default.

This presumes that “cursor is suddenly allowed to be off-screen and not visible” is less confusing.

>Seeing the cursor at all times gives you some point of reference, and once you release the tool, you know where your cursor is.

Seeing is an inferior means of knowing where the cursor is compared to intuition. When I move the cursor, I know where it is with no conscious effort because I treat it as part of my hand. I disable mouse acceleration to make this easier. I don't need to look at my hand to know where my hand it. My subjective experience of mouse clicking is the same: I look at the target and the mouse cursor automatically appears there. If you allow software to move the mouse cursor you weaken this intuition.

>I don’t understand what this means. If it’s not off-screen then it’s automatically also not wrapped around.

When the cursor moves off-screen, it could be displayed with position modulus the screen width/height. Additionally, the cursor shape could be changed to make it obvious it's not the true position. This might make sense if you really need to know the exact off-screen position and the GUI control you're manipulating doesn't provide sufficiently precise feedback.

>This presumes that “cursor is suddenly allowed to be off-screen and not visible” is less confusing.

It is less confusing because other than extending the range of the mouse off-screen, the mouse behavior doesn't change. As soon as the off-screen action finishes, the mouse cursor snaps back to the position it would have otherwise been in.

An alternative option would be to snap back to the position it was where the special off-screen mode was initiated. This might actually be better, because it makes the off-screen mousing mode an extension of moving the mouse while it's lifted off the mouse pad, which users already have intuition for.

> Seeing is an inferior means of knowing where the cursor is compared to intuition. When I move the cursor, I know where it is with no conscious effort [...]

Realize it consciously or not, visual feedback is a critical part of this loop.

> As soon as the off-screen action finishes, the mouse cursor snaps back to the position it would have otherwise been in.

The cursor jumping to the edge of the screen, which is not somewhere the user ever saw it and may be outside of the application, seems worse than any current issue while still being insufficient for most legitimate use-cases.

I don't really see any fake cursor approach that isn't going to behave awkwardly in practice - e.g: is it your real (invisible?) cursor or fake cursor that can click to focus another application, and what happens to your cursors when you do so?

Just letting the user deny mouse control for an app (like on Wayland) seems sufficient to solve your annoyance. Maybe adding a separate permission for control while unfocused, since that's rarer. No need to break all windowed applications with reason to capture/move the mouse.

I generally never want programs to go fullscreen because I like to keep taskbar shown, so I can keep track of time, notifications and whatnot.

Well designed video games that rely on fast and precise mouse input capture the cursor during the gameplay until menu is shown.

The only times I have to go fullscreen is for the games that fail to capture the cursor and where accidentally clicking outside of the game window leads to a loss.

Can't imagine a non-game program other than a video player that I would want fullscreen.

> But if it's windowed then it should be impossible.

I have one monitor, so fairly often have games/editors windowed with something else alongside them (a video, documentation, …). There are also uses where the mouse is only captured temporarily - like FPS-controls flying mode in Godot and Blender. Some image editors also allow for things like moving the cursor with arrow keys, which I find useful.

If any program I used moved my mouse pointer regularly, I'd quit using it. This is right up there with programs that move UI elements around or pop them up as I'm trying to interact, causing the wrong actions to occur.

Windows has a “snap to default button“ setting which does the same.

Saves you a bit of movement on large screens, but since it jumps it doesn’t lead the eyes which makes is disorienting.

> ...but since it jumps it doesn’t lead the eyes which makes is disorienting.

What happens when you enable "mouse pointer trails"? Or is that a feature that died like a decade or two after manufacturers stopped using the extremely slow LCDs that made use of the feature all but mandatory on machines that used them?

That could actually make it work -- but I just tested and, sadly, the cursor still gets teleported without a trace.

Sad! I wonder if more than a handful of people at Microsoft know that the "cursor trails" feature still exists.

I’ve never heard of that feature in Windows! Unfortunately, that means I can’t judge it.

But I have used mouse warping in other environments and I’ve never been caused disoriented. It never occurred to me that such a problem is possible. In general, it seems like this problem can be avoided by 1) moving the mouse only to changed states of the screen, 2) only if they’re small enough to be easily observable, and 3) only for repetitive tasks.

> But to me the challenge is that it’s hard to imagine everything that needs to be preemptively captured and prohibited. I have to imagine this stuff for living, and I literally did not think anyone would just move a mouse pointer like this.

In requirements engineering, you do only codify things in the positive sense. The reason behind is that the system shall only do things that are described. If there are no things described, the system shall do nothing.

You can forbid a lot of things, but then you would have pages upon pages of requirements telling you things you may not even wanted implemented in the first place.

vim-athena would automatically move your cursor towards the command buttons whenever it made a popup appear

i thought that was genius, until i upgraded to vim-motif, which would instead move the popup to where your mouse cursor is

In the early days it was pretty common to move the pointer to the active element when one started navigating with the keyboard.

But yeah, it feels like somebody physically grabbing your hand and moving it.

I hoped for a moment this was a mouse pointer replacement that adds a tiny human character, who starts the day with energy and vigour... only to become more decrepit and worn out as 5pm approaches.

I guess f.lux could be considered similar?

An even worse, or at least equally bad, UI fail in that Lightroom screenshot/gif: I can't even tell which buttons are supposed to be clickable and which are disabled/grayed out.

The effort put into making sure you know how to turn this feature on makes me question why it's so important to them, is the 3rd party paying them for this data?

Even major features in Adobe apps the furthest they go is those video popups rendered using webviews so they glitch into existence as a white box.

I work in a text editor that relies on mouse warping heavily. It doesn’t feel wrong. The opposite is true, it makes otherwise mundane tasks fast. It takes the pointer to the newly created windows (if you have a dozen of windows, that matters), it makes it easier to locate currently selected text (again, having a dozen of windows makes it necessary) and get to the next occurrence, to shove a window together with the pointer so that you don’t lose the control over it, and more. Sometimes it even takes the cursor back after a short temporary action.

I truly miss this level of detail in the outside world. It feels as if you’re resizing a window but the mouse pointer doesn’t move.

The Lightroom example is nowhere close to that. Looks like a thoughtless optimization.

The most intriguing case for hijacking is in Neal's "cursor camp" game linked by the author. In my own designing of applications, navigating a 2d space exceeding the boundaries of a window has a whole host of design problems. Clicking and dragging to offset your perspective is the best solution, but it is less intuitive and eliminates otherwise accessible click interactions.

Letting the view follow the pointer (therefore moving it relative to the device screen) would be remarkable for this genre of UI.

This is infuriating in certain video games, like the StarCraft II single-player campaign: you're in the middle of 4 different things, your base is being slowly nibbled on by a few run-by zerglings, need to expand, babysit workers, creep / pylons / what have you. Then the game decides to show you something. It grabs your vision cone and forces it where it wants you to look -- slowly.

This specific action -- taking the thing away from the user, while they're actively engaging with it -- should be punishable by sacrificing the designer to Satan straight up. A bit harsh but justified.

The "delete someone else's pointer" in Figma is great. It would be even better if that deleted it off their screen as well.

Tangentially related: one of my all-time favourite neuroscience papers by Iriki et al. [1] showed that the mouse pointer becomes part of the body schema in a real, measurable way.

Basically there are neurons whose receptive field (ie the subset of the outside world that causes the neuron to fire) is "everything a monkey can reach". Flash a light in that area, that neuron fires, flash it beyond that area, neuron stays silent.

Now if you give the monkey a rake, the neuron's receptive field immediately grows to encompass the space it can now reach with the rake too: the rake becomes part of the body schema, not part of the outside world [2].

But if the rake instead is just a stick but it controls a mouse cursor on the screen, the _area of the screen_ the monkey can interact with with the cursor becomes part of the receptive field of that neuron too. That suggests that the cursor itself becomes part of the body schema.

TL;DR don't mess with people's mouse cursors, it's like cutting their limbs off.

[1]: https://pubmed.ncbi.nlm.nih.gov/15588812/ [2]: A wild Heidegger appears and talks about Vorhandensein and Zuhandensein.

> TL;DR don't mess with people's mouse cursors, it's like cutting their limbs off.

“Don’t mess” is a very broad range that includes things like removing the mouse cursor at nine PM. Of course, no one should do that.

But a narrower conclusion “let the mouse warp at certain predictable cases” doesn’t contradict your thesis.

It’s an established thing in CAD programs (since a time before there were any personal computers) to move the cursor for you sometimes. I seem to recall this is one of the points of contention between KiCad and Wayland at the moment.

I have a vague memory that when Windows was newer and idioms less solidified, maybe in the mid to late 90s, programs moving the cursor was more common. Am I mistaken?

This reminds me of that old flash site with an animated rabbit chasing the mouse cursor (thinking it’s a carrot). Still trying to dig up a working link.

There's one with a cat (and various other creatures) - oneko

https://github.com/glreno/oneko

> in the earlier days of Figma, when I prototyped an interaction where you could select someone else’s pointer and press Backspace to delete it:

Aw that looks fun!

There's a class in Java (Robot) that lets you move the mouse like this, I messed with that when I was younger.

Wouldn't work with direct mapped input devices (such as Wacom pads)

Somewhat related, but useful?: I setup AutoHotKey to, after I use alt+tab to switch to a new window, AHK will move the mouse the center of newly activated window.

Turns out it’s handy. After switching apps the cursor is in a defined location and closer to anything I want to click.

It’s weird but it works for me.

Having your pointer move against your will feels incredibly wrong. I used to us this to my advantage in the computer lab, by plugging in my mouse to my classmates' computers and screwing with them. Once, I was chatting with my buddy, looking him in the eye, all while fumbling around plugging my (wired) mouse into his desktop's USB port. It just didn't register with him, and a few minutes later I just started subtly messing with the movement of his cursor. Good times.

https://news.ycombinator.com/item?id=21431778

DonHopkins on Nov 3, 2019 | parent | context | favorite | on: Evolution of the Scrollbar

Open Look scrollbars would be impossible to implement properly in the web browser without using the Pointer Lock API that let you "warp" the mouse cursor position, because when you click on the elevator or cable, the elevator cab actually pushes the cursor up and down with it.

https://developer.mozilla.org/en-US/docs/Web/API/Pointer_Loc...

Here's a demo of Open Look scrollbars from Brad Myers' "All The Widgets" video, which he produced for the ACM CHI 1990 conference. It also has demos of many other kinds of scrollbars and other widgets. (Sun Microsystems, Inc. OPEN LOOK, 1988):

https://www.youtube.com/watch?v=9qtd8Hc90Hw&t=12m57s

>All the Widgets (Fixed v2) - 1990: This was made in 1990, sponsored by the ACM CHI 1990 conference, to tell the history of widgets up until then. Previously published as: Brad A. Myers. All the Widgets. 2 hour, 15 min videotape. Technical Video Program of the SIGCHI'90 conference, Seattle, WA. April 1-4, 1990. SIGGRAPH Video Review, Issue 57. ISBN 0-89791-930-0.

When I was working on The NeWS Toolkit at Sun (an Open Look toolkit implemented in PostScript), Jonathan Payne implemented the Open Look scrollbars, and I thought they had such a cool interesting 3D shape with their elevator, cable, anchors, ridges and arrows, that I carved an 3D Open Look scrollbar for him out of soap as a birthday gift. (He also implemented JOVE: Jonathan's Own Version of Emacs for Unix, and JED: Jot EDitor for TNT.)

https://commons.wikimedia.org/wiki/File:The_NeWS_Toolkit_scr...

Google Wave also had elevator-like scrollbars with arrows on the elevator, but since they were implemented in the web browser without the Pointer Lock API, they could not push the cursor like Open Look scrollbars, so they implemented a weird shadow and didn't move the elevator until you moved the mouse away, with was very confusing to users:

http://ignorethecode.net/blog/2009/11/15/google_waves_scroll...

>Using Google’s scrollbars, it’s also possible to scroll by clicking, but since the arrows are attached to the thumb, that creates a bit of a problem. Typically, the thumb indicates the scroll position. But since you can’t move the thumb while the user is clicking on it, Google has introduced a second element, a «shadow» that is displayed below the thumb. This shadow indicates the current scroll position:

http://ignorethecode.net/upload/217/click_scroll_google.mp4

>If the user moves the mouse away, the thumb will eventually move to the shadow.

>I did not notice this until a commenter on Hacker News pointed it out. He also notes that the scrollbars seem to perform poorly with longer waves.

http://news.ycombinator.com/item?id=944012

https://web.archive.org/web/20100308003557/http://www.comple...

>The Non-Standard Wave Scrollbar

>Figure 6-1. Unlike the scrollbar in your web browser, Wave's scrollbar is the same height no matter how long the list it's scrolling, which keeps the up and down arrows always the same short distance away.

>The scrollbar on the right side of Wave's panels works a bit differently than the scrollbar in your web browser. Like most scrollbars, you can drag it up and down to scroll, or click its up and down arrows to move it. Unlike most scrollbars, the Wave scrollbar's height doesn't change. It's always the same, small size, which puts its up and down arrows in close proximity to one another, as shown in Figure 6-1. Google's intention is to benefit people accessing Wave on mobile devices or netbooks with a limited mousing area, but it has thrown off some preview users.[2] Google explains "the deal" with the scrollbar in Wave's Help section:[3]

>You might find that the scrollbar in Google Wave behaves a little differently from scrollbars in other Google products. To use it, you can drag the bar or you can use the arrows on either end of it—clicking the arrows without moving your mouse allows you to very quickly scroll up and down the page.

>Even at this early stage, at least one developer has created a Google Chrome extension that reverts Wave's custom scrollbars to Chrome's native scrollbars.

Google Wave: Scrolling. How to use the scrollbars in Google Wave.

https://www.youtube.com/watch?v=Lxt2LXCSw5Q

A Pissed Off Tutorial For Google Wave

https://www.youtube.com/watch?v=4Z4RKRLaSug

Reply on news.ycombinator.com