State of multi-player Wayland | Hacker News Reader

State of multi-player Wayland

blinry.org

171 pointsmarvinborner110 comments

> This is obviously not a great solution, and VNC in general feels like ancient technology

It's not obvious to me. This looks like software ageism. :)

There are at least 2 things that could be done to improve performance in the author's setup:

  1. Use the --gpu flag which enables dmabuf capture and h264 encoding via vaapi. 
  2. Connect using a dedicated VNC client like TigerVNC. This reduces latency by roughly 50 ms.

For more details on 2. see second paragraph in the following blog post https://andri.yngvason.is/measuring-latencies.html

That being said, noVNC has gained hardware decoding of h264 via WebCodecs since the writing of that blog post, so performance will have improved some. It's unlikely that the 50 ms latency has gone down, but it might be worth measuring again since browsers may opt to render video through a dedicated overlay plane that bypasses regular rendering. However, it would be silly not to synchronise frame presentation between overlays.

Well software has gradually improved and VNC hasn't so of course it's "software ageism". In any case if you were seriously not aware of VNC's shortcomings compared to modern alternatives:

1. Hilarious security. Passwords silently truncated to 8 characters, etc.

2. Inefficient graphics. Very laggy on all but the best connections. Just sending an H.264 video stream is better these days.

3. No seamless window mode.

4. I've never needed it tbh but I doubt it supports remote sharing of USB, printing, audio, GPU etc

As the author of wayvnc, I would say that I'm pretty well aware of what VNC can and cannot do. The RFB protocol (VNC) has improved over time like other protocols.

1. Wrong. There are many extensions that improve upon authentication and encryption.

2. This mostly depends on the implementation, and not necessarily the protocol and/or its extensions.

3. This would require deep integration with the window system on both sides, so I doubt that this is even a thing with RFB on Linux. There certainly aren't any wayland protocols to do this.

4. No one has ever asked me to implement any of that with the exception of audio, but there is also an extension for that.

Extensions extensions extensions...

> There certainly aren't any wayland protocols to do [seamless windows]

I think wprs can do this (it calls it "rootless") but I haven't actually got around to trying it yet.

re #2 see metavnc (old at this point)

https://metavnc.sourceforge.net/

TLDR: it leveraged a specific client that could

a) connect to multiple vnc instances b) display them in a single window c) act as the "window manager" / "desktop environment" - doing this by enumerating menu entries from each instance and creating a composed whole. Enumerating the active applications from each instance and composing them into a single "task bar"/"window switcher" interface.

this is probably close to 20 years old now.

Ohh, thanks for that recommendation! I'll have to re-test with the --gpu flag. I added it to the instructions in the blog post.

And you might be right about that ageism comment… didn't realize more efficient encodings were a thing now! Thanks for your work! :)

To add to the long list of caveats: This also breaks assumptions GUI toolkit and applications make about only one of their windows being focused at a time: `gtk_application_get_active_window`, `QApplication::activeWindow`, etc.

Mouse pointer doesn't necessarily imply keyboard focus (for cursing)... but there is an issue for mouse enter/leave events, etc..

I guess they could add a parameter cursorIndex defaulting to 0!

I have never liked split keyboards but I might if each keyboard was also a mouse with its own cursor, it seems fun to develop the skill to work that way. Would also be useful at times if the the Thinkpad trackpoint and trackpad could each have their own cursor and I could leave one somewhere. Or maybe a cursor stack like pushd/popd, registers would probably work better; instead of virtual desktops just give me one massive desktop with cursor locations I can jump between, switch to cursor 3 it saves the location of the current cursor and jumps.

>I have never liked split keyboards but I might if each keyboard was also a mouse with its own cursor

You might enjoy the KeyMouse (https://www.keymouse.com/).

Wayland seats are sets of inputs, to be handled together, each seat having its own cursor position, focused element, and so forth. That’s not what you want, I believe.

… unless you do want the left half of your keyboard to type into one window, and the right half of your keyboard into another.

I have a UHK split keyboard with a keycluster module (which has a small trackball) on one the left half and a larger trackball module on the right half.

Even though there is only one cursor it's been a really powerful way to work, being able to control the mouse with either thumb.

Then there are the alternative keybindings that allow moving the mouse using the wasd keys as well.

The original title, "State of multi-player Wayland" is a much better summary of the article. This ain't just about the pointers! In fact, the author is very explicit about that:

> I think there could be a term that’s describing this concept even better – in the title of this post, I picked “multi-player”, to make clear that it’s mostly about multiple people who are collaborating. I’ve also seen “multi-cursor” or “multi-pointer” (which are missing the keyboard component).

In fact, you can also have a different touch input devices per seat!

I'm not aware of actual hardware that would make this feasible, but if you invent a touchscreen which recognises between two people's different fingers, you could have a client which tracks who clicks where without any protocol changes.

I'm building a compositor now, and I've been thinking about multi-seat from the beginning, but I know there are lots of places that assume a single seat, and so this sort of thing won't work for me. It's actually a lot more work than you'd think to support this...

One very nice use case that I haven't seen anyone else talk about is touchscreens.

I have assigned my touchscreen to be in a separate seat. This allows me to interact with apps without moving my cursor or changing my window focus.

For instance: web browser on the left, with a document. Text editor on the right. I am writing some notes with my keyboard, scrolling the document with my touchscreen. The editor stays focused, no back-and-forth with the mouse. It feels very natural this way, instead of having your cursor jump where you touched the screen and change your M/K focus.

Also, large touchscreens can be really nice for collaboration, with multiple users interacting with one or multiple apps at the same time.

Given that a lot of the use case is interacting with two different programs simultaneously, even just supporting it at the compositor level is pretty valuable: then it should "just work" for that use case. You can't really do a whole lot about the fact that most Wayland clients are making assumptions about seats; well, you can, but that's a lot more work...

> It's actually a lot more work than you'd think to support this...

Like: how would one support multiple keyboards? Is there an experimental or logical model for that?

Multiple keyboards is trivial - unless you meant multiple active cursors. (If you've used a USB keyboard attahced to your laptop - you've already experienced multiple keyboards. Both work.)

Multiple mice is esoteric.

Multiple mice (1 pointer) is not esoteric at all... I have a completely normal scenario with 3 (normal mouse, track pad, track point) right now.

And at home, I have a mouse on the left side of the keyboard and another one on the right.

Given the example, I meant multiple cursors. I assumed it would have been obvious.

It's not obvious if your core statement is "multiple keyboards is trivial".

Assuming that multiple mice obviously means multiple cursors while in the same comment assuming that multiple keyboards obviously does not imply multiple input cursors is in fact pretty confusing to me

Fair point.

Of course you're correct re: laptop and aux kb - I was thinking of "seat" (and multiple seats) being: "Wayland seat refers to one 'seat' at which a user sits and operates the computer."[0]

So a keyboard being attached to a seat, whose input might be associated w one active window, while another kb is associated with a different seat and different active window. Perhaps I'm just overthinking it.

[0] https://wayland-book.com/seat.html

If you have a USB mouse then you have multiple mice too, in that sense.

Back when we were working in the office, this was honestly a preferred pair programming setup for us. Two people, and each just had their keyboard and mouse.

Interestingly enough, it seemed that the inputs from multiple mice onto the same cursor is summed up by default. So you could fight cursor movement you disagreed with by moving the mouse in the other direction.

Very silly, but quite funny.

It's not a direct answer, but Nucleus Coop (Windows only) allows playing multiple instances of the same videogame with different savefiles and allow using multiple keyboards and mice (is it mice for the hardware thing?)

It's an incredible hacky piece of software that does work by putting together like 10 oss. Incredible and cool.

They have some UX around this

I keep thinking of trying to see how far I get re-doing Niri in River.

The one other constraint that I'd really like to see relaxed is about displays, about being able to merge and subdivide screens as I please. Having a huge ultra-wide display really would be so much better if I could divide it up somehow. This is kind of the headline feature of stilch, which is a pretty neat compositor, https://github.com/wegel/stilch .

Niri has some tickets on it too. Merging monitors into a big display, and splitting a display (moments before the multi-pointer ticket was filed in Niri), https://github.com/niri-wm/niri/discussions/1285 https://github.com/niri-wm/niri/discussions/3160

There's been all kinds of rough incredibly hard things going on with Niri just trying to figure out how to handle some of the more interesting copy-paste scenarios, and with really weird sequencing issues across basically ephemeral inputs, that have really plagued the project and made life incredibly bad for downstream apps. This is just my opinion, man, but I think the lack of ambition about these ridiculous over the top features eventually starts hampering what turns out to be relatively prosaic asks. Better models for multi-seat, multi-cursor help scenarios like remote-desktop, and video-chat remote control, that at first don't seem needed, but are fundamentals that pay dividends if you shoot for them.

It’s good to see the minimal drag to border behavior coming to all the OSs and DEs these days. I’m a weirdo and use i3 so I relate to this concern and am very comfortable with tiling for lots of applications. However there’s sometimes I wish I could just tell my display manager to treat the single WH wide monitor as two separate displays with dimensions, e.g. W(H-x) and W*(H+x).

Not sure the best level to implement this. There’s probably already solutions for this now that I think about it…

Stilch does that. niri#3159 requests that. Both in my comment above. :)

I wish libinput gave the option for pointer inertia, where the pointer glides to a gradual rather than a hard halt after you stop moving it. I’m limited to a trackpad these days and it’s a really nice feature on my Steamdeck that I’d love to have on my Thinkpad.

Come to think of it, Steamdeck is Wayland? How do they implement it?

>I wish libinput gave the option for pointer inertia, where the pointer glides to a gradual rather than a hard halt after you stop moving it

Why? That seems like it would make it a lot harder to move the cursor precisely. (in fact the main implementation of it on linux I know of was a joke in IOCC)

How precisely can you scroll on your phone? I don't think anyone finds it awkward.

I get the impression that you're thinking of a much jankier implementation of "inertia" than that, and to be honest I wouldn't be surprised if that's what we'd get anyway.

An unambitious inertia describes the linux community of the past decade pretty well.

It works similarly to the steam deck actually. You flick and then tap/hold to stop.

Does this mean the inertia only applies if you remove your finger at the end of the movement, and so non-flicky movements are interpreted normally, with the cursor stopping as soon as your finger stops? That implementation sounds a lot better than how I interpreted the original comment; I thought they meant it would apply to all movements, even when your finger stays down the whole time.

Yeah. It would only work on trackpads. It’s particularly good when you don’t have a massive MacBook style trackpad.

You'd think so but as the person above mentioned, this is how the steam deck/controller does it and it works well. It mostly affects faster movements like flicking the mouse across the screen, leaving smaller movements unaffected.

The opposite applies to my use. Humans are generally great at predicting where moving things will land - we evolved for that. It feels great flicking the cursor around and stopping it at the right point. It’s just like scrolling a list on your phone, I for one get great precision doing that. It’s especially great on small touchpads.

I highly agree. The synaptics driver on an Asus Eee PC I had had this feature and I really liked it. It just makes sense: It is a thing you touch and flick around. Having it immediately stop is like the list you’re scrolling immediately coming to a hard stop once you release the finger.

> How do they implement it?

In firmware.

Semi-related - I think that controllers would be a very nice single input device for many kinds of workloads, especially if you can occasionally just dictate text. It's kind of sad that it's not supported all that well.

cant read the entire article on ios because the left part is just somehow cut off :(

Same on Android, insane that people aren't validating for mobile in current year

Reader Mode works great.

Setting a minimum width that works on your screen is a bad habit that’s luckily mostly dead, in favor of responsiveness.

Insane author here. I think I just fixed this issue, if you wanna give it another try!

It works on Firefox Android.

Works, thanks!

This feels like a step in the right direction from this talk on desktop UX. Finally some cool innovation here!

https://youtube.com/watch?v=1fZTOjd_bOQ

It's not quite innovation, from TFA:

> X11 (the X Window System) has its own multi-seat extension called Multi-Pointer X (MPX), which I also had a lot of fun with a couple of years ago.

On the web, PointerEvents and touch events both support multiple different cursors. https://developer.mozilla.org/en-US/docs/Web/API/Touch_event... https://developer.mozilla.org/en-US/docs/Web/API/PointerEven...

You’ve been led astray by the altered title on this submission. The web does not support the concept described in this article (seats) at all. It supports tracking multiple pointers, but it’s all still single-player, in the terminology of this article: you’re still limited to single focus, single typing, single pointer.

I've always thought this was a cool idea. It's obviously been on the minds of Wayland protocol developers since the beginning, otherwise it wouldn't be possible!

I thought it was the other way around, X11 has supported multiple cursor since 2009 and Wayland didn't add support for it since toolkit developers weren't making use of it anyway.

Enabling in X works via:

    xinput list  # find second mouse id
    xinput create-master second
    xinput reattach <mouse-id> "second pointer"

and to get rid of it:

    xinput reattach <mouse-id> 'Virtual core pointer'
    xinput remove-master 'second pointer'

Works great for clicking, but keyboard doesn't work in some applications when enabled. Dragging windows might attach them to the wrong cursor and other issues. Playing around with xinput can also crash some apps (libgdk-3 here) or leave you in a state without a keyboard or mouse.

Wayland: you can have two cursors but no, you can’t have screen sharing.

Which desktop are you running where you can't figure out how to take a screenshot? Maybe some of the more tech-savvy folks here at HN can help you figure it out.

Yes. Show me how to make zoom screen sharing work without pipewire and prayer.

Since you edited your post I'll rephrase my question: Which desktop are you running where you can't figure out how to share your screen? Maybe some of the more tech-savvy folks here at HN can help you figure it out.

KDE plasma. go wild.

Here's a post from 2023 where the majority of people say it just works, except for one guy who tweaked his env and broke it. https://www.reddit.com/r/kde/comments/18gfffw/is_zoom_still_...

The fact you're griping about implementation details like pipewire makes me think you also tried micromanaging your env and broke something. If you use a fresh install and leave everything default, I'm sure it would work for you like it does for everyone else. I use KDE too btw.

> micromanaging your env

All the "micromanaging" you need to do to not have screensharing is one of the following; 1. Don't have a started dbus session 2. Don't have portals installed and setup 3. Don't have pipewire installed and setup

Meanwhile on X what you need is: 1. Have X

So no, it wouldn't work, the reason why it works for those people in the Reddit post is because they use distros and sessions that do all the things I listed that make screensharing not work if any one is not done.

PS: Wayland does have testing protocols for screenshots and screenrecording like 'wlr-screencopy' and some DMA buff one I forgot the name of. But for screen sharing there's nothing that Wayland implements itself afaik.

Without dbus, X would also have plenty of its features fall flat, it's the de facto IPC mechanism of Linux desktop for better or worse, so it's like "my car won't start if I don't have any fuel" complaint.

Are you using a distro that is missing some of those core OS services? If so, which distro does this not work out of the box in 2023? (let alone 2026?)

Or, did you remove some core OS services, and now you're complaining when various things started to break?

>micromanaging your env

PERSONAL Computer

That means you're free to use or break it however you want and nobody will stop you. If you put a nail through your water pipes, you'll also flood your PERSONAL house, but the end result is still nobody's fault but your own.

Sure, but if one wants to play tester, don't complain when you eventually find a bug.

Is there a reason you don't want to use pipewire?

screen sharing is not a static screencap that you would need to email or something

He edited his post to say something different. Nonetheless I'm leaving my original comment as is.

Screen sharing has been fine out of the box on every Wayland desktop I've used.

I have been dreaming about a split keyboard with dual trackballs and two cursors for the past few months. This may be the kick i need to actually get on it.

Don't let your dreams be dreams; there is the Svalboard Datahand [0]. Though, you may need to sell a kidney to purchase it.

In fairness to the creator, it's a niche product that would have taken quite some time to develop.

I quite like BenVallack's video about the Svalboard [1] (and his videos in general about his journey through the world of split keyboards).

[0] https://svalboard.com/

[1] https://youtube.com/watch?v=-Lz_FNoYHNM

They didn't call them mouse cursors with Ataris and Commodores because almost nobody had heard of a mouse, but there was more than one joystick port so multi-player was built-in from the beginning.

I hadn't heard of a mouse either but when the trackball appeared, mainly to play Centipedes and Missile Command, I used it as a single-button mouse anyway, not only to point & click but to drag & drop which I had never seen demonstrated.

So I made a little 2-player game, not fancy, each player stayed in their own window.

Of course nobody had ever heard of "Windows" either since MS-DOS wasn't even out yet.

I remember the multiplayer of Mahjong on the Amiga worked by just plugging in a second mouse. Voila, two pointers.

I have fond memories of playing Settlers in split screen this way on the Amiga

Focus of controls is the hard part. You end up needing a Per-seat version of focus in an application.

Yup, and it's quite non-trivial.

The typical GTK application out there can't handle multi-touch and can rarely even handle several mouse events (e.g.: mouse-wheel scroll while dragging), so I doubt they have full blow support for multiple seats.

Most of the focus issues should be solvable at the library level.

I immediately recognized the avatar of this person. They are the one with the DIY soft drinks which has been posted on HN before: https://blinry.org/diy-soft-drinks/

I know him for a scrollbar demo. Seems he does a lot of stuff. Great guy. File him next to lcamtuf.

After using a space mouse for CAD it always seems weird how dual mice isn't a more common thing.

one interesting use case is surface management in ways that cant be solved easily by swapping desktops or which is clunky this happens most on single-screen where have to swap apps out that have related actions and want to rearrange or copy paste between or scroll sync and native app so cant scroll sync by extension think rapid fire macro mouse events except not just serial which can run the experience eg browser tab and electrion app that sync scroll or scroll c/p and where macro recorder might come close but not really

While op uses systemd/udev's internal property mutation mechanism to configure seats, not that sway (and likely other compositors) allow you to just configure the device-to-seat assignment in your configuration or even at runtime.

Finally I get to play The Settlers II in split screen mode again

The more "typical" use for a feature like this is drawing tablets and touch screens. You don't necessarily want a touch/drawing tablet input to move your actual "main" mouse cursor.

Wayland has a separate wl_touch type that is also associated to a wl_seat. So that use case can be accomplished on a single seat setup.

That’s unrelated. The typical device will only ever have one seat. From Wayland core, that seat may have a pointer, keyboard and touch; and from tablet-v2 <https://wayland.app/protocols/tablet-v2>, you can add tablets and their tools and such, all still in the one seat.

Interestingly, for me this usecase isn't typical at all. Ever since I switched from X11 to Wayland (on GNOME) I've had the bug that when I'm drawing with my Wacom tablet, there's two mouse cursors and the tablet doesn't influence the main cursor anymore.

I thought this is a bug in the Wayland implementation: I find it utterly unintuitive to work like this. (I just have no idea with whom to file a bug report, which is why I haven't.)

Are you implying that this is intentional behavior?

I use two mice just to be able to switch from the fatigued hand to the other without shifting the mouse repeatedly. I wish it was easy to have two cursors in KDE.

Very timely post. I wonder, is it possible to clone one single mouse cursor, to duplicate exactly same actions on two windows placed side by side?

Maybe RedHat should contract this guy to finally get pointer warp implemented for Wayland everywhere it needs to be.

Innovative stuff. I wonder if similar things are possible on Windows and macOS.

This is actually something I've always wanted, but people told me it's dumb. I want two cursors and two keyboards, with two "focused window" panes. If Mouse A clicks Pane1, keyboard A presses should go into Pane1. I think this would be so fun for multiplayer games on the same computer for example (think two Minecraft instances running).

It's possible with wayland seats, if the compositor supports it. I've used sway this way. Assign M+K A to seat A and M+K B to seat B. I only wish I could change the focused window decoration depending on the seat (say: one blue seat and one yellow seat).

It's not as weird as you might think: laptops may have touchscreens (which can be multi-touch) on top of the "normal" cursor. IIRC, X.org can't really handle such cases cleanly.

That's not what this article is about. You still only have one mouse cursor in that case, but two things (the touchscreen and touchpad) drive it.

This is about actually having multiple, independent mouse cursors on the screen at once, with different input devices controlling different cursors.

Sometimes that's true, but sometimes (imo the superior approach) the touchscreen is independent from the cursor. I think touchscreen Chromebooks are like that, for example.

Oh, that seems like it would be quite nice, actually. It is vaguely annoying when my touchscreen moves my mouse cursor.

It depends on the DE and the application to some extent. The touch events are seperate to mouse events and the mouse cursor moving, but AFAIK usually there is an emulation layer for compatibility with applications that don't support touch events which also moves the cursor, and this behaviour can change depending on the application (either it sets some flag or responds to the touch events in some way). Digitizer pens are the same way.

Cool! Where can I get the turtle mouse cursor?

Glad you like it! :) I uploaded it here: http://files.blinry.org/emoji.tar.gz

You can find installation instructions in the script of last year's workshop: https://blinry.org/cursor-workshop/

and we can also assign this to particular apps like, mouse 1 and keyboard one can only interact with notepad !! etc !!

This is cool. I feel like I shouldn’t be the only one who as a kid connected up an extra mouse to their computer with their buddy and was disappointed to still only have one cursor.

The way screen sharing solutions like Teams handle the multi cursor thing is to just keep the main cursor jumping around, which works, but meh.

Having multiple cursors within the same environment theoretically enables such interesting collaborative experiences. Imagine a large surface in which everyone has their own workspace (cluster of windows), but you can seamlessly go between them …

Another possible use case might be mixed reality/VR displays, for simultaneous mouse cursor + eye tracking/control.

Sexy Flanders

On the note of the two different kinds of seats, a tangent: better multi physical seat support is something I am passionate about, specifically, I want Windows-style "Fast User Switching" on Linux with RDP. Today's Linux RDP support usually acts like screen mirroring; you get a mirror of a physical seat. With GNOME I believe there is also now support to do remote login: you can log in remotely, and even disconnect and reconnect to a remote login session. But what you can't do is continue a physical session remotely or a remote session physically, and the reason why is sorta to do with seats. Sort of.

See, when you start a graphical session on modern systemd-based Linux, a new logind session is created, and it is immutably tied to a seat. So you can't just remove the seat from a running logind session and then make it remote, or add the seat to a running logind session and make it physical. I actually tried patching systemd-logind to do this, but when I proposed it upstream, Lennart Poettering himself actually took the time to explain that this wasn't the intended way to do it: instead, you would actually just create a new logind session and drop the old one entirely, or literally whatever you want.

Which to me sounded perposterous, but then I realized that it really wasn't that ridiculous. As it was, when you boot a desktop using systemd-logind, the desktop itself doesn't typically run much of anything directly in the session context anyway - the compositor and almost all services actually run as user services, as in systemd-land having more than one active desktop session per user isn't really supported. Okay so we can do this!

And so I set out to prototype this with KDE, but there are still a few issues...

- First and foremost, $XDG_SESSION_ID. Believe it gets set by the systemd PAM module somehow? I don't fully recall. Either way, it gets used by various programs to determine the current logind session, but if our graphical session can "roam" between them we need a mechanism that can handle changing over time. I propose we stuff the current logind session ID into a file in $XDG_RUNTIME_DIR instead.

- Secondly, compositors need to be redesigned to handle this. Compositors today can hotplug displays and input devices - that's a good start. Unfortunately, at least kwin can't switch backends at runtime. You can really go one of two ways, and I tried a bit of both and either can work but both are a bit ugly: one is to make a single backend that can handle either case and the other is to allow switching the backend at runtime. Either way, when running remotely the card* DRM nodes are unavailable so you need to switch to renderD* nodes instead. You can also use both, and only use card* for scanout when available, but it seems a bit trickier.

- Finally, I genuinely have no idea how this could or should be handled in non-systemd environments. I have admittedly forgotten how that all works. It would be rather lame to simply require logind for this to work, but sadly it is the path of least resistance at least at first.

And maybe some more, but otherwise the concept does indeed broadly work. I mainly have only tested it in virtual machines with DRM native context, but I can see everything working switching in either direction at least once before things start to get screwed up. Even OpenGL clients and audio playback can move seamlessly! It's just like what Windows accomplished over 20 years ago!

Aside from convenience and being cool, it would also solve a couple problems. No need to worry about the limit of only one session per user: just move your session with you. No need to worry about your physical session being unlocked when remotely connected: when you log in this way your physical seat gets disconnected from the session entirely and the display manager takes it back over.

I actually would really like to at least make some progress on ground work that would be needed to make this happen, although I'm not really that sure that anybody cares. But either way this post's brief discussion about physical vs logical seats activated a few neurons so I couldn't help but dump this out.

I would appreciate this as well, TBH. Window's handling of remote desktop has been well ahead of linux's for a long time, but there's an irritating artificial limit of one active user on most versions of windows. Last I looked, even just headless wayland was still very awkward to use.

Imagine working with two separate mouses at the same time - no keyboard.

Now I know why Web devs, frontend developers hate us

Reply on news.ycombinator.com