PipeWire and fixing the Linux Video Capture stack
blogs.gnome.org
blogs.gnome.org
>And also we can have patchbay applications that supports video pipelines and not just audio, like Carla provides for Jack applications. To be clear this feature will not come for ‘free’ from Jack patchbays since Jack only does audio, but hopefully new PipeWire patchbays like Helvum can add video support.
There actually have been patches for JACK to enable it transport video just like it already handles audio & MIDI. We never accepted them into the mainstream.
The specific problem with video compared with audio is this: for audio, settling on 32 bit floating point as the canonical format for audio (and note, JACK did not prevent you from using other formats, even internally; people just didn't bother) was not even remotely controversial. Nothing is lost and much is gained via this choice.
For video, however, there are a plethora of possible format choices, each with their own distinct reasons for being, and often significant computational costs associated with conversion. This means that for video it's fairly problematic to define a single format for an entire workflow, which in turn means that figuring out when/where conversion between formats occurs is a significant task. For JACK, we decided not to try to do this, and left the system handling only audio & MIDI.
The only way to use Zoom from my point of view is through their web app. And every time I start a meeting from their web interface, they're using every possible dark pattern to force me to install their official app.
Using it at Fedora Silverblue 34 (Gnome 40 on Wayland) and screen sharing works (using it often at work).
Their webapp is quite bad as well though and behind in features especially on Firefox on which, for whatever reason, there's no gallery mode.
They should drop their crap desktop app and focus on building a descent webapp instead.
Not sharing its own windows is nice and all, but I don't think Zoom even does that on other platforms. It's weird how much effort they have put in to doing the wrong thing.
I switched to pipewire recently and the experience for Bluetooth is amazing.
On an average day I have audio output set to HDMI while I listen to Spotify and work; when I plug in a wired headset for a meeting the gnome switcher dialog pops up but when I select headset, it doesn't work since moving to pipewire, output remains with HDMI and input from the laptop internal mic. I have to go into Sound every time and manually switch to headset out and headset mic.
It doesn't seem to have this issue if I'm not using HDMI output.
So putting a script /etc/acpi/ should do the trick, but is a bit hacky.
I'll switch my work laptop back to pulseaudio as being able to switch on plug/unplug is critical for my average day.
My personal laptop can stay on pipewire since I exclusively use Bluetooth there.
Hopefully it's a feature that'll be implemented soon.
Well a bit concerning in a way. Does this change require GNOME and/or Wayland ? The way I read it it seems so, but little detail there.
I wonder because I use fvwm2 as opposed to Desktops, which I really do not like at all.
No need to go that far, this site supports reader mode.
> Does this change require GNOME and/or Wayland?
PipeWire is not bound to Wayland or a specific desktop environment, it runs (mostly) fine under GNOME, KDE Plasma as well as sway. The desktop interfacing is done via flatpaks xdg-desktop-portal, as long as your desktop implements that you should be fine.
AUR has packages for replacing PA and jack with pipewire, but when I tired this I ended up with a dead ardour and not even timeshift could get me back to a working system.
pipewire-alsa 1:0.3.38-1.1
pipewire-jack 1:0.3.38-1.1
pipewire-jack-dropin 3-2
pipewire-media-session 1:0.3.38-1.1
pipewire-pulse 1:0.3.38-1.1
systemctl --user enable \
pipewire.service \
pipewire-media-session.service \
pipewire-pulse.service
You mentioned qjackctl, is there a reason I must keep using that instead of having everything defaulted to PW pipe
disabled - the service is not started automatically
masked - the service can't be started at all, even explicitly requested by user or app
Pipewire implements the API of all the system it replaces. So you can keep using Jack routing tools while everything is actually using Pipewire.
https://wiki.archlinux.org/title/PipeWire
Your session environment might also need to be aware of how to interact with PipeWire, which by default is via pulsaudio compatibility interfaces.
Required (practically, probably pulled in with just pipewire):
pipewire pipewire-media-session pipewire-pulse
Recommended:
pipewire-alsa pipewire-jack lib32-pipewire lib32-pipewire-jack
You might also need to reconfigure your audio settings, but that is unfortunately desktop session specific. (E.G. KDE has it in one place, gnome desktops another, etc)
"To apply the changes you must restart your computer"
It is just sad.
Both PulseAudio and PipeWire work best when running the underlying kernel drivers in "exclusive" mode, which means only one sound server can be running at a time. In order to ensure that works correctly, you need to:
1. Close all applications
2. Shut down PulseAudio and prevent it from auto-restarting
3. Start up PipeWire, and then restart your desktop environment if you want any hope of the volume applet working (or figure out which of the processes to kill if your volume applet in another process)
4. Start up your applications again.
5. There might be some applications that have tried to play audio that you were unaware of! Like your file manager, which can play a custom "bell" for mandatory audio alerts when trying to type in a place that doesn't accept characters. Oops! Better close that down too and start 1-4 over again.
Restarting the computer is an easy and safe way to apply these changes. That people didn't follow all these steps in this exact way meant the computer started to break in strange and unobvious ways, not that it was ever supported.
If you want to help out designing smoother ways to change out core system components without reboots, feel free to do that. The people involved might not find it worth the trouble to fix, though. I think it's just sensible to restart the system.
See https://wiki.archlinux.org/title/PipeWire#Installation
I’ve had zero issues.
PipeWire is truly a godsend for screen recording on Wayland, not very long ago, if at all, every desktop environment / compositor had their own home brewed protocol and developing anything related to screen recording on Wayland was a massive pain [1]. I still can not fathom how Wayland could even exist this long without any decent way to do screen recording and I also do not get why there is no official protocol or at least extension for this!
But thanks to PipeWire some first projects finally support screen capturing on Wayland. Both Firefox (via webrtc) [2] and OBS [3] directly interface with PipeWire. For my personal project Weylus [4] I opted for the gstreamer plugin as at the time the documentation for PipeWire itself was very lacking and I could not figure out how to get it to work. Fortunately gstreamer is better documented and it turned out to be rather easy to use. But it shows that all this is still in an early stage as I hit quite some bugs, probably the most egregious (maybe even somewhat funny) one is KDE's kwin_wayland crashing if you hover the cursor over any window close button while screen recording [5], sadly it doesn't seem to be fixed yet. For some more bugs I've hit, see [6].
Something I am a little worried about is that flatpak with their xdg-desktop-portal [7] is now responsible for the de facto standard how screen recording is negotiated on Wayland. I fear that this means if flatpak does not need a feature, even if it makes sense in another context, it probably won't ever be implemented. For example something I require for my project are proper window names and their position as well as size, but apparently this is nothing essential for flatpak and the issues I have opened have largely been ignored [8]. This is by no means meant as an accusation, just an observations that things are not all good.
[1]: https://github.com/H-M-H/Weylus/issues/3#issuecomment-652670...
[2]: https://webrtc.googlesource.com/src/+/refs/heads/main/module...
[3]: https://github.com/obsproject/obs-studio/blob/master/plugins...
[4]: https://github.com/H-M-H/Weylus
[5]: https://bugs.kde.org/show_bug.cgi?id=435042
[6]: https://github.com/H-M-H/Weylus/issues/3#issuecomment-808940...
[7]: https://github.com/flatpak/xdg-desktop-portal
[8]: https://github.com/flatpak/xdg-desktop-portal/issues/created...
It likely won't ever get an official Wayland extension, for a number of reasons. Screen capturing is inherently insecure and so it shouldn't really be happening over the Wayland socket. It also doesn't need to happen over the Wayland socket or be related to the windowing protocol at all, attempting to do it that way is just re-accumulating technical debt left over from X11.
>I fear that this means if flatpak does not need a feature, even if it makes sense in another context, it probably won't ever be implemented.
I wouldn't think of it so much as just flatpak, but the API needs to be usable from within a sandbox, and it needs to be done in such a way that it can be reasonably presented to the user.
>For example something I require for my project are proper window names and their position
This is a really bad idea, don't do this. This is guaranteed to mess with the window manager and run into any number of syncing issues with window sizes being reported incorrectly. It's also a misuse of the screencasting API. I checked the issues you posted and I don't think you're going to get much farther trying to hack around the API. If you want to do this right now then you'll have to integrate with the specifics of the window manager or shell. There is no way around it, I really doubt you will get much interest in doing it another way.
I see, I do not know much about the internal workings of Wayland. The problem I do see though is that screen recording is an essential feature that has been neglected for years due to the lack of a standard.
> This is a really bad idea, don't do this.
While I agree that getting window positions may be questionable, providing a human readable title for the streams that have been selected seems obvious, yet is missing.
> syncing issues with window sizes being reported incorrectly
This seems rather theoretical to me, while of course true, I doubt it is going to matter in practice.
> It's also a misuse of the screencasting API.
Hmm, do you think there is a better solution, that isn't giving up and having each desktop environment do its own thing?
> If you want to do this right now then you'll have to integrate with the specifics of the window manager or shell. There is no way around it, I really doubt you will get much interest in doing it another way.
This is also my conclusion, alas it's not happening and one of the reasons why I think Wayland is not quite there yet.
In my opinion, what you're asking for is an entirely new thing. You want an API for window management, but there has never really been a window management API on Linux, and every desktop environment was already doing its own thing. It has nothing to do with Wayland specifically. Under Xorg, the X11 APIs can sort of be hacked to do what you're asking under certain circumstances, and it might work ok on some window managers, but it's not going to work in every case. (And it doesn't play nicely with sandboxes either)
To get the human readable title seems like it would be an issue with how Mutter publishes the stream, i.e. not an issue with the portal itself. But I haven't looked into this enough in detail. I also don't know enough about your application to know why you would need the window sizes and positions. My only point is that if you intend to use that to draw an overlay over a screencast, or otherwise try to reconstruct what the compositing process is doing, then that's not going to be completely accurate. And it can't really be accurate without exposing various internals of how the compositor works, which puts any proposed API design back to square one...