Improving Steam Client Stability on Linux
ttimo.typepad.com
ttimo.typepad.com
At least the first one (the getenv thread safety fix) will hopefully make it into glibc 2.41 and it should be quite safe to backport. It turns out that setenv is easier to handle because glibc already never frees environment strings. It's concurrent unsetenv that is rather tricky. Without some snapshot approach, getenv would return null pointers instead of environment variables values that are actually set. I don't want to introduce locking into getenv because getenv without setenv has been async-signal-safe for so long that it would likely break applications.
The environ handling fixes are a bit more controversial because vfork+execve make it complicated to avoid memory leaks, but these further fixes are less important to the stability of the graphics stack.
A purely hash-based implementation is not possible because there is putenv, and some applications expect modifications of environ to be visible via getenv.
Decades old footguns - aaargh!
> Decades old footguns - aaargh!
Indeed...
The tricky problem they have is wrapping it into some sort of lock could cause so many issues. Places that dont deadlock, suddenly could. Not a fun problem to solve. In practice it is usually not too bad as you are usually not changing your env vars much. But when you run into it, ugh.
Flatpak fixes this problem. You don't need gamescope unless for stuff like scaling. The only few times you need to fiddle with anything on desktop is changing proton version or adding a launch variable from protondb.com (just like on steamdeck)
You seem to think steam on Desktop Linux is somehow different from steam on steamdeck Linux. Like I said in my previous post, the only "fiddling" you need to do is copy pasting launch commands and choosing a different Proton version from a drop down _just_like_on_steamdeck.
It's also an immutable distribution, like SteamOS (except it's based on Fedora Silverblue instead of Arch) so there is no "incompatibility with your exact distro setup": you have the exact same distro setup as every single Bazzite user.
PC gaming is very enticing but we all know that’s simply not how it goes down. I would love to build a PC that is literally just discord and Steam. I want to run it in big picture mode for the most part and treat it more or less like a console.
I have really enjoyed my steam deck and it has fit that desired role pretty well. But obviously it is just not that powerful. It’s impressive for what it is, but for me it’s basically a great indie game machine with the occasional AAA option that is tolerable. A well-built machine with that kind of UX (minus the known idiosyncrasies of the deck) would be fantastic.
To answer your question more directly: Most Linux distros do not offer this either. Bazzite is the closest I’ve seen.
https://wiki.nixos.org/wiki/Steam#Gamescope_Compositor_/_%22...
I feel your anecdotes are at best outdated. Desktop Linux has come along way.
Linux is a great experience these days but you do have to tinker sometimes. You have to mess with drivers and settings and command line. It may be minimal for people comfortable with computers but it's not as friction-less as you're claiming.
If you buy hardware that is compatible with Linux, then you won't really have to tinker (at least, any more than you would with any other OS, for example, tweaking resolutions, etc). Unfortunately, it's newer hardware that typically requires the tinkering. If you don't want to tinker, I would recommend going with generation n -1 or even n -2. If you go with the latest and greatest, expect to have some tinkering required.
Distro choice does of course matter a great deal. I've been using Fedora as primary OS now for many years and absolutely love it, and it's what I recommend to most people. Ubuntu and derivatives are good of course, though the older kernels do often decrement the generation of hardware. For example, Fedora on n-1 is going to be pretty good. Ubuntu might still lack some support at that age, so should go with n-2 or n-3 to be safe.
My big takeaway is this; if you are comfortable using the Steam Deck to play games and install software, you will not struggle to get Linux to run games. Pretty much anything that isn't a gaming laptop is going to have some form of support, and even the famously crappy Nvidia drivers were recently updated to support Wayland and other new Linux protocols. Now more than ever before, using Linux to game is probably easier than getting the equivalent experience on Windows.
by this standard Mac OS is still a hobby OS because it can't be installed on random hardware.
No, it isn't too much to ask that you make sure the hardware you buy works with the OS you intend to run. If you find Linux fiddly in the modern era it's solely because of this.
> in such a way (downgrading/using older components) to accommodate linux
When I buy a Mac or a windows machine I don’t have to purposely avoid newer hardware to ensure it works.
It also matters how far along in the product life cycle it is. If it came out last week, it may not be supported yet. If we're nearing the refresh point then it may be supported.
> When I buy a Mac or a windows machine I don’t have to purposely avoid newer hardware to ensure it works.
But you are also comparing apples and oranges (pun incidental) and shifting the goal posts. If you buy a Mac, then you aren't building a gaming PC, which is what the rule of thumb pertains to. You're buying a complete system that has been integrated and tested. You can do the same thing with a Linux machine from various vendors (Lenovo, Dell, Framework, among others), in which case you don't have to do any investigatory work because (just like with the Mac) it's been done for you by the manufacturer.
From https://repo.steampowered.com/steamos/README.txt
SteamOS version 1 'alchemist' and version 2 'brewmaster' have been discontinued. No further updates are planned.
The SteamOS 'clockwerk' prototype has also been discontinued and will not be released.
But I haven't heard anything one way or the other in a while. But as it stands, they have stated that they plan to do a general release. Unless there is another source where they say they changed their mind?
Some of the recent SteamOS release notes have included references to Asus's handheld, which has reinforced the community expectation that it will eventually be available as a distribution you can install on 3rd party hardware. If you go read interviews from Valve employees (Lawrence Yang comes to mind), I believe they've publicly stated that after the OLED shipped, they wanted to start focusing on porting to other devices.
If so, it is kind of bizarre they haven't reached out to the Bazzite maintainers at all.
In general, it seems like it would save them a tonne of effort if they'd switch from a bespoke Arch-immutable spin to making a spin of Silverblue, something that has been meant to be immutable from the beginning.
They already have a system that works exactly how they want it. They already rebased from Debian to Arch to get it there. They have enough Linux staff on contract to build and maintain that system.
Maybe Bazzite is closer to their goals; maybe it's not. It's certainly not a slam dunk that the best thing they could do is throw away the thing they've been building for years to join a community project on GitHub that's trying to clone that thing.
There are many gaming distributions (e.g. Bazzite, Jovian/NixOS, Nobara, Chimera…) that take this approach.
They're usually just standard desktop distributions (Fedora or NixOS) with gaming packages configured. There is a Russian teenager who's trying to cobble together a SteamOS clone using as many Valve packages as possible. His project is called HoloISO.
The compositor is what runs two layers down, under the window manager.
Java does what you say, but it still presents problems for JNI code using libraries that want to getenv().
That said, GLIBC is pretty good at documenting all the dangerous functions, so it is possible to add locking/copying yourself.
extern char **environ;
Regardless of anything else, how about:
* deprecate direct access to `environ` and add functions to replace it. Have a macro that indicates this and provide a canonical compatibility shim for people to copy if they might use old libcs.
* using linker magic, change the behavior of programs depending on whether they attempt to access `environ` or not, so old-API programs are still thread-unsafe but new ones are thread-safe.
It's amazing how much you can do with the conditionally-linked object files from a static "library". Much of C's cross-TU "UB, no diagnostic required" is inexcusable since we can detect it quite easily with zero overhead (at least, for static linking) using today's linkers by deliberately causing multiple definition errors.
Compatibility with old-ABI programs probably means fixing `environ` is not that simple, but you are libc and libc is in control of dynamic linking ...
Also the changes you mention require changes in the linker scripts distributed by a variety of toolchains and would need to check if the libc target was a blessed one. In effect the deprecation would never move to obsolete and kept around forever.
To clarify the above post libc is in control of dynamic linking through the dl*(dlopen) family of functions.
I don't think this would work here because it likely changes semantics too much, and not all binaries that need a thread-safe getenv/setenv combination can be rebuilt, especially since compatibility with both variants from the binaries would likely some changes to each application/library.
Not if third-party dependent libraries use getenv/setenv. (The article mentions this as a continuing problem with the steam client.)
I mean what kind of threading library doesn't have shared memory or message passing?
I'm guessing this mostly happens in situations where the main process can change variables like HTTPS_PROXY and a different thread is running a library that checks those variables before firing up a TCP socket.
26 years ago people knew this API was broken but didn't fix it due to inertia of breaking buggy programs further.
There really shouldn't be a need to change your own process's envvars. For subprocesses just use the proper exec function. For anything else there should be a clear API to call rather than changing a global variable and hoping some code far away from yours rereads it and handles things correctly.
This change would make old programs leak memory every time they call getenv() without the subsequent free(), but since the current version also leaks memory that doesn't seem like a dealbreaker. As an added bonus the new version could be made thread safe by wrapping the strdup() in a mutex and doing similar work on the setenv() side.
From a hygiene perspective - freeing the return of another API is an anti pattern. If you need the caller to release objects there should provide a FooLib_bar_destory(bar) or similar.
Personally, if the return object is a basic C type, I'm not a fan of creating a wrapper function to call free(). This is one of those code purity things that I don't think buys you anything.
Is setenv really a Linux API, since it's neither defined by Linux (it's in POSIX) nor implemented by the Linux kernel (it's entirely in userspace)?
And the reason the colleague dubs it the worst has nothing to do with its specification in POSIX (which doesn't require the function to be thread-safe but also doesn't prevent it from being made thread-safe), but rather its specific implementation in glibc, which is the C library in use on all the Linux distributions that Valve support Steam on and all are equally dubbed as "Linux"
Languages like go just implement this feature internally and don't have this limitation.
The limitation is contemplated in POSIX itself; from the posix documentation:
> The setenv() function need not be thread-safe.
Perhaps the leak is "better", in at least there won't be non-OOM crashes, but it still leaves a bad taste in one's mouth.
For a long running program like Steam (that is for some odd reason calling setenv…?) … I'm not sure which is better. Better would be not calling setenv, which it sounds like they've worked on.
But honestly, it seems like one might as well do exactly that (strdup & return)? At worst, nothing calls free(), and it is equivalent to the macOS strategy of "just leak the memory" to make it threadsafe. But at best, a program could #ifdef its way into "oh, this semi-sorta-nonstandard-behavior on this particular OS" and call free(), getting both a thread-safe & non-leaking implementation.
Of course, we could have just getenvdup() which does what you just said. Let each application chose which variant is best for them: leaking, or thread-unsafe.
But the Steam client is really strange. Sometimes it works for months, and suddenly a game won't start, or something doesn't work, and I have to do weird stuff to get it working like purging all files or reinstalling. It doesn't make sense, it's like the Steam client rots.
Even being only one version behind is dangerous for such a huge attack vector.
1) Keep it 32 bit
2) Have a single store-frontend
At least, it came to a conclusion that it steam itself should be responsible for managing runtimes for games.
Yes please
The article mentions that they use exevpe for spawning children processes. So what usages of setenv(3) would remain?
I'd be interested in a way to do static binary analysis to get from those symbols to a call tree, as well.
I don't see a way to check for **environ usage though, the compiler could turn this one into anything.
Could someone elaborate this for a non-developer? Why would you use `setenv` (which I assume is functionally similar to `export key=value`, but correctly me if I'm wrong) (extensively) for spawning processes?
Using setenv is mostly always a hack that relies on a bunch of assumptions that could easily change and be hard to debug.
edit: what's probably happening is that execve is four or five abstraction layers deeper (possibly in a third party dependency) than where the env variable need to be set without a clean way to pass the values through.
I primarily use Windows, both as an end-user and an amateur programmer. From my experience, most programs on Windows don’t do this. If parameters are needed, they’re usually passed as arguments, while environment variables are used for more permanent settings, like %PATH%.
It can also be a way to pass license information or other configuration settings.
I would like to see at least in-process environment modification discouraged. Rust is dealing with the issue by considering getenv unsafe when coming through C, but getting rid of the read side is much harder than the write side.
One can use `hidepid` parameter when mounting procfs to hide cmdlines.
I don't know why this is not implemented today by default in most distros. Probably history reasons.
I think it goes back to where windows came from. That environment space in DOS was not exactly huge (256 bytes at one point?). In unix it seems like it was much larger and expressive.
Which is a Good Thing, but unfortunately it makes dealing with proxy support so much harder - especially as there are just so damn many HTTP libraries that people use...
You can modify the JVM's buffer, or spawn a new ProcessBuilder, or a few other things. But it's a nasty place to be.
So yeah you definitely could (there'd be other reasons why java wouldn't be a good choice).
TBH reading that folks were calling setenv before an exec to propagate env vars to a child made me sad. I would guess that other use cases were leveraging environ as a poor man's global variable - which is also unfortunate.
On another note, I believe Zig does the same thing.
TL;DR: if you have Steam running for more than a ~day or so, you will run out of window handles so you won't be able to open any new graphical application/window until you restart Steam.
Using Steam Chat appears to make the issue worse (it happens earlier).
This has been documented under https://github.com/ValveSoftware/steam-for-linux/issues/9094 but for some reason that issue has been closed.
I personally just restart Steam every day but if someone else encounters this issue and doesn't know why their windows are not opening, this is why :)
I am using KDE/Wayland but I've observed this under X11 too.
You have a slave sentence there. These are incompatible with albeit.
1. I leave Steam running all the time (although I never open Steam Chat).
2. I leave Steam running all the time (albeit never with Steam Chat).
"I never open Steam Chat" can stand on its own as a sentence, but "never with Steam Chat" does not and thus can be appropriately modified with albeit.
There are lots of little discussions like this that I'd love to have, but which sometimes lead to a lot of downvotes.
The one that's so annoying that I've actually developed habits around it is the tendency for the library window to freeze when it becomes unfocused - it's not all the time, but it's often enough that I now habitually close the window anytime I defocus it
The performance issues have made me not want to even open Steam anymore so I have nearly completely stopped playing. (not related to this issue but several games have also added kernel anticheats, killing off Linux versions, which also has contributed to this)
https://github.com/search?q=repo%3Aioquake%2Fioq3+ttimo&type...
(Up to date mint, cinnamon)
Integrated graphics to be sure, but i'm usually only using it to stream games from PC or laptop and the performance is fine for that in game.
I shopped around for computer parts with complete IOMMU support just so I could map the discrete GPU to the virtual machine and achieve near native performance... Only to discover they are exceendingly hostile to users who do this VFIO stuff.
Just yet another reminder not to "buy" games on these platforms, I guess.
So just any standard/decent motherboard bought within the last 3-4 years?
> I wish they'd make it more virtualization friendly.
Which games do you have issues with when virtualizing? I've only been locked out of Halo Infinite and I enable all hyperv flags in libvirt.
> Just yet another reminder not to "buy" games on these platforms, I guess.
If you buy a game on Steam and you're unable to play it they'll refund you straight away, which is why everyone loves Valve and Steam even when the client is crappy.
I couldn't be happier with my setup, which I assume is similar to yours. (2 GPUs)
It isn't at all obvious that IOMMU is supported by even current top of the line motherboards. I looked at a lot of products and have yet to see the VT-d and AMD-Vi keywords mentioned in technical specifications. To confirm support I had to read their UEFI firmware manuals and look for instructions on toggling the virtualization support.
https://pcpartpicker.com/forums/topic/466120-ecc-ram-and-iom...
At least ECC memory support has started to show up in technical specifications. Looks like IOMMU is not quite there yet.
> Which games do you have issues with when virtualizing?
From what I'm reading many games take virtualization as evidence of cheating and have no regard for false positives. I'm currently assuming any game with battleye or easy anti cheat will issue permanent bans on virtualization detection.
> If you buy a game on Steam and you're unable to play it they'll refund you straight away
Somehow that doesn't bring me much peace of mind.
> From what I'm reading many games take virtualization as evidence of cheating and have no regard for false positives
There's a lot of misinformation on the internet, I haven't heard about any cheats used in the wild using a VM and haven't had issues.
>> If you buy a game on Steam and you're unable to play it they'll refund you straight away
> Somehow that doesn't bring me much peace of mind.
If zero risk purchases doesn't bring you peace of mind I recommend dual booting or running two rigs instead of using bleeding edge technology
If I spam the ‘retry’ button it’ll eventually work, but it’s a massive PITA.