It is a bug in Xwayland. Theoretically the rest of the world is not affected. Apparently some people think that breaking compatibilty is the way to go.
There are a few programs I work on where we "officially" disabled big-endian support, because it's just impossible to find devs to work on it. It's also hard to test, you can't test it on github actions for example.
I'd be happy to see some people step up for testing big-endian, but at the moment it would be a huge time-sink for many projects, when it's not even clear if there are any users.
None of which are supported officially by Debian, with notable exception of ppc64el (POWER8/9 little-endian). The only official port with big-endian is s390x. [0]
The simplest one I could find is netbsd, which can run big-endian on an Rpi3 and some other boards.
https://wiki.netbsd.org/ports/evbarm/ (see the earmv7hfeb note)
Eg. Instead of:
uint16_t x;
#ifdef BIG_ENDIAN
x = swap(*p);
#else
x = *p;
#endif
uint16_t x;
unsigned char *q = (unsigned char*)p;
x = q[0] | (((uint16_t)q[1]) << 8);
This exercises the same deserialization from little endian regardless of the hardware representation of integers. This means you are testing how it will act on big endian even when running on little endian.
So the padding, alignment, endianness, all leak. And it sounds like X does something like this: clients expect the endianness of serialized data to match the local machine and the server can optionally account for this.
First, the structs need the nonstandard pragmas and attributes to not pad.
Then, all accesses of 16, 32, or 64 bit quantities need to go through a helper function.
Even with that baggage, the syntax of having struct member access is can still feel more convenient than more verbose serialization code.
There's also methods for native byte order, but thats almost never the right thing to use.
I think Go does the same thing. I heard about this approach first from Rob Pike, who would rant about this to anyone who would listen. He said to think of "bytes on disk/network" and "an integer in RAM" as different types that you need to convert between. When you do, the API should specify the endian-ness of the bytes on disk / in the network. He argued you shouldn't do "byte swapping" at all based on endianness. Just parse the int, byte at a time and use the same code on every computer.
The real problem is that C makes it easy to convert between a pointer to an integer and a pointer to some bytes in RAM. Just never do that using pointer casting, and everything works out great.
https://doc.rust-lang.org/std/primitive.u32.html#method.to_b...
It is a network protocol, just put a proxy between client and server that byte swaps the affected fields and changes the endian flag. No need for hardware.
Convert from bytes in (whatever) endian order to bytes in memory when data is read and written using a uint32_t parse_le_uint32(uint8_t[4]) method or something similar.
Asking unironically.
Most of the other notionally bi-endian architectures nowadays have settled on one. For example, SPARC is just about exclusively big; conversely, ARM64 got rid of their big-endian mode.
BE is still common in networking hardware (due to being the traditional "network byte order") and mainframes. The most widespread BE hardware would probably be the cheap un/rebranded routers with MIPS SoCs.
ARM also supported both, but you don't really see ARM used in big endian mode today.
This is a big problem, because we may be introducing subtle bugs that are actual bugs that just happen to work because the hardware is arranged in such manner.
A lot of our stacks these days is built from source and very portable. We can certainly do better than everyone using the same CPUs.
It's arguable whether that's a bug if it doesn't have any practical effect. I'd say that it's not.
This is different than say a video game that works fine on hardware, but breaks in emulation, as sometimes happens. The game isn't buggy, the emulation is (or at least, it isn't complete), even if the game is writing to 'weird' addresses; it works on the specified hardware.
I think at some point we should just bite the bullet and say that this kind of stuff is not really meaningfully standard anymore, and the few niches that need it can be accommodated with extensions. It doesn't really make sense to place a tax on everything else.
I don't care about non-8-bit bytes either, and a bunch of other things, but having to deliberate add checks to "say no" just because someone might decide to use your code on some odd machine is a pure waste of time and even encourages the sort of lock-in and "environment nitpicking" that proprietary software vendors love.
As the saying goes, common sense is unfortunately not so common these days.
Ignoring the existence of big-endian computers is not really common sense. They DO exist. And it'd be very useful to test in both little and big-endian architectures. I cover the endianness tests by running my TravisCI pipelines on amd64 and s390x, but both run on Linux, which is sub-optimal. I should probably add AIX so that I'm not implying the OS is Linux and importing linuxisms into the code when it should be more portable than that.
If doesn't have any effect until it does and, out of nowhere, you start getting crashes on some new hardware category that you thought was supported. That's not a great place to be.
Even before the move to x86, Apple was making sure OSX (which was based on NeXTSTEP, which was very portable) would not bake in PPC-isms that would hurt its portability. Apple learned the lesson with the painful port of MacOS classic to PPC and that care with OSX paid handsomely with the moves to x86 and then to ARM.
Not because I had any actual need to do this, just because it was fun.
You can get a small AIX system from IBM Cloud for US$50/month. I've thought about doing that – maybe some day I will. If I really wanted to I could afford it.
But I have no plans to ever use Wayland until they get Network Transparency anyway, so as far as I am concerned it is not a huge deal for me (yet). So I will annoy the developers :)
Yes, I do understand the Wayland people are working hard and I appreciate their work, but I wish they would make Wayland easily portable to other UN*X Systems.
Part of the reason Wayland is hard to port to other Unix systems is because most of them are really far behind in kernel APIs compared to Linux. Wayland doesn't really have userspace drivers like X11 did, the current implementations assume that the kernel is just going to handle all of this. The other kernels really need to be implementing evdev, DRM and KMS to get the full range of support for input and graphics hardware. Without those APIs, those other kernels are on their own trying to figure out how to get modern hardware to work and how they're going to plug random drivers into Wayland, when that model is obsolete as far as Linux is concerned.
Other Unix systems are indeed far behind implementing Linux-specific interfaces. Windows and macOS will likewise have a hard time.
Because portability was never a real concern in the Wayland community, it's entirely unsurprising that most of Wayland ecosystem is Linux-specific. In fact, even the parts of Wayland that could be trivially made more portable, such as adding kqueue support in parallel to epoll, still haven't materialized. (See https://gitlab.freedesktop.org/wayland/wayland/-/issues/72 and https://gitlab.freedesktop.org/wayland/wayland/-/issues/214)
>FreeBSD and NetBSD have epoll-shim which implements epoll through *BSD-specific kqueue
The shim works fine already, why fix what isn't broken?
And it's not that those other systems lack Linux-specific interfaces, it's that the ones missing those APIs lack any interfaces for these things at all! There's no way to be portable here, there's nothing to port to! If they decide to implement this eventually, why not copy the Linux API that's already open source and we already know it works, instead of reinventing the wheel? Alternately, Wayland implementations could add (limited) support for userspace drivers but why bother with this when we already know it's an inferior solution that should eventually be replaced with a kernel API anyway? And BSDs have already implemented parts of those APIs separately so we know they don't actually have a problem with them!
If it works fine, why not import it into the project?
> Alternately, Wayland implementations could add (limited) support for userspace drivers but why bother with this when we already know it's an inferior solution that should eventually be replaced with a kernel API anyway?
FreeBSD and OpenBSD are slowly doing exactly that. But DRM, GEM, etc, aren't just any generic API, they're a constantly evolving blob of Linux-oriented calls and data structures. Though at the end of the day this is probably the most defensible aspect of it. And an important distinction here is that the drivers issue pertains to Weston, whereas the kqueue issue applies to libwayland, the ostensibly environment agnostic library which implements the Wayland protocol (libwayland and the Wayland protocol are basically one and the same as the protocol is basically an old-school RPC mechanism inextricable from the implementation as a practical matter).
But none of this need be debated if we just dropped the pretense that Wayland or the Wayland community cared about anything other than Linux. Wayland is simpler than X less because it's better designed and more because it redefined the problem. Network transparency? Not Wayland's problem. Portable across OS and driver environments? Not Wayland's problem. Input device management? Not Wayland's prob^W^W^W....
Why? The shim is generic, it's for porting any Linux programs.
>And an important distinction here is that the drivers issue pertains to Weston
It would also affect the other Wayland implementations, which would need to adopt the same driver system. Those drivers would have to duplicate a lot of code from DRM and Mesa, it's just not worth it to put all that code into another library and then make Weston use it.
>But none of this need be debated if we just dropped the pretense that Wayland or the Wayland community cared about anything other than Linux.
Again, what is there to port to? Those other systems are behind in supporting the APIs needed for modern hardware. There's no alternative to DRM, GEM, KMS, etc.
>Wayland is simpler than X less because it's better designed and more because it redefined the problem. Network transparency? Not Wayland's problem. Portable across OS and driver environments? Not Wayland's problem. Input device management? Not Wayland's prob^W^W^W....
I think you know that X11 wasn't exactly good at any of that. I would think BSD users appreciate all of that because they all have different opinions on how all those things should be done.
I'm honestly confused when I hear people say they still use X forwarding and it has no slowdown. It's known to be one of the worst possible options for remoting, the core protocol doesn't even support image/video compression. RDP runs circles around it.
This is exactly my confusion, why people say there is one. I guess things would be different if I was trying to do it over low bandwidth high latency sat connection or something. At least with connections as slow as 50mbps I have not noticed any issue, I don’t have anything slower to readily test to see what people are talking about. I’ve no doubt it’s theoretically slower due to e.g. compression but that doesn’t seem to be a big practical concern at least for most connections and use cases.
I’ve found it highly concerning because the other solutions will not work for me, but adherents seem to insist what’s working for me doesn’t.
Edit: I want to say that nearly everything in X11 is like this. It seems to work ok at first glance but it breaks horribly when you get outside the design parameters of a mainframe in 1987. So basically, transferring software-rendered images over a high-bandwidth LAN connection and that's it.
I understand that not everyone has > 50 mbps connections but I don’t think this is a contrived fringe edge case either. Meanwhile, the present alternative of “just use rdp” or “just install this package” is completely infeasible for me. And I mean, ant some point, bandwidth is a factor no matter what you do, though I agree more efficient is theoretically better. Unfortunately my job is going to get a lot harder down the road when X is retired for good.
I doubt that X11 is going to disappear for good. It will still always be around for running the odd old X client or for historical curiosity. Just like how you can still emulate the original Macintosh (released the same year as the original version of X!).
So I guess you can say it’s someone else’s problem but it sure feels like my problem.
When sgi ported gl to X11(glx) they were very careful to use only the network transparent communication paths. The idea was you would run your program on the supercomputer down the hall and display it on your badass graphics monster of a sgi workstation. The feature has only worked intermittently since then as future maintainers were not so careful.
edit: I found a guide, no idea if it still works, The feature was always rather niche and fragile.
This comment from the article fits:
> Having it be an option that's turned off by default just ensures that it will rapidly end up broken because of its "niche"ness, and then removed because it keeps causing breakage. Do it all the way or not at all.
The problem is that X11 is actually in a zombie-like state right now. It's not dead because it mostly works, sorta, because people keep stitching extra limbs on when enough people bitch. However, it's not actually alive because nobody really wants to work on it and ...
Every single architectural assumption that X11 made is now wrong.
CPUs are stupidly powerful. Triangles, not bitmaps, are the fundamental primitive (to be fair, this one may be starting to flip back as GPU computing becomes more ubiquitous). Network is vastly slower than CPU and latency has gotten ferociously worse. Input methods aren't just mouse and cursor. Sharing is a problem and secure should be the default. etc.
I have written code for X11. I have written initialization code for graphics cards to deal with X11. X11 coding is hell.
Wayland may not be the answer, but X11 needs to die before we will get to that answer.
Yes, and it is a bad idea. Trusting your apps is how we've gotten in the state we are today with virus scanners and the like. Generally you want to trust as little as possible (at least in tech).
This is only true if you run closed source software on non-free systems. I honestly can't remember that I ever had to use a virus scanner on Linux or UNIX.
So far this never happened on free operating systems running free software on X11. Hence I refuse to believe this is a realistic threat scenario. Severe sandboxing is only necessary for untrusted/non-FOSS software otherwise it just harms user experience an productivity especially for power users.
Yes. It is possible.
On a modern system the browser should run ideally in a virtual machine without any access to hardware or filesystem.
You can go checkout a CVE listing website if you don't believe it.
The thing you may be not thinking of, is that the software itself doesn't have to be intentionally designed to be malicious for it to do harm. Lots of software is written in memory unsafe languages and is full of nasty subtle bugs that can be exploited.
There are also supply chain issues, your favorite open source project's maintainer might have their github credentials phished, and code to do something nasty sneaked into an otherwise normal seeming update. This has happened and is not just a theoretical scenario!
Big endian is so dead, you can literally design a binary network protocol that has all fields 64-bit aligned, little endian, slurp the packets into a buffer, cast the buffer to a struct, and be fine across ALL architectures you'll likely run in production.
This, by the way, is why X11 itself is slowly bit-rotting into irrelevance also: no one wants to maintain it. Most of the protocol exists strictly for legacy applications -- modern ones use the X server as a framebuffer compositor and that's it.
We can't expect people to work for free to support our platforms with no benefit to them.
If you really care, I guess donate some hardware, or pay some developer to work on this, or provide support yourself! I know that sounds rude but it is what it comes down to.
It seems like in this specific case X11 will still support whatever exactly this feature is but will put it behind a config option. The idea above applies in general to open source work though.