Opening Windows in Linux with sockets, bare hands and 200 lines of C
hereket.com
hereket.com
Of course, as any Gen-Z graphics hacker will tell you, That's Just Not How Things Work Anymore, and rendering is and should be client-side, leaving the compositor to only present the final display. But it was fun while it lasted.
As long as you only want to do very basic graphics -- no antialiasing, no color blending, no subpixel coordinates. And it isn't even that fast; everything ends up rendered on the CPU, possibly even all on a single thread, using code paths that haven't seen much optimization in the last 20+ years.
Even xf86-video-intel, the intel X11 driver package, on my system depends on pixman.
In the end, modern 3D APIs are also just a 'wire protocol': rendering commands are recorded by the CPU into local command buffers and played back by the GPU (which is often connected to the CPU by a comparatively slow bus), the only limiting factor is the amount of data that's communicated between the CPU and GPU - and of course any additional latency that would be added by a network connection.
For comparison, SGI used a wildly different architecture underneath, which heavily impacted performance - it's also why "xsgi" would default to indexed colour and you'd use 24bit/32bit visuals only for some windows - the server would use indexed colour visuals on the actual framebuffer to reduce memory bandwidth and composite everything together in hardware.
It's also why there were "X11 Overlays" for GL - it meant you could easily implement parts of the GUI in X11 and still render on-top of direct-rendered visuals that bypassed X11.
But I was really thinking about going further with all or nearly all of the X server executing on the GPU with only cpu shims for input peripherals, networking etc.
X11 could be implemented similarly - essentially sticking a complete standalone "X11 terminal" on an Add-In Card - the current GPU architectures aren't necessarily fitting for implementing it directly, but combine essentially an embedded OS running on extra CPU handling the protocol interactions and I/O then use a possibly more tailored interface to talk to GPU (compare with AMD's promise of HSA, or Xbox post 360 or PS4/PS5 architectures where GPU and CPU are on common memory).
NeWS-style engine (possibly with something simpler than full postscript) would work great with that, especially if you had properly shared interface libs on it.
I exaggerated, but only a little bit. Even the widget library zoo has somehow found an equivalent on the web.
XWayland is here to stay for a long time.
And if there's a need for this it's trivial to get the same DX via a library that's optimized for small on-the-wire serialized format of scene descriptions. (For whatever backend over whatever wire, local shared memory message passing, unix socket, TCP/UDP, WebSocket, etc.)
If you have a remote server with UI, you can set up a X Window server on your Windows/MacOS machine and forward via SSH X messages to use GUI apps on your server, but view the result locally. The Responsiveness of the UI depends on your network capabilities.
I don’t think x over ssh is useful for much, can you imagine a browser this way?
A bit rough around the edges but brilliant for some usecases.
That's fine on a LAN with low or sub ms latency. If you need to connect to a server on the other side of the country, you'll want a less chatty protocol, e.g. X2GO (which gives additional benefits, e.g. of restartable sessions).
It worked really well up until a certain version (18, I think) - after which something changed in the UI toolkit and it was suddenly much more laggy. I'm guessing the UI toolkit's newer incarnation required a lot more round trips.
It in fact works perfectly fine.
The catch is latency. In my case both hosts are on the same colo.
I remember, 20+ years ago running quake3 over remote X to my roommate machine and I was surprised that glx actually worked over the network with no issues.
My personal experience with Oracle products has always involved terrible UX, so I'm not surprised they built a terrible X client.
I've run browsers over X, sometimes actually remotely, and it's OK, but not great. I wouldn't recommend it for video, audio is out of scope, and start up can be very slow sometimes. It's something that used to work better, IMHO. Sometimes you need to try a couple browsers and see which one works better. As a sibling notes, it all feels untested, and I'm sure if developers were testing, they'd fix some of the things that are easier to fix.
It’s almost always more usable to run a desktop session in Xvnc on the server and connect to it with a VNC client, because if you get disconnected, you can just reconnect.
This isn't very true. The X protocol is very async and lets you batch plenty of things when a response is required.
1. TCP streams require stalling when packets are dropped to keep the stream in order
2. X _requires_ by design that commands are performed in order
Which means that using something that can do UDP and manages it's own sequence ordering can do significantly better. This is why things like RDP, PCoIP, etc could do full frame rate HD video 15 years ago and you still can't with X protocol over the network.
Breaking up the screen into small 16x16 chunks or so, encoding on the GPU, and shipping that turns out to be significantly faster.
Especially when you take into account that virtually _nothing_ draws with X using X drawing primitives. It's almost all using Xshm for anything non-trivial.
> Which means that using something that can do UDP and manages it's own sequence ordering can do significantly better. This is why things like RDP, PCoIP, etc could do full frame rate HD video 15 years
No it isn't. It's because those things actually compress the video, and X-forwarding generally doesn't. The transport protocol is completely irrelevant, it's just a bandwidth problem.
I've X-forwarded Firefox between two desktops on 10G Ethernet. I can watch 4K video, and I genuinely can't tell the difference between local and remote.
But: > This is why things like RDP, PCoIP, etc could do full frame rate HD video 15 years ago and you still can't with X protocol over the network.
Compression is going to have a much bigger impact over a large motion than most anything else; you can stream video over HTTP 1.1 / TCP thanks to video codecs, but X (sadly i think, seems like such an easy thing that should have been in an extension, but even png or jpeg never made it in) doesn't support any of that.
> It's almost all using Xshm for anything non-trivial.
Xshm is not available over a network link and it is common for client applications to detect this and gracefully degrade.
Interestingly, that is xlib's behavior moreso than inherent in the protocol. xlib assumes connection lost is a fatal event, but if you're doing your own socket, you can choose to do this differently. (or, even with xlib, you can throw an exception from the connection lost callback and regain control before letting it abort)
Some server state is lost, but it is possible, with some care, to recreate that from your client upon reconnecting. You can even connect to a different server and somewhat seamlessly migrate windows over.
But yeah it isn't commonly done.
Emacs is the one program I know of that can actually do this. It can pop up frames (Emacs lingo for windows) on multiple displays at the same time and even mix tty and X11 frames. The Emacs session survives connection loss fine, frames on other displays continue working and you can reconnect if desired.
The one caveat is that Gtk used to have a bug that caused it to uncontrollably abort on connection loss but I build my Emacs with --without-x-toolkit (so it uses raw xlib, no toolkit) and that configuration has always been robust and performant. If I remember correctly the Gtk bug might be fixed now too.
Xvnc is fine if you want a desktop session, but usually I just want one program, so X11 is a better fit for me. Once you have the infrastructure setup, it's really easy to go from a normal shell to running a graphical program. For me, I use a Microsoft Windows desktop environment, have X11 forwarding enabled on my usual shell targets, so I just start the X server and run the program. I could have the X server autostart (it's unobtrusive if there's no clients), but I haven't yet. On a mac, the Xserver starts automagically (once installed), so it's just a matter of enabling X forwarding in your ssh config to hosts where it's likely. Even less work if you run an X desktop.
This is a huge improvement over most "web applications", whereby your client (web browser) sits there looking like it's doing something when your network connection goes away. Worse still, you don't know whether the last (trans)action you sent to the server was processed or not.
I hear this might become a native built-in feature in Windows but for now I just run an X client.
https://devblogs.microsoft.com/commandline/wslg-architecture...
The terminology is of course a bit counter-intuitive, since it's program-centric rather than computer-centric - the local terminal is the server and the remote computer (or rather program) is the client, utilizing the server as a display device.
It's a bit like a text terminal (VT-100 etc) except of course the X terminal has a network connection., With the VT-100 if you wanted to connect to a remote system you'd have to use a modem (acoustic coupler) to dial into a terminal server on the remote system. I don't think anyone ever made a text terminal with built-in telnet and ethernet.
Maybe not, but there were terminal servers with 10 serials ports for the glass terminals and an Ethernet connector.
I still have a USB terminal server for when I want to play with old stuff like my DECTalk (VCR-sized speech synth) which has two DB-25 serial ports on it.
Yeah, with no compression, that's right.
(Talking about native implementations here not xlib bindings).
I remember seeing this on HN a few months ago and thought it was super cool.
Also just wanted to let you know that I sent you an email. Don't feel obligated to respond or anything, just wanted to let you know because I sometimes get emails from people on HN but they go to an inbox I don't often check so I often don't see them until long after unless they tell me in a comment that they emailed me.
With the important distinction that all those protocols very intentionally use human readable ASCII requests/responses. Of course, for performance sake, ease of debugability was sacrificed. Which increases the value of a standard library to hide the protocol.
Typo: Simple Mail Transfer Protocol
And I suspect many of us tend to think of those as magical abstractions best dealt with through libraries as well:)
Simpler times, when we thought we could trust everyone on the net :)
But I'm pretty sure it's more complicated than I think. :-)
With Wayland, the protocols are specified in machine-readable XML files, not unlike how XCB works, and typically a program called a "scanner" would read those XML files and perform code generation. So if you wanted to generate proper Wayland bindings, you'd need to write the scanner. The scanner is not terribly complicated. I did it in Go with around 750 SLOC, not counting the supporting code. It's not released, but here's just the code generation part, if you are curious. Last I used it, it could generate all of the Wayland protocols, and I was successfully using dozens of them from my own Go bindings.
https://gist.github.com/jchv/adcb6de1c9dc3d0112dea704d753803...
Of course, if you just want to implement Hello World, you don't need to use code generation, and the resulting amount of work to interface with X11 is ultimately similar to that of Wayland. But there are two huge caveats:
- Right now, the EGL interface for Wayland requires that you pass libwayland wl_surface structures. That means to get on-screen hardware accelerated contexts in Wayland, you have to link to libwayland (maybe making compatible structures would work if you're into that kind of thing, but definitely no easy options.)
- X11 has many server-side features that Wayland doesn't. Wayland's core protocols don't specify even window decorations or cursor shape, so a Wayland client has to implement drawing window decorations and loading cursor themes to properly support all Wayland compositors... even if they would prefer to use xdg-decoration and cursor-shape-v1. This is on top of things that no Wayland compositor has support for, like an equivalent to XRender or indirect rendering with OpenGL; those things just don't exist.
So it's far from perfect. Though, it's also not like X11 was free of cruft or design issues, either. I'm pretty sure with X you also have to implement cursor themes yourself, by virtue of linking to XCURSOR. And while you can use XRender and other server side rendering features, which will be way more efficient for remote usage than simply opening a GL or Vulkan context, most software (GTK, Electron) doesn't do this anymore, and either ships buffers over shared memory or opens a direct hardware accelerated context.
Personally, the problem I'd really like to see fixed is the EGL ABI one, but I'm pretty disillusioned by the bullshit in anything RedHat indirectly touches (e.g. Freedesktop.org), so even if there's a chance I could help, at this point, I'd rather spend my time working on other things.
Windows sort of has this with RDP, but it's tied in with the app's GDI desktop, and I don't know whether it works without buying a bunch of video cards for the headless app server. NeXTSTEP had Display PostScript (remote rendering worked like printing!) but macOS lost support for it.
On wayland from I am also getting good results with waypipe opening individual apps from VMs to make it a poor man's QubeOS without the complexity and without having to open the whole remote desktop.
Just last week I was debugging remotely an art installation which uses my software, https://ossia.io and was running on a Pi 5, I compared X11 and VNC and X11 was really much more useable even over the internet.
Even if this were true (which it isn't), there's a lot more to a GUI than the G. A lot of nice interoperability is provided too, like clipboard integration, dragging and dropping, mixed windows on the same taskbar, etc. Far more pleasant to use than awkwardly going to a full screen thing to get a window out.
I just tested in an ESXi VM[1], and Windows 10 will happily boot and accept an RDP connection without any (virtual) display hardware at all:
https://jasomill.at/HeadlessRDP.png
Whether a particular computer will boot Windows without display hardware depends on the firmware — in my experience, some workstations will, some won't, and I honestly haven't seen a modern x86 server without some sort of integrated graphics to support out-of-band management, often with a VGA port as the only physical video output.
[1] By hand-editing the VM's .vmx configuration file, setting
svga.present = "FALSE"
hwm.svga.notAlwaysPresent = "TRUE"
https://jasomill.at/vmx.ss.html#L-808
In fact, two definitions and a couple trivial syntax fixes are enough to get Chez Scheme to evaluate it without warnings:
$ awk -f - vmx.ss <<EOF | scheme -q
BEGIN {
print "(define / fxdiv)"
print "(define (log . args) (for-each display args))"
}
{
gsub(/\(\)/, "'()")
gsub(/ 0x/, " #x")
print
}
EOF
$
A more real answer, is the X server manages access to the input and output devices... Roughly speaking, it lets a client define a region (rectangle) and get clicks and mouse motion and keyboard input sometimes (maybe too often), and lets the client send things to be displayed. In modern times, that's mostly images, but it used to be lines and curves and letters and things, or like OpenGL display lists. The server can tell the client when it is exposed and need to redraw or it can use a backing store to keep obscured parts of the region local to the display. Additionally, clients can adjust other client's resources (this is what a window manager does) and clients can communicate with each other through the server (no full mesh like in Erlang dist) ... that part is a bit confusing.
On Windows and macOS (either one), there's no sense of a network involved, similar things happen, but mostly with system calls, I think. Otoh macOS X has all those mach ports? does UI go through that? But there are X11 servers for most platforms that integrate reasonably well, so it's not like you can't use X concepts there, it's just a bit more setup to get started. Windows also has RDP which can be used to run a program on one computer and display it on another.
The benefit of this approach is you can take better advantage of asynchrony... Many GUI libraries and toolkits run in a synchronous model where you do a request and can't continue until you get the answer. That's fine when everything is fast, but when there's a network between the client and server, it's better to send requests when you have them and only wait for the response when you need it. (See also xcb vs xlib)
Real life scenario: you have a headless server that has GUI software installed at a remote site. You connect to the network from home with a VPN. Then SSH in to the server with "ssh -X user@server" and run the GUI program in that terminal. The GUI appears on your local display. SSH sends the X11 protocol traffic through an encrypted tunnel from the remote server to the local X11 display server, because of the -X.
X11 got network transparency out of the box (a sibling comment touches this), and the capability of switching out the components more easily, while Windows had less work to do to smooth out the overall desktop experience.
But for many years now, they switched to an approach of rendering all the graphics on the host side and then sending a video stream to the client using standard video compression. I think this compression based approach scales better for the common use cases, especially office apps where the display is mostly static. I guess the approach has almost gotten good enough that you can play games remotely that way.
The problem is that the more complex the UI, the sooner you reach a threshold where "just transmitting the bitmap" is faster and/or less data.
E.g. consider rendering a simple button with X11: You'd "just" send a request to render a rectangle, maybe fill it, and send the string for the label. 2-3 small requests. But then the moment you have a UI with a gradient, a drop shadow for the text, a differently shaped border, a shadow for the border, you suddenly add on enough requests that it's very easy for the numbers to look different. Especially because compressing these bitmaps reasonably well tends to be easy.
Modern X11 clients increasingly render into client-side buffers already because even when using X, that's often better when on a local machine and too few of us use it over the network often enough for that to be optimised for.
Having the option in the display subsystem of picking either based on what will perform best is a good place to be, but the more complex the UI the less often the simple primitives will be worth it.
There is no 1:1 correspondence on Windows. Some of what X11 does is part of Win32, other parts are in explorer.exe, others are built in at deeper layers; there are multiple alternative systems available on Windows too.
Ultimately what X11 gives you as an app is a way to draw something on the screen, and to get input events from users. X11 also coordinates this between multiple separate apps, so that you get movable windows and focus and other such behaviors without having to have each app coordinate manually with every other GUI app. Copy-paste is another similar cross-app functionality that X11 offers.
The way X11 does this is through a well defined protocol. Instead of relying on system calls, you open a socket to some known port and send drawing commands there, and receive input events from it (of course, many libraries abstract this for you). Because of this, it can work transparently regardless of whether you are drawing on the local machine or on a remote one. So, X11 itself can work as a Remote Desktop solution as well without the need of a separate program or protocol (though there are significant differences with pros and cons).
As far as I know, you have to ask the server for the active bit depth and send bitmap data in a matching way, meaning every client needs to have code to convert to every pixel format under the sun.
Maybe that's not true though, I'm not sure.
For example, if a struct contains a char a followed by an int b, typically b would be at offset 4, even though sizeof(a) is 1.
To get tight packing, you need to explicitly tell the compiler to disable padding.
C structs have packing, alignment, and endianness gotchas that can be portably addressed with byte-level de/ser.
Example: https://gist.github.com/cd089675a6088b0b8482c005a0e3897a
Also a lot requests have dynamic size and cannot be as easily packed and serialized to be sent over the "wire". So for example you could put pointer to string into struct, malloc required space, then assign pointer. After when you want to send data you would have to write serialization methods to put it in correct order. It is all fine and should be done when you build bulletproof system to manage X11. But here these techniques just draws too much attention away from X11.
In some cases you could manually pad your structs to satisfy lowest common denominator architecture requirements, and use compiler extensions mark them as packed (which should be a noop on the architectures you've considered), if the protocol padding/alignment is compatible with those requirements. But the code still theoretically could break on ABIs you didn't consider.
Manually packing and unpacking bytes is the safest way to do things in production.
Of course there is __attribute__((packed)) and __attribute__((aligned(8)))
Learn about struct packing and let the compiler fill in the "array" for you.
Anyway, a good way to learn stuff. Hard mode, FTW!