Hacker News Reader

Comment by throwaway82652 | Hacker News Reader

I hate to be the one who has to say this, but this is a bad idea. If this is being done for practical reasons then don't waste your time with this. It might work for some clients but you'll get stuck in the same place that every other re-implementation of X11 over the last 30 years got stuck at, which is supporting any kind of weird quirky behavior in Xlib and its large collection of client libraries, and the rest of the quirky behavior in the X server. Eventually you'll end up copying large sections of code out until it slowly transforms back into Xlib and Xorg and effectively it becomes another approximation of an X server. If that's fine with you then by all means keep going, but please don't continue along with the idea that you can avoid this. It has been tried many, many times before.

View on HN

> If this is being done for practical reasons then don't waste your time with this.

Too late, I did, and it's already being used "in production" to power GTK on Haiku!

> It might work for some clients

Well, GTK was the one client I really cared about, and it works for that, so...

It should work for Tk, too, but there's one really annoying bug somewhere in Xlibe which is causing Tk problems; and I didn't figure out what it was in the Tk codebase (this is doubly strange because Tk itself has Xlib compatibility layers to run on other OSes, you would think they would have excised any such quirky behavior.) The "AndroWish" developer said he'd try to take a look and figure out what was going on, at some point.

> you'll get stuck in the same place that every other re-implementation of X11 got stuck at

So far as I've been able to tell, this is now the single most advanced Xlib compatibility layer ever written (well, that's publicly available and locateable, anyway. I wouldn't be surprised if there's more advanced ones behind closed doors.)

At least, I don't see any others powerful enough to run GTK, do you? So I think I've not gotten stuck at that same place then!

>Too late, I did

This reply makes no sense, I would greatly advise against falling for the sunk cost fallacy. If all you care about is GTK, then you could have just done a native port of GTK. That would actually be a lot easier than trying to reimplement every X client library. With GTK4 it should be even easier because the backends were simplified and a lot of the weird X idiosyncrasies were removed, and you might actually have a chance to get hardware acceleration to work without having to go through the pain of dealing with GLX. The problem with Xlib is that's is not just the one library, it's actually about 30 libraries. So like I was saying earlier, it's cool that you have one, but the hard and annoying part that isn't worth it is doing the other 29. Nobody has gotten that far, and if you do get that far, the best way to do it would be to copy a huge amount of code. So the goal here can't really be reached in any practical terms, unless you just want to stop here.

It isn't practical to write a port even for every major GTK version, much less for every other toolkit used with X11. Firefox for example is still on GTK3, and so are many other applications.

Haiku should want some form of generic compatibility, and this is one way to do it (I would have been lazy and used the embedded X server approach for it if I wanted to do it for some OS, but waddlesplash apparently has different tastes). Perhaps one day someone will want to do the work for a 'native' GTK4 port, but that does not degrade the important of this work for Haiku.

Using a hack of Xlib to port firefox is probably the worst possible approach, your rendering performance will be horrible and you still get to deal with the pain of the impedance mismatch from using a different API from what your system is designed to do.

I get the desire to take shortcuts to get things ported quickly but that will not actually help in making these applications properly usable.

Firefox was just one example. There's no practical way to port everything a user might desire - everyone has their own 90% as the saying/joke goes. Haiku needs a generic mechanism, and well, this is one way. Apparently waddlesplash likes this way, and is willing to invest doing the work. This is commendable and will help Haiku adoption. It's also technically interesting and cool enough for a HN post.

I will say that I don't think it's the best path for an alternative OS, and other AltOSs might wish to pick a different path. IMHO, Reimplementing POSIX is a mug's game, a lot of work, while Linux will always be ahead and POSIX isn't much good anyway. Microsoft showed the way with WSL2. This would be a better path for alt OSs in general:

A. Implement the OS with almost no compromises. Ignore POSIX/X11/etc. if they don't match the vision.

B. Port/Create a few application to show and use the benefits of the OS and whatever it brings to the table.

C. Implement an equivalent of WSL2 which should give broad compatibility at the relatively low cost of keeping up with a Linux kernel, given the kernel's rather good backward compat record we wouldn't have to be very prompt with porting new versions.

At which point users would have a good reason to use the OS (B.) and no practical shortage of apps (C.) even if the experience with these apps would be somewhat lacking. One can then concentrate on improving the OS or on porting the most used applications. This acts like a Cuckoo - in the beginning, there'll a mostly Linux userland with a few AltOS apps. Eventually one ports enough so that the system is really AltOS with a few Linux apps on the side.

I don't know how much work it was to do what waddlesplash has already done, but it seems like it's enough for GTK3, and almost enough for Xnest. If they get Xnest to work, then they can use that for anything that is doesn't use xlib, which sadly isn't a lot[1].

Maybe GTK4 would be easier to build a Haiku backend for (as was done for QT), but that doesn't help anybody run GTK3 applications on Haiku does it.

[1] xlib is kind of terrible IMHO; it adds mandatory abstractions that are superfluous at best and in many cases (again IMHO) harmfully force synchronicity on the asynchronous protocol that is X11. Lots of xlib based programs wait for a response when they have other things they could be doing, and the result is poor experiences blamed on X, when xlib's fault. sigh

> xlib is kind of terrible IMHO [...] force synchronicity on the asynchronous protocol that is X11. Lots of xlib based programs wait for a response [...]

Was fixing that the intent behind libxcb?

(honest question, my understanding of X stuff is limited, I just feel to recall Awesome depends on libxcb instead of xlib and read some vague blurb about it back then)

> Was fixing that the intent behind libxcb?

Yes.

Like I noted in another comment, XCB is basically just a loose wrapper around the X11 protocol. It is indeed a lot more verbose than Xlib, and modern Xlib is actually written on top of XCB, so you can use the two in the same application on the same X11 context if you really need to for some reason.

> If all you care about is GTK, then you could have just done a native port of GTK. That would actually be a lot easier than trying to reimplement every X client library.

LOL! Do you really think I woke up one day and said "hmm, I need a GTK port. I know, I'll reimplement Xlib!" without even taking a look at the GTK codebase? Of course I looked at the GTK codebase, and so did a number of other Haiku developers, and we came to the conclusion that writing a native Haiku backend would be a ridiculously unpleasant task (as one other Haiku developer pointed out, GDK backends are typically > 1 MB of source code, and in a dialect of C that I am, uh, if I put it politely, not a fan of.)

It turns out that writing an Xlib compatibility layer was, in the final analysis, a far more expedient and efficient way to port GTK. If I went the long way around, I would have had to write probably over 100k lines of code across GTK, Cairo, and who knows how many other projects (I stopped counting how many dependent libraries I built in the first stages of this attempt.)

Meanwhile, Xlibe does the job with only ~7000 SLOC. That's compared to ~23000 SLOC in the "gdk/x11" folder alone, not to mention all the other X11-related files scattered around the GDK and GTK source trees, or all the ifdefs throughout the codebase, and all the other projects that GTK depends on which interact with the X11 layers in GDK. There is a reason it took literal years to get GTK on Wayland natively...

> With GTK4 it should be even easier because the backends were simplified and a lot of the weird X idiosyncrasies were removed

I find that hard to believe given how deeply some of those X idiosyncrasies are embedded.

But anyway, supposing you're correct: Why should I care about GTK4? Even GIMP of all projects, the (more or less) flagship GTK application and one of the primary reasons a GTK port is generally desirable, has decided that for now, a GTK4 port is "out of scope" (no, really [1]) at least for the immediate future. So even if a GTK4 port was more technically feasible, why exactly would I care if none of the software I want runs on GTK4?

Oh, and let's not forget: with an Xlib compatibility layer, I can also get GTK2 and then whatever applications are still stuck on that; plus plenty of other X11-only things which otherwise I wouldn't be able to get.

[1]: https://gitlab.gnome.org/GNOME/gimp/-/issues/6440

> you might actually have a chance to get hardware acceleration to work without having to go through the pain of dealing with GLX

I didn't yet try to implement GLX but it looks more than feasible. Sure, it's not the most pleasant of APIs to deal with, but implementing it doesn't look so bad.

> The problem with Xlib is that's is not just the one library, it's actually about 30 libraries.

If you mean libraries like Xt, Xaw, Xmu, Xpm, etc., those just run on top of Xlib itself, they don't interact with the X11 protocol directly, and so I can and did build and use those unmodified directly from X.org. For all they know they are running with a strange X11 server that provides an odd but completely valid featureset. Xeyes, which you can see a screenshot of in the README, uses Xt and Xaw, so clearly those work already in this environment.

> Nobody has gotten that far, and if you do get that far

I think I have gotten "that far." I can run Xt/Xaw applications (albeit with a few font rendering glitches, though I just didn't bother to patch that), I can run GTK, and sure, I didn't implement every single drawing mode or every quirky feature, but I don't really care about running Motif, so why does that matter so much?

> So the goal here can't really be reached in any practical terms, unless you just want to stop here.

How about this: You tell me some X11 application you think I can't run under this, and we'll just see if you're wrong (or why nobody cares.)

>LOL! Do you really think I woke up one day and said "hmm, I need a GTK port. I know, I'll reimplement Xlib!" without even taking a look at the GTK codebase?

No. I would also greatly suggest against making these wrong assumptions. But from this comment it actually does sound like you didn't look at the GTK4 codebase.

>It turns out that writing an Xlib compatibility layer was, in the final analysis, a far more expedient and efficient way to port GTK.

This analysis is extremely incorrect and wrong. I'll get more into the details.

>If I went the long way around, I would have had to write probably over 100k lines of code across GTK, Cairo, and who knows how many other projects (I stopped counting how many dependent libraries I built in the first stages of this attempt.

Implementing the full X11 protocol with all the extensions and all the quirks in the server is much, more more than 100k lines. Also this entire sentence is extremely incorrect and I'm very confused as to what you were trying to do, you don't have to do anything in Cairo or Pango because those support client side rendering and have supported that for a really long time now. I assume you were basing this analysis off an ancient Haiku port or something that was trying to do things the old way? That stuff is obsolete.

>Meanwhile, Xlibe does the job with only ~7000 SLOC. That's compared to ~23000 SLOC in the "gdk/x11" folder alone

This comparison is complete nonsense. Please don't sink yourself to the level of doing this. The GDK X11 backend includes support for a bunch of extensions that you didn't implement. So no, it's not "doing the job" at all. IIRC if you look at the GTK4 backends for the more modern windowing systems (Wayland, macos) those have both shrunk to around 13000 SLOC. A native Haiku backend would probably be even shorter if you dropped all the optional features and only implemented the bare minimum, I bet you could do it in... around 7000 SLOC. There is nothing magical about any particular approach to this, it's all the same code you have to write in either case. Draw a box here, render some text here, translate these input coordinates, etc. You can't avoid this by implementing Xlib or doing any other approach.

>and all the other projects that GTK depends on which interact with the X11 layers in GDK.

Well actually those will still break with your approach if they link against the other libraries or depend on some X server functionality.

>There is a reason it took literal years to get GTK on Wayland natively...

Actually it took years because the protocol was still being worked on while the backend was being developed.

>I find that hard to believe given how deeply some of those X idiosyncrasies are embedded.

Well it's true, they were removed in GTK4, the entire rendering pipeline was rewritten.

>I didn't yet try to implement GLX but it looks more than feasible.

You could get it to work but it's not going to be feasible without also hacking the mesa implementation or re-implementing several parts of mesa yourself along with the GLX extensions. IMO this is not a good way to spend your time, please don't waste your donors' money on this. If you have a working EGL backend already you can avoid all that and just use the EGL support that GTK has already.

>But anyway, supposing you're correct: Why should I care about GTK4?

I don't know, you tell me. When you say "GTK" to me that means all versions of GTK that are currently in use by applications. If all you care about is GIMP then I would still say a better approach is to do a native backend because then you can also just skip everything that isn't used in GIMP. It's the same approach really.

>Oh, and let's not forget: with an Xlib compatibility layer, I can also get plenty of other X11-only things which otherwise I wouldn't be able to get.

Well no, this is also wrong, to get the full X compatibility (including network support) you will still need to port the full X server. And if someone ever wants that, basically all the work you did is going to be a waste and will have to be thrown out because then you're right back where you started with needing an X server. This is not a new revelation, Linux distros have been through all these conversations before, several times over the last 30 years in fact. I'm not telling you this because I'm trying to give you a hard time, I just wish other OSes would stop repeating the bad mistakes made by Linux.

>If you mean libraries like Xt, Xaw, Xmu, Xpm, etc., those just run on top of Xlib itself

No this would be the other libraries that implement protocol extensions.

>How about this: You tell me some X11 application you think I can't run under this, and we'll just see if you're wrong (or why nobody cares.)

You're being unnecessarily combative about this, please stop. I actually wish I was wrong about this, and it was as easy as you're suggesting, but experience says it's not, and also the problems that you've stated with your approach aren't helping either. But anyway, according to what you said, right now it's any application that requires XInput or XInput2 or requires 3d acceleration or wants to get run over the network. Or even just stuff like GIMP will have degraded functionality, for example you can't use drawing tablets with that right now. I would really advise against making these kind of comments if you're trying to represent a project or get more funding.

> But from this comment it actually does sound like you didn't look at the GTK4 codebase.

I gave it a cursory glance, but indeed I didn't look at it much in depth, because all the big GTK-based applications don't yet run on it, so it was a moot point.

> Implementing the full X11 protocol with all the extensions and all the quirks in the server is much, more more than 100k lines.

Sure, but I have already stated, multiple times, that we neither need nor want to implement all the extensions and quirks.

> you don't have to do anything in Cairo or Pango because those support client side rendering and have supported that for a really long time now.

GDK's backends presume Cairo has support for native "surfaces"; e.g. the GDK X11 backend uses Cairo's Xlib interfacing layer to draw stuff to the screen. I didn't look too closely, but the other backends seem to make similar usages on Windows and macOS. So I would have probably had to either revive the BeOS interfacing code in Cairo, or would have had to do lots of software copies in a Haiku backend of GDK in lieu of a Cairo interface.

Pango indeed doesn't deal with native "surfaces", so that wouldn't be a problem.

> I assume you were basing this analysis off an ancient Haiku port or something that was trying to do things the old way?

Nope. There may have been some work on GTK 1 for BeOS or something like that, I can't recall, but I didn't go looking for it, much less analyze it.

> The GDK X11 backend includes support for a bunch of extensions that you didn't implement.

For now. Some of those extensions (like XSync and XInput2) are probably worth implementing. This also ignores all the other X11-related code scattered throughout the GTK/GDK trees, anyway.

> A native Haiku backend would probably be even shorter if you dropped all the optional features and only implemented the bare minimum

Okay, but that's again GTK4, and totally ignores GTK2 and GTK3 which quite a lot of applications still use, and the entire point of a GTK port is to run GTK applications. Not much use in porting a version of GTK that very little supports.

> You can't avoid this by implementing Xlib or doing any other approach.

No, but I can decide that the GTK codebase is so far outside the realm of my and the other Haiku developer's programming tastes that we'd quite literally rather write an X11 implementation than deal with it. Even the minimal time I had to spend with the GTK code in a debugger while fixing incompatibilities in Xlibe was more than enough, thanks.

Furthermore, GTK is entirely C while Haiku's APIs are C++. Is GTK ever going to accept a C++ backend upstream? Almost certainly not. So if I am going to have to maintain this totally outside GTK, forever, why not do it in a way that's much more resistant to GTK changes?

And, again, there's more than just GTK here. Someone took Xlibe and apparently got some audio plugins from Linux (LV2, I think) to work with minimal code changes.

> Well actually those will still break with your approach if they link against the other libraries or depend on some X server functionality.

Again: which other libraries? You mean like Xinerama or something? If something actually needs Xinerama, or any of those other libraries, then we can implement those, too. (So far, those have turned out to be entirely optional, I haven't yet tried to build something that couldn't just disable those features.)

> Well it's true, they were removed in GTK4, the entire rendering pipeline was rewritten.

Okay. What about input events? Does GTK4 still have keysym values that map 1:1 to X11 keysyms, or did they break with that?

> Actually it took years because the protocol was still being worked on while the backend was being developed.

It's my recollection that Qt had a working Wayland backend long before GTK did. Am I incorrect?

> You could get it to work but it's not going to be feasible without also hacking the mesa implementation

I don't intend to hack the Mesa implementation of GLX, I'd just implement GLX on top of the Haiku OpenGL APIs directly (which use Mesa internally but that's an implementation detail.)

> IMO this is not a good way to spend your time, please don't waste your donors' money on this.

GLX is unimportant enough that I don't intend to spend time on it at present anyway. When/if there's a demand for it, I'll take a stab at it. But I've worked with GLX before, and I don't think it'll be nearly as much a challenge as you seem to think it will be.

> If all you care about is GIMP then I would still say a better approach is to do a native backend because then you can also just skip everything that isn't used in GIMP.

There's also Inkscape, wxGTK, and plenty of other GTK applications besides GIMP. Not to mention X11-only applications like AzPainter, etc. So it's not "just GIMP," but if whatever GTK version I'm porting, GIMP doesn't support, that's a good argument I picked the wrong GTK version.

> Well no, this is also wrong, to get the full X compatibility (including network support) you will still need to port the full X server.

Yeah, but I wasn't talking about that in the section you replied to, I was talking about things like Tk or AzPainter or the like which have X11/Xlib backends, and so by implementing this we also get those, not just GTK.

> And if someone ever wants that, basically all the work you did is going to be a waste and will have to be thrown out because then you're right back where you started with needing an X server.

Not if you port the X server using this, via Xnest! Which, like I mentioned above, I did get to compile and attempt to start but it crashed, and I decided I wasn't going to spend more than five minutes poking at it in a debugger because I didn't care enough about having X11 forwarding. If someone else wants to, then I imagine it will be much easier to patch the X server and Xlibe than writing a totally new X11 server backend for Haiku.

> This is not a new revelation, Linux distros have been through all these conversations before, several times over the last 30 years in fact.

I am quite aware, I've read some (or even many) of them.

> I just wish other OSes would stop repeating the bad mistakes made by Linux.

Hear, hear!

But this isn't one of those. As you pointed out yourself, nobody else did this before successfully; and it seems we've now already been more successful than Linux. Isn't that evidence that, just maybe, we aren't repeating a mistake here?

> No this would be the other libraries that implement protocol extensions.

As noted above, either most applications treat those as optional, or we can just implement them, too (Xft for instance would be pretty easy.)

> You're being unnecessarily combative about this, please stop.

I'm not sure why I couldn't say the same about you.

My offer still stands: tell me the name of some X11 application you think I can't run under this (obviously stuff like "window managers" don't count), and I'll make a pass at it and we'll see what happens.

> I actually wish I was wrong about this, and it was as easy as you're suggesting, but experience says it's not

I didn't anywhere say it was "easy." I'm drawing on many years of experience with both the Haiku APIs, the X11/Xlib APIs, graphics programming and widget toolkits in general, and even with that there's a lot of good old trial and error.

> But anyway, according to what you said, right now it's any application that requires XInput or XInput2

Quite literally because I just didn't implement those yet. I read through the XInput2 headers briefly and while they look particularly unpleasant in some ways, nothing looks technically impossible or even difficult, just tedious to deal with.

Plus it's another thing we would only have to do once: implement XInput2, and now all GTK applications, AzPainter, and anything else that can support it now has drawing tablet support on Haiku.

> I would really advise against making these kind of comments if you're trying to represent a project or get more funding.

Look, a little snark never hurt anybody.

The fact is that you are being extremely pessimistic about whether this will work at all, when in fact it is already being used "in production," and most of the other things you are saying are not possible or very difficult I have already investigated and made similar assessments to the ones I made before I started this.

So all evidence points to you being largely wrong, and your supreme confidence that you are correct despite all the evidence to the contrary makes it quite attractive to have a bit of fun while trying to demonstrate where you are incorrect.

That's nothing new in open source, or even software development in general, where much worse like actual insults and derogatory comments are much more an unfortunate norm, and toxicity is often the rule of the day. Plenty of stuff that goes on (or at least used to) on the Linux kernel mailing list would get you warned and then banned from Haiku's discussion channels. But at the same time, there is a place for criticism, even strong criticism, in any healthy discourse.

I'm curious why this approach was chosen for GTK, whereas Qt was ported directly. Is there some detail about GTK that made the approach taken with Qt infeasible?

Yes: the GTK codebase is, uh, not fun to work with and write backends for. See my comment above (https://news.ycombinator.com/item?id=31192099) and also one of my replies on the Haiku forums (https://discuss.haiku-os.org/t/xlibe-an-xlib-x11-compatibili...) for more details.

7 kLOC is a lot smaller than I had guessed this must have been. Pretty cool!

I am not even sure about what quirks you mention from Xlib. That said, I do think that having this server-less is going to be challenging. Even if the author claims this would ease things like DnD, I am not entirely convinced...

Reply on news.ycombinator.com