Porting the Unity Editor to Linux: Stuff I Wish We’d Done Then
natoshabard.com
natoshabard.com
> Same policy as with our runtime; in order to keep our own sanity, we will officially support Ubuntu Linux.
…
> Installer will (most likely – it’s one of the things we didn’t do yet) just be a .deb package.
I really wish folks would support Debian first, and let Ubuntu support flow from that.
The great thing about the Linux community is that it's full if crazy smart people who will go out of their way to make sure stuff is working on their distribution. And of course if something isn't working on your distro, it'll be just like the player -- send us a bug report or poke us on the forums, and we'll do what we can. But in order to not spread our resources too thin (especially considering that this is starting off as an experiment), we have to set some boundaries, and we need to do what will make sense for the largest number of users.
And case sensitivity problems will exist as long as we have a mixture of some platforms (Windows/OS X) using case-insensitive filesystems by default, and others (Linux) using case-sensitive filesystems by default.
While generally it's also true the other way around (if something is made for Ubuntu, it's likely it will work on Debian without big issues), it would be nice for companies to recognize Ubuntu's roots and make the support official.
Of course in practice the community will quickly report back to you something like "hey, change your dependency to include libjpeg-turbo8 or libturbojpeg0, in Debian there's no libjpeg-turbo8" (real example from early Steam package), however, maybe it's just me, but when I see something having official support for Ubuntu only it sounds like "we were too lazy to give it a try on other distros", while explicitly stating support for Debian-based distributions starts sounding much more professional :)
That said, the reason why they support Ubuntu is that it provides static targets. 12.04 will be supported until late 2017, which means that they users two years from now can install a .deb built today. Likewise, supporting 14.04 means .deb packages will work until late 2019. Especially for companies that want to develop on Linux, there's a huge benefit in first-party support from your vendor for the next four years.
(I was working on the Windows -> Linux path, but for some cross-OS examples that aren't that I was surprised to learn that while toupper follows your locale on Linux, OS X uses the "C" locale by default. Or that shm_open has a tiny size limit on OS X.)
I think the best you can do is (1) write a lot of tests; (2) whenever you add an OS-specific snippet, use the same ifdef string so it's easy to grep for.
Who made the decision to have case insensitive file systems? If I had a time machine they would be high on the list of people to visit and give a stern talking to.
While as a programmer, I prefer the level of precision case sensitivity provides, I'm hard pressed to make the case that it is better for an average computer user.
Basically, you can't "just do case insensitive everywhere", that doesn't fix the confusion and inconsistency.
[1] http://www.i18nguy.com/unicode/turkish-i18n.html (see table titled "English vs. Turkish Case Mappings") (this is just the first example I could find with a quick search)
I stand by what I said for ASCII, but in a Unicode world, I think you're right. When the mapping stops being simple, the principle of least surprise reverses in the favor of case sensitivity.
Indeed; if everything were case-sensitive, or case-insensitive, we definitely wouldn't have these problems. But alas, it's not, and in 2015 here we are still having problems with case sensitivity. :-P
> While as a programmer, I prefer the level of precision case sensitivity provides, I'm hard pressed to make the case that it is better for an average computer user.
I also personally prefer it ('a' and 'A' are not the same character!!!!), but I'm not sure if thats because it's inherently 'right' or because I'm used to Linux where case sensitivity is more or less a given. :-)
Problem is that this goes back a long ways, e.g. back to the days of 36 bit computers.
Why 36 bits? Because that was just enough to encode 10 decimal digits, the standard back then for scientific computation.
Back then as in the days before core memory was used, so not much main memory either, and it was still expensive and painful after core was developed. So with that limited a budget, 6 bit character systems were common, that's enough to get things done and you can fit 6 in a word. One reason mainline Lisp doesn't use '?' in function names, why old languages like it and FORTRAN are or were case insensitive.
mv: ‘test’ and ‘Test’ are the same file
/tmp $ touch test
/tmp $ mv test TestThis 2001 page has some possible reasons. (Much simpler; some benefits).
"Of course we tried to be smart in the early days, but if you don’t set up a way to actually verify that what you’re doing works on a case-sensitive file system, then it will never fail that some well-intentioned programmer throws a toLower() in somewhere and ruins the party."
Because yeah, if nobody's running or testing a code path, it will absolutely decay. You can't write anticipatory code to work for hypothetical future requirements that you don't necessarily understand and are never testing.
I'm a little curious to know what they were doing.
EDIT: Dammit, I should have kept reading, the very next sentence has: " Lots of compilation errors that involve this-c++-template-thingy-with-lots-of-angle-brackets does not match that-c++-template-thingy-with-lots-of-angle-brackets-that-has-a-const-somewhere-in-the-middle."
and personally I would have thought that of course you need the consts to match and it would have been this way for literal decades now.
In what way are the native GUIs better? Are they faster/more responsive or do they look better?
Which would be more work over the e.g. 1,2 and 5 year period maintaining three guis or one? (Or is QT not quite cross platform so one ends up maintaining three inferior versions anyway?)
This is actually an annoying problem. As a linux user lack of consistency between apps is frustrating but similarly I know people that struggle with changing from mac to windows regularly.
You can just go test it yourself, go run VLC and see how "native" it feels. Its using Qt everywhere except Android.
But if you want to develop cross platform native looking software... Qt is absolutely it. Only option. And it gets better each release. Otherwise you are going the route of the Unity guys, rewriting your program for every single system.
If you want to develop cross platform native looking software, Qt most certainly is not the only option, wxWidgets is. In fact, as you yourself pointed out, Qt isn't really native, just tries to fake the native look and feel. wxWidgets actually is, as it's a thin wrapper around the native toolkits.
IMHO it is a better idea when the application enforces the same UI across platforms (like QT) than the platform expecting the same UI for all programs. On top of that QT does support native look&feel.
A lot of productivity software uses Qt. If it were not using native look and feel it would piss everyone and their grams off when system level keyboard shortcuts didn't work, when menu bars weren't oriented system native, when it didn't use the system file dialog, etc. Integrating with the OS is essential to workflow unless the application literally is your entire workflow.
You last sentence tells me you have never used a QT app on a Mac, because it does not look and feel native.
I still prefer it over any other free IRC client for OSX simply because the feature set rocks.
Precisely because users (at least iOS and Mac users) do care.
Doing native is without a doubt the right call.
Since most Qt apps on linux come from the KDE ecosystem, they often pull in a lot of KDE dependancies, which means a lot of extra packages lying around your system, creating clutter, which might bother some people. The relatively heavyweight nature of most Qt apps makes them less appropriate on systems that use just a window manager rather than a full desktop environment.
Sure, a lot of GTK apps pull in a bunch of GNOME dependancies, but there's also a large contingent of GTK apps from the lighter-weight XFCE world that don't.
Outside of linux, Qt has won pretty decisively, mostly because the GTK developers aren't interested in making GTK apps look good on Mac/Windows, so long as they run.
Although I don't actually mind as I find that once you've got one KDE app you end up installing more as they are often quite a lot better (for me) than Gnome ones - and disk space is pretty cheap these days anyway.
https://developer.ubuntu.com/en/apps/
The "out of place" argument is one I commonly used by Mac and Linux users because they want everything to look native in their OS of choice and disregard dev time (to use a different window framework for each OS) and look in every other OS.
No one can win that argument to favor THEIR own OS look over another.
Qt apps are pretty reliable across platforms and are speedy to develop so that's what should be important instead of getting an impossible native look.
Besides, the whole "wasted effort because there's 2 projects that 2 similar things" is a fallacy.
As a Qt developer let me give you a big resounding no and ask you a question: Have YOU developed with Gtk in Windows?
We chose GTK over QT for Linux because Unity 5.1 ships with CEF (https://en.wikipedia.org/wiki/Chromium_Embedded_Framework) as the embedded browser and CEF already pulls in a dependency on GTK).
Yes, file system case-sensitivity and dispersed strictly hardcoded pre-processor directives were the problem, but they were the least of the problems for me (every codebase is different, so I am not doubting posted article). The differences between POSIX and WinAPI, GNU-isms in the code (Windows version was compiled by Visual C++ 2002 at the time), and support differences between C++ STL version were bigger problems.
My approach was to choose a C library specifically designed to support multiple platforms - Apache Portable Runtime (I also considered QT, NSPR, and GTK at the time). This helped tremendously and I would recommend doing the same to anyone doing similar effort or - better yet - start using something like that at the design stage.
We're probably going to still only offer "official" support for Ubuntu, since that is what we build/test on, but we'll try to fix what we can when problems are supported on other distros.
Costs have dropped with C++11, Unity and Steam for Linux. Sales have risen since Linux' market share is rising (very very slowly). Also of note is that Linux users seem to have beefier setups than their Windows peers, according to Steam Survey (12GB RAM vs 8, 2GB VRAM vs 1, bigger screens and more cores). More money spent on hardware correlates with more money being spent on games.
The editor is, however, largely written in Unity (uses the same APIs, etc, in addition to a bunch of private ones).
To explain my argument, I'll try to elaborate a bit more on the points raised by the author:
1. Case sensitivity in paths.
So, as others mentioned too: on a case insensitive filesystem like Windows, you'd have no chance to verify (i.e. test) "case-sensitive-correctness" anyway. So, you'd fail on this here and there anyway. Ok, maybe using the OSX FS with case-sensitive option would be a guard. But, OTOH, I believe OSX has it case-insensitive by default anyway, no? So, by testing in case-sensitive, you're actually testing the "least used path", which sounds not so smart, actually... especially given that switching from case-sensitive to case-insenstive (i.e. the other way) has its own surprises, then...
2. No OS #ifdefs
Um;... so... like... what else to use instead?...
3. "Assumptions":
3.1. Compilers.
This point seems actually to be more "no C++11". OK; yeah; sure; but remember: then you'll get maimed for being backwards, "corporate-ish", you'll get no "smart young hackers", people will get more frustrated...
Not that you'd really have a good alternative to C++ (Java/Python need huge and messy runtime env; and if you need performance of C++, you need performance of C++; if you started now, you could maybe at least try using Go as a kinda compromise, but depends on your actual needs).
You know, whichever way you choose, you have pros and cons. And, supposedly, some time ago when the decision was made, the pros were actually outweighing the cons; so, it wouldn't sound so smart to me to choose otherwise...
3.2. & 3.3. "Assumptions about [GUI]"
Sorry to say, but: GUIs are just not portable. Every GUI environment has different "idioms", "interface guidelines", etc. Windows has different. OSX has different. Linux has different. Heck, different versions of the OSes have them different! On Linux, even in the same "OS version" (um, distro? kernel?) you have multiple "desktop environments" with different widgets, layouts, etc (KDE, Gnome, ...). Conversely, on Windows, Microsoft likes to sometimes pull you in multiple directions at the same time too, e.g. when new MS Office introduces new widgets, or new Explorer does the same, or you have Metro+classic, or whatever. Dunno about OSX; I dare suppose it's not all roses there either.
I seem to believe now, that if you want to have a cross-platform app with a GUI, you should really write the GUI for each platform from scratch (only trying to reuse some parts). Or, consciously break with staying consistent with any OS interface guidelines.
As to "copy & paste": on Linux it has actually totally different underlying mechanism(s) than on Windows. So, you probably have to rewrite even low level parts of it very differently. Not just menu entries.
So - I don't really see what's there to complain on. The app was written for some 2 OSes; now guys port it to a third one; awesome! just, sorry, you know, there's gonna be some work needed in order to complete that. <shrug>
The author is a lady.
You misread. The problem was that it was written in the form of if Windows do this, else do something that only works on OS X. The alternative is an explicit if for OS X, so that when you have a third OS to support you don't have to add that.
#else #error #endif
If you work this way, you immediately know where to change your code.
I'll clarify a few things:
1. Unity was originally written for Mac OS X, then later ported to Windows (not the other way around). Given that it's more than possible to have a case-sensitive HFS+ filesystems (since, I think OS X Panther released in 2003?), IMO it would have been more future proof for us to be sure to develop on a case-sensitive filesystem. Unity users over the years have tried to run Unity on a case-sensitive HFS+ filesystem and failed, so it would have been nice for all of those guys. :-)
2. Note that the point in the article was not to avoid using OS-specific defines (that's impossible), it was to make sure to do something sane (like #error) in the #else case that would make it immediately clear something need to be implemented there.
3.1. Yes, the higher-level point is indeed, "Be smart/cautious about what language features to use for the sake of portability".
3.2 & 3.3. Of course it's just a matter of opinion, but I wish that we'd considered a more platform-agnostic approach to GUI handling when the number of platforms that Unity was ported to went from 1 to 2. I (and others) are still a fan of the menu-system-written-entirely-in-Unity-GUI idea, but others (within Unity), for very good reasons, disagree. Fair enough.
Anyway, thanks for sharing your thoughts; when I write about anything I'm always looking forward to hearing a different perspective, and one of the main goals I had in writing the post was to promote constructive dialog about what it means to write "portable" code. :-)
(The whole thing only existed in the first place because they are stuck on a completely outdated Mono version, due to Xamarin changing the license of newer Mono versions trying to extract money from Unity.)
Right now, the GC performance of Unity is horrible relative to what you'd get on the JVM and there's no reason to believe IL2CPP is going to be any better.
When you don't have a generational collector, you have to seriously constrain your allocations, crimping your immutable style.
The Unity roadmap suggests they're going ahead with a Mono upgrade (I believe they dynamically link every platform sans iOS):
http://unity3d.com/unity/roadmap/
So it's probably not on the cards (no pun intended). However, the Mono upgrade will bring a generational GC to the platforms on which it's usable (which will fall somewhere on the spectrum of Editor, PC, !iOS).
The only problem I have with Mono today is Miguel's decision to completely disrespect and disregard the Linux Desktop: http://tirania.org/blog/archive/2012/Aug-29.html
Mono changed the license so that it could be used in Unity to begin with. Mono was available under a copyleft license. To everyone. Including Unity. Unity couldn't/wouldn't accept it under those terms, so folks at now-Xamarin said, "We'll offer it to you under a commercial license if you won't take the copyleft offering." That's standard fare. Unity said, "Okay, deal." Mono continued to improve over the years. Unity was interested in those improvements. Mono said, "Okay. You can get those under a commercial license, too." Unity balked.
So what we're really talking about is that Unity wants the improvements that have been made to Mono, but doesn't want to pay for them.
Get this: They're still free at any point to accept Mono under the copyleft license that's available to literally everyone, but they won't accept it under any terms other than their own. That's it.
Do you know the specific licensing terms that Xamarin offered to Unity? Probably none of us here do. Given that, why would you assume Unity was the unreasonable party, and not Xamarin?
Regarding the copyleft license, while it is an option for everyone in general, it isn't an option for Unity, which needs to ship on platforms that do not allow such licenses (iOS, game consoles). Products like Unity will always need a commercial license as long as such platforms exist.
I don't know what this means. What we know of the situation is that Unity doesn't want to pay a commercial license for Mono. This is even the way that pro-Unity/anti-Mono folks frame the situation. What do you have in mind here?
> Regarding the copyleft license, while it is an option for everyone in general, it isn't an option for Unity
That's all well and good, but it doesn't have much to with any obligations Mono has. Those are completely synthetic restrictions that the platform vendors are placing on the people targeting them. How does Mono get the blame for Apple and Nintendo's choices here? Why do Apple and Nintendo get treated as bedrock? Even so, that's bedrock that Unity knows they're working with. It's their choice to target those platforms.
EDIT:
> That seems like a very unfair description of the situation.
Unfair. Every reference to the Unity/Mono relationhip I've ever come across has been from someone who heard Unity's creative retelling of the conflict and parroted it exactly, down to overtly identifying Mono as the bad guy.
We just don't know the answer, and you're assuming the party offering X and Y is reasonable. But it might not be. We don't know.
edit to your edit: if you heard people on the internet being wrong in some way, that doesn't justify you being wrong in the opposite way.
Companies A and B couldn't come to agreement to license something under terms Y that include $X in payment. Okay. There's neutrality in that statement. My comment says that (and no more).
What let's not do is say, "That makes company B a bunch of jerks." That is what the comment I'm responding to does, and it exists in a sea of others with the same tack.
Overton window and that.
> So what we're really talking about is that Unity wants the improvements that have been made to Mono, but doesn't want to pay for them.
which paints Unity as the bad guy, when for all we know, it might not be. Perhaps Unity is totally OK with paying for a license, but the Xamarin terms would have bankrupted it.
But thank you for clarifying what you meant. I agree there is no point in saying either party is the bad guy, since there is not enough public information to know.
In what way at all? Entity A doesn't want to pay entity B $X. That's the only thing the comment of mine that you quote says.
There are shops around that sell quadcopters for hundreds of dollars. I want a quadcopter. I don't want to pay hundreds of dollars. Have I just painted myself as a bad guy?
I repeat: the only reason my comment sounds pro-Mono is because, at neutral, it resists the spin of the one that paints Unity as a victim and Mono as a fiend.
In particular, it sounds like Y is offering Z at a standard rate, as in "Sam wants a Chipotle burrito, but doesn't want to pay Chipotle for the burrito." Sam doesn't look good here - does Sam expect to get the burrito for free? Why does Sam think he is entitled to that?
Readers may not realize that there isn't a standard, objective rate for commercial licenses for Mono. Especially for the single biggest consumer of Mono, Unity, which uses it in products that its customers ship - a particularly complex situation.
Again, maybe you didn't intend to, but it sounded like you were saying something very negative about Unity. A less biased way to say it might be
"Unity and Mono could not arrive at mutually acceptable terms for Unity to license newer versions of Mono."
What we also don't know are the terms for the first negotiation. It's as possible that Mono made one offer and Unity countered with an insultingly lower one, then Mono accepted due to being a poor position to negotiate something "reasonable", but are no longer in a position to have to accept whatever is offered, now that Xamarin is off and running.
We don't know. Hypotheses non fingo.
AOT-compiled code can also be very fast. I'm aware .NET has options there too, but il2cpp is literally focused on doing just that, and doing it very well.
The same people who couldn't maintain their fork of Unity are now writing their own home-grown runtime, complete with their own garbage collector.
I don't see a chance of this working out "well" for any sane or bizarre notion of "well".
The same people who couldn't maintain their fork of Unity are now writing their own home-grown runtime, complete with their own garbage collector.
I don't see a chance of this working out "well" for any sane or bizarre notion of "well".