The Deletion of gcj
tromey.com
tromey.com
Bootstrapping becomes harder and more legacy software needs to be accumulated in order to actually build new software from source.
(I'm working on GNU Guix, a functional package manager, and I wrote the package definitions for the JDKs.)
At first, very few assumptions was made about the Unix bootstrapping process, and the mantra was that you should build the managed components on a Windows machine which already had .NET.
Just to get something going, the Unix community decided to use mono to bootstrap the process on non-Windows platforms. This effectively meant you could bootstrap .NET Core on almost any platform where mono was around. That gave quite some platform reach out of the box, and on overall an improvement much welcomed by Microsoft.
But later, after Unixes like Linux and OSX become better supported, the bootstrapping method changed to downloading a previous build of .NET Core for the current platform and using that to build the latest sources. I.e. going self-bootstrapping, which sounds fancy on paper.
The cost of that? Suddenly it just wasn't doable (without considerable knowledge and effort) to bootstrap .NET Core from source any more on platforms not currently considered first class.
I'm not just talking about "strange" outlier OSes like FreeBSD, NetBSD or Alpine Linux or whatever. Just trying to build for Fedora 24 (which doesn't have published packages) fails. This is because the Fedora 23 packages are linked against library-versions which are no longer in Fedora 24.
You'd think that be easy to fix, but now, even with Fedora 25 betas starting to ship, there's still no official Fedora 24 package exists from Microsoft, so Fedora users (and developers, and porters, etc) are SOL.
So now basically just trying to bootstrap for a previously supported mainstream Linux-distro is seriously PITA, next to impossible. In my honest opinion that's a major regression just for the sake of being self-bootstrapping.
So yeah. You should never underestimate the value of having secondary platforms available for bootstrapping your own. You may come to depend on it.
This is actually really sad to hear, because I've been waiting to see proper RPM's for the CoreCLR in Fedora for a long time, but having to constantly bring in new binaries to bootstrap the package is a huge headache that few package maintainers would be willing to take. Maybe provide the option to build a statically linked build that could be shoved into a different package and used to bootstrap a new dynamic version?
When I think of bootstrapping something like .NET, I would assume that most high-level stuff (i.e. large parts of the standard library, compiler and other tooling etc) are all 100% managed. So to run those, all you need is the VM and the part of stdlib that is implemented in native - the rest is just bytecode, and shouldn't have any OS dependencies, so you could easily redistribute and reuse the older build.
On the other hand, VM and native parts of stdlib, I would expect to be written in C or C++, and so, while that's where all those pesky dependencies are, that's also the code that you should be able to build with just as C/C++ compiler.
So the process would work like so: 1. Build VM. 2. Build minimal stdlib (native parts). 3. Use the preceding to run an older build of full stdlib and compiler, precompiled into bytecode. 4. Use the preceding to rebuild the compiler. 5. Use the preceding to rebuild the stdlib.
Which is also something that a fairly simple script should be able to tackle, such that the dev only needs to do "make bootstrap" or something similar.
So, where does this picture break?
It breaks at point 3. There's 2 distinct things which breaks here, much due to what you call poor componentization.
To build the stdlib, you need Nuget (which is written in .NET) to resolve binary dependencies and the Roslyn compiler (also written in .NET). These two components are wrapped up in a simple to use all-in-one command line utility, much like Clojure's leininginen: Basically one tool to rule them all. This tool is also written in .NET.
This means you need a prebuilt .NET version, including the base stdlib, to run the toolchain and compiler needed to build the stdlib itself. In theory this should not have to present any issues. Basically so far so good.
The problem is that what you need to download will contain 1. corerun/corehost runtimes used to bootstrap the process (platform native binaries) and 2. The managed stdlib, which also contains platform specific binaries (Win32 system-calls on Windows, glibc on Linux, another Libc on FreeBSD, etc). These are not all bundled in one big package where binaries for all platforms are included, but as separate packages per supported platform (Windows, OSX, Fedora 23, Ubuntu 14, Ubuntu 16, etc)
So on a previously unbootstrapped platform you can run into any of the following issues:
1. Native components are dynamically linked, and may fail to resolve libraries if they have had their soname/version bumped, even though you can build these components yourself just fine. This is obviously a structural issue with the build process, which IMO should be fairly doable to solve.
2. Because the compiler itself is .NET -based, it relies on the managed stdlib, and this this library itself must be built for your platform (FreeBSD's implementation differs slightly from Linux, etc).
So on a platform where nobody has ever done any work before, the download for step 2 will fail and the build will crash.
To rectify this you will basically need to get acquainted with dotnet's multitude of individual repos, how they tie together, in what order they should be built, how their build-outputs should be combined, and all kinds of un/under-documented stuff, and do that manually... Phew! ... To package a base "SDK" which can be used to initiate the normal coreclr build-process, which previously failed. You may need to fake this SDK for several dependent repos.
That single step is incredibly complex, and I think only a handful of people on the whole internet knows how this is done.
Compare this to how things were previously: The process was bootstrapped automatically using mono, and every single coreclr developer knew how to get things up and running.
That this was greenlit without any further objections is something I find hard to believe.
> The problem is that what you need to download will contain 1. corerun/corehost runtimes used to bootstrap the process (platform native binaries)
if they're native, why not just build them right there and then? Or is that where the dependency resolution tool (that is managed) creates a circular dependency?
The other problem, if I understand correctly, is that the managed code has a bunch of #ifdefs for various platforms. If, instead, it selected one codepath or another depending on the platform at runtime (e.g. for something like Process, which has to invoke radically different APIs depending on the platform, make it a very thin wrapper around Win32Process/UnixProces/..., and pick the appropriate factory in the static constructor), the same managed code bundle could be used everywhere. Except for a brand new platform, of course, but that is a problem that the Mono solution also has (if you have to target something that doesn't run Mono).
You typically do... But then the next build step says "DL this pre-made SDK kit" and doesn't overwrite it with what you've just built. So you will need to patch the build-process (in several repos) to overcome this.
> The other problem, if I understand correctly, is that the managed code has a bunch of #ifdefs for various platforms. If, instead, it selected one codepath or another depending on the platform at runtime (e.g. for something like Process, which has to invoke radically different APIs depending on the platform, make it a very thin wrapper around Win32Process/UnixProces/..., and pick the appropriate factory in the static constructor), the same managed code bundle could be used everywhere.
Correct. And so is your analysis.
But for several reasons that was not the solution which was chosen. Having an extra vtable or stackframe for every platform action invokable throughout the entire .NET framework was considered something to avoid as far as possible, with regard to performance and memory efficiency. And that's absolutely a valid concern to have, since it effectively will affect all .NET applications.
Remember: The goal here is for .NET Core to server as a base for .NET as delivered on the main, commercial Windows platform (including Azure!) with as little changes as possible. And in the cloud-space you don't want to impede your own performance.
Disclaimer: Things may have changed by now, but that at least was the state last time I poked into things.
This sounds like more of an issue with Fedora than anything else. If Fedora 24 can't run programs for Fedora 23 (not ancient programs that happened to work on Fedora 23, but programs that were specifically built for Fedora 23), that's pretty ridiculous.
In all honesty it's virtually never an issue, most libraries maintain API compatibility and everything gets a mass-rebuild before alphas start to ship (though everyone tries to notify affected maintainers when they are going to do a soname bump on the mailing list) - so as long as you don't depend on old libraries for bootstrap of new package versions it's fine. There's plenty of compilers in the repositories that bootstrap themselves every release without giving grief, so Microsoft needs to either provide an external bootstrap mechanism again or reduce the dependencies needed to get it bootstrapped to preferably nothing more than glibc.
As a general-purpose application framework/runtime, .NET probably has far broader dependencies than any one application. Maintaining binary compatibility for a dynamically-linked .NET framework is a tall order.
I've seen issues like this for significantly smaller packages on Ubuntu.
One example being exactimage, which for some reason weren't included in one single release around 14.04. Trying to copy a single binary across versions here, would cause similar breakage because of the versions of linked dependencies.
I think this is "symptomatic" of the Linux world and actually fairly common in software where source is assumed available and recompiling using that source a feasible task. Things like this is usually just not problem in the real world when things are properly packaged.
Again: for .NET the problem was not the dependency itself, but the root bootstrapping process.
There are pros/cons to both approaches.
For languages that have effectively abandoned that route, what would satisfy your requirements? Which languages specifically handle things in a way you consider correct? Is the effort involved worth it if only reproducibility is the end goal?
I don't think its safe to say that a bootstrapping compiler is the only sane way. Even if System F is a fairly simple language and what GHC compiles to, having to keep a bootstrapping compiler around just for bootstrapping purposes seems like mostly wasted effort IMO.
It's not perfect, since now you have a dependency on Docker (or equivalent), but maybe it's still an improvement?
There's also some efforts like [snap](http://snapcraft.io/) to make it easier to release binaries with all their dependencies packaged together, but I expect making it work for a complex project like .NET is difficult.
You can bake that into one binary in additional segments, but it can be fiddly and require more extensive code changes to make various fopen() calls read from the binary instead of a standalone file.
Having this path is important from a security and user freedom viewpoint: we could build OpenJDK from source.
Not having such a bootstrap path means relying on huge opaque binary blobs, which is a net loss.
I'm happy to talk either here, or my email's in my profile, if you prefer.
However, the main problem with GCJ was it used conservative garbage collection - boehm GC - and as such was fundamentally broken for long running programs (what java is good for). Without wanting to denigrate it too much, it seems to me that a lot of 'good' work has been put into this approach which was essentially a cul-de-sac.
I had a related rather general thought recently about documenting features in projects. Sometimes the hardest thing to know is whether something is in fact a good idea or not (both in theory and in practice) and the amount of effort/quality of documentation is probably not a good indicator of this. So in the case of the GCJ garbage collector, there was always a lot of talk about how clever the garbage collector was and all the technical aspects, but unless you were already an expert, how would you know if it was infact a good idea or not.
[1] That is this one, there are several now doing similar things (!). http://www.xwt.org/
The code lives on here https://sourceforge.net/projects/vexi/
Reading this post made me wish I had spent more time with 'gcj'. Though in all honesty I know I gave up on it because I didn't want to go the tougher way - the the normal JDK was just there. Creation is beautiful, and its ending bittersweet for the creator and the witnesses.
You can rebuild teams like that, find those projects, etc.. but it feels like an eternity when you've had it and then don't have it for a while.
One Java AOT option goes away, and a new one is arriving.
http://stackoverflow.com/questions/3032727/java-jre-vs-gcj
Did anyone used GCJ for something in production?
In the end, the full java spec doesn't lend itself to aot, because if you claim to be "real" java you have to include support for runtime classloading, which means there has to be a bytecode interpreter embedded in there somewhere. My thoughts are now, if you really want a java/c# style aot language, vala is the way forward.
What about D?
In .NET Native you have to explicitly inform the compiler of any kind of reflection behavior you might require.
No restrictions on reflection, unlike .NET Native (which has slightly different design goals).
I've used it many times for just this reason with great results. Plus, the runtime was re-licensed to MIT/X11.
https://gcc.gnu.org/onlinedocs/gcj/About-CNI.html#About-CNI
Could be used for scripting and extensibility purposes.
Is this something that could be implemented in the Java JDK / OpenJDK? Understandably a lot of work, but Oracle is looking at AOT now, for performance reasons.
The performance improvements could be taken even further by something like this, Java to native code compilation.
The best solution is probably to replace the C++ bits with Java, and go full JVM.