Native Clojure with GraalVM
janstepien.com
janstepien.com
So if you're using Clojure and Leiningen and are annoyed by the long startup times... there's potential :)
One main problem I also see is the distribution. Currently leiningen only needs "some sort of java executable" and then the rest is pulled in. So would you have to have GraalVM installed or could this also be bootstrapped.
Performance can vary quite a lot from task to task, also compile times suffer a lot and you lose some dynamic features.
So from what I understand it might be useful for clojure cli apps, but at this point I am not sure it brings a lot, but this is early days.
The cool part is the polyglot capabilities imho.
Then there's the elephant in the room: the licensing and potential changes in the long run.
For clojure if you want decent cli/startup experience there's always cljs, lumo or you can go with something like fennel-lang, a clojure looking lisp that "transpiles" to lua(jit), so 0 overhead compared to lua and super fast startup.
For general CLI tools, yes, I'd also look into cljs at this time.
With tools.deps and the need of some companies for more advanced build setup a lot of people are actually thinking about these issues.
But I'd be surprised indeed, I'm not watching the whole clojure space closely, I can only say that no one hit the #leiningen irc channel or posted something on the bugtracker/proposed a PR yet.
And the client/server model.. sure, nailgun and grenchman were a bit clunky, but none of these efforts have gained widespread adoption, afaik. Most people don't just go out of their way to tweak their build tool.
I was not thinking about these two, nor was I thinking about a patch for boot/lein.
That said the slow startup issue is only really valid for cli apps (and maybe CI), if your dev workflow relies on re-starting your repl all the time, it's a broken workflow.
Developing in a repl is nice, but it's not the only allowed or legitimate or even good way to do development in Lisp or other dynamic, interactive languages. Clojure has an issue with slow startup (another tradeoff made for good reasons that don't fit everyone's situation)--denigrating other people's development approaches doesn't solve that, it comes off as defensive.
Me neither, and I also carry decent Lisp cred.
I know what I'm doing to the extent that I can write a bunch of code in a file, and have it work, more or less.
If I experiment with changes to code, I want to see the diff (as in git diff).
Actually developing in a REPL seems very scatter-brained; you don't know what you're running since you've been mutating things left and right.
Those that develop in a REPL actually develop in some kind of text editor which sends expressions to the REPL; that's what allows them to save the code properly to a file. Problem is, you now have three versions of the code to keep straight: editor, file and image.
Okay, so you have massaged things into working. Well, which is the code that is working? Obviously, the code that is in the image. But of the textual codes does it correspond to? If you just save the content of every edit buffer to disk, is the code on disk exactly that code that is working in the image?
Even if I have to work with a running image (say there is a lot of state that is difficult to reproduce from a clean start), I'm still going to work with some files, which I will edit and completely save to disk, and load all together as a unit into the running image, instead of evaluating individual expressions out of an edit buffer.
edit: a bit more on zprint & GraalVM and a comparison to the equivalent code on node:
graalVM native startup time: 0.055s
node startup time: 0.260s
https://github.com/kkinnear/zprint/blob/master/doc/graalvm.m...
That's actually really bad in Graal. I compiled a few classes project in Java, JS and Ruby. Compilation time was measured in minutes and memory usage most of the time was >8GB.
SubstrateVM was designed more to make Graal/Truffle based interpreters like TruffleRuby and FastR have fast startup time with small binaries, which is a large and complex project, more than it was meant to compile small, random one-off Java/JVM applications for CLI usage. From what I remember in my few random experiments the compile time isn't quite linear in size at least (e.g. compiling a 10 line Java class file takes 2 minutes, but compiling a 100 line one takes only marginally longer, comparatively.)
(Using SubstrateVM also means that there is no JIT for the JVM class files you compile to native code, so you're dealing with a purely ahead-of-time compilation scheme. Graal's online JIT still kicks in for SubstrateVM-based interpreters like TruffleRuby, however. This is an important distinction to understand.)
Hopefully they can improve the compile time requirements a bit for cases like this, since it's surely more popular and common case that people are looking for, as opposed to writing Truffle-based interpreters.
> So you have no excuses not to build your CLI tools in Clojure now.
Well I do disagree with this part, see other sibling comments (not by me). I'm not even sure GraalVM is really production-ready, notwithstanding the potential licensing and long-term issues.
I do agree it's at least worth trying.
The big minefield is that you do not really know which libraries will go boom and where, as any place could make use of class loading.
https://ioavisopretty.readthedocs.io/en/v0.1.34/exceptions.h...
Aren't all those crticisms double for Ruby? And Node?
You're adding a runtime, a package manager, and packages to your system with all three but two of them are just raw source code ...
They are severely misinformed, then, and it would be a good deed to give them an opportunity to learn more about this.
Which is: Java is fast. Its startup time is impressive, and the runtime performance is generally also good. RAM usage could be less impressive, but I read that it can be optimized for memory consumption, achieving pretty good results - I don't have any experience here, so I can't really say.
Anyway, my problem with Java, is that it is not designed to be used on its own. To use it, you need a build system, and the language doesn't give you one; which is why there are many different systems, which you need to navigate somehow. Having to learn these build system (XML? custom Groovy implementation?) is what made me dislike Java. It's not that different in C++ or C#, but it adds to the feeling that you work with an ancient technology, not a modern one.
That's however not the problem with Clojure - it at least has a single build system + package manager - Leiningen. To me, what disqualifies it, is the startup time of its REPL. Well, that's only because I have a luxury of not needing to do anything on the JVM - so I'm speaking from a position of a hobbyist polyglot programmer considering new languages to learn. I would probably reconsider if I had a job where JVM would be a must, but the specific language was not yet chosen (how probable that would be is another matter). Anyway, back on topic: I don't have Clojure on this machine, but last time I tried it, it took more than a second before the prompt appeared. That is slow even when considered on its own, but it borders ludicrous (speed) when compared with other Lisps, which seems to be the right frame of reference here. In that comparison, it's... Well, just to let you know how bad it is, Common Lisp and Chicken Scheme, two Lisps I do have on this machine, have startup (+ shutdown!) times like this:
-▶ time sbcl --eval "(exit)"
This is SBCL 1.4.6-1.fc28, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.
SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses. See the CREDITS and COPYING files in the
distribution for more information.
sbcl --eval "(exit)" 0,00s user 0,00s system 94% cpu 0,006 total
-▶ time csi -e "(exit)"
csi -e "(exit)" 0,00s user 0,00s system 93% cpu 0,005 total
1) I get it that "you're supposed to have a deamon process in the background, so you don't need to start that often", but this, in itself, is basically announcing surrender. In Common Lisp you're supposed to do the same, but not by necessity!
2) Yes, I tried alternative runtimes/implementations. Indeed, they are better. However, until they get, provably, 99% the same semantics as the reference implementation, they won't be adopted by any significant amount of people, which has enough of the drawbacks to make me not seriously consider using them. Anecdotally, last time I looked, I've seen lots of lein plugins for setting up projects with ClojureScript on the frontend and some (pure) Clojure Web framework on the backend. There has to be a reason behind that.
TLDR: Java is very fast, Clojure uses JVM in ways which are inherently slow on that VM, but refuses to honestly migrate to a more suitable platform, because of Java ecosystem, which is also a major selling point of the language.
At least, that's my understanding of the situation - if I'm wrong, I'd be happy to get corrected.
> TLDR: Java is very fast, Clojure uses JVM in ways which are inherently slow on that VM.
JVM is really not the best platform for hosting either dynamic languages or functional ones.
Heres the time taken by csi, clojure , racket , sbcl. >> time csi -e "(exit)"
real 0m0.088s user 0m0.020s sys 0m0.027s
>> time clj -e "(System/exit 0)"
real 0m1.944s user 0m3.272s sys 0m0.253s
>> time racket -e "(exit)"
real 0m1.100s user 0m0.503s sys 0m0.296s
>> time sbcl --eval "(exit)" real 0m0.060s user 0m0.023s sys 0m0.025s
>> time newlisp -e "(exit)"
real 0m0.089s user 0m0.011s sys 0m0.033s
Clojure is the slowest in this.
I wrote a small utility program in Java recently. It was getting slower and slower as it grew, so I profiled it. It turned out the slow parts were every part - the first time that part executes. E.g. parsing the command line arguments was slow the first time, but fast if I did it a second time. Same for reading configuration files, opening sockets, etc. I assume this has to do with a combination of lazy class loading (which is slow) and the JIT compiler not being warmed up.
Unfortunately for Java (and its users), you rarely need to parse your arguments a second time, so you never get to experience the case where Java is fast. It needs to be fast the first time, and it's not.
This sounds a bit out of date, as the vast majority of builds these days are Gradle, and those that aren't are probably 99% Maven.
However your point stands in that in both cases the build technologies are fairly arcane - few people who use them understand them well and they do form significant pain points whenever you need to go beyond the simple uses.
I know a senior dev who loves Ruby and Elixir and won't consider touching anything in the Java ecosystem ever again after a terrible experience at a government consulting company.
So yeah. I think the best shot I have at convincing him to try it is to point him to something like Clojerl: https://github.com/clojerl/clojerl
If a customer needs to have something in some tech I dislike, then that is what I use, regardless of my opinion.
Whatever makes the customer happy comes first.
I think the reality is there are a lot of people for whom hatred of anything Java is a fashion statement, a strange geek form of virtue signalling. Oracle is not significantly different to Microsoft or Apple after all - they are all companies that like money and have sued Android related companies over IP rights to obtain lots more of it. If anything Oracle's moral case is a lot stronger as Java represents a much greater investment than FAT32 or rounded rectangles or the other stuff these firms demanded ridiculous sums for. Yet hackers are just A-OK with praising and promoting VS Code, macOS, Swift, etc.
I don't personally mind. The more people who ignore the Java ecosystem for fashion related reasons, the more my competitors reduce their strength and relevance. If your senior dev friend is off writing code in Elixir and struggling with low performance VMs and tiny library ecosystems, I'll be outpacing them with better VMs, better tools and a huge ecosystem of open source libraries. No problem!
It's interesting how in the software world people whom you'd expect to be entirely rational make decisions based on feelings and hearsay. I don't understand this.
I say "in the software world", because I see more objectivity in hardware design. Hardware engineers rarely black out entire technologies just because they don't feel like it.
I don't actually see much value in having Clojure on LLVM as you'd have to build a whole ecosystem from scratch instead of being able to rely on the mature existing ecosystem that the JVM provides.
Why would you require an existing C lib? These days you have lots of JVM apps doing heavy lifting: from bigdata stuff (hadoop, presto, kafka, others) to webapp stuff (undertow, check techempower benchmarks). For those special cases you can wrap a C lib with FFI, but in my 10+ years of JVM experience I only needed to do it once, and it was because I was streaming video on a jnlp (java swing/java web start) app which worked remarkably well, considering I supported windows,linux and mac.
I wonder how AOT can deal with this scenario. Is it possible to AOT but still have the fallback of classloading at runtime?
Maybe they could add a whitelist of "known" entry-points for class loading, as to avoid going through a list of 10,000 inner classes that existed at compile time but nobody will ever instance by name.
uh ? no, new classes are added at runtime all the time - at least in C++, dynamic loading of plug-ins is extremely common
but you don't do this. you register the classes according to what is on your file system and you load one that matches your criterions or you ask your user to choose one in a UI of some kind.
What people are talking about is different. It's when the app genuinely loads plugins that weren't available at compile time, or generates bytecode on the fly.
no, but you can add, dynamically at run time (such as from a user supplied file) a new implementation of the interface. And because as part of the JDBC interface, a connector is able to register itself too (via classloading magic).
JDBC is just an example of a common plugin architecture for extensible software. But perhaps one way to make this work with AOT is to bring along the AOT compiler, and AOT the connector jar when it is being loaded, and treat that binary same as any dynamically linked library!
I'm not familiar with some of the dev patterns here, but why not just load all the classes for the databases you want to support, from the top, and compile them all in? Is that not practical?
A ton of stuff ends in up doing class loading, bytecode generation, dynamic serialization.
The easiest off the top of my head is webapp containers like Tomcat or Jetty. As it runs your load a whole zip file with a full set of jars on the fly afterwards.
you cannot distribute some connectors (e.g., the oracle connector or the mysql connector), in the case of jdbc anyway.
And it's not always possible to know ahead of time the classes you want to pre-load, if your application is like eclipse with a plugin architecture which can allow users to install custom plugins.
The SubstrateVM flavour can't do dynamic classloading or go as fast as HotSpot, it's strictly less powerful and compatible. However the static points-to analysis it does along with other optimisations mean much smaller binaries and memory footprints. If those matter to you and you can sacrifice classloading, great. If you can't, experiment with HotSpot's AOT support (which also uses Graal under the covers) and you can get faster startup time, but without the memory savings.
For some purposes this matters and for some others it doesn't. I can certainly see the benefit of quick startup for e.g. CLI tools. And for servers, the security benefit of running a single standalone binary blob inside a multistage Docker container might be easily worth it.
tl;dr: surprisingly Clojure code benefits a lot from aot compilation
I wonder what the performance impact will be for Clojure code.
https://github.com/oracle/graal/issues/447#issuecomment-3995...
Not everything needs to be free beer.
As such, native binaries will start faster, but run slower.
https://github.com/oracle/graal/issues/447#issuecomment-3995...
NGEN, MDIL for WP 8.x, .NET Native for UWP, Xamarin AOT, Unity IL2CPP, and the internal compilers used at Singularity and Midori.
To be owned by Larry Ellison.
I'm not advocating to make GraalVM a mandatory dependency, just saying this could be an interesting project for people who are annoyed at the status quo, and there are a lot of startup speed complaints.
Unfortunately my day job has nothing to do with any JVM or Clojure anymore, so I can't divert any time there.
That said there's a lot of FUD around Oracle ownership nowadays but most of it is moot since OpenJDK is the fundation of Oracle's Java.
The fact you can download and build it doesn't mean much in general tho.
> its remaining technical limitations will eventually be adressed
It's not clear there are any technical limitations that separate CE and EE. Supposedly EE is faster but I don't think anybody has proven this with real benchmarks. All of this is clearly documented in https://www.graalvm.org/docs/faq/.
https://github.com/oracle/graal/issues/447#issuecomment-3995...
GitLab and RedHat have a quite different history with litigation than Oracle does.
In the end most open source software I use is ultimately funded by better paid-for versions that people opt in to. The sky doesn't fall.
I feel people easily forget or act as if past actions/history isn't a metric to judge a company by. You can't predict the future with your feeling you can only do it via past data/actions. That or is it tribalism dogma that their bread and butter tech is being harshly criticize?
The only company besides Oracle that cared about Sun assets was IBM, and I very much doubt they would not have done the same thing to Google. After all, Sun would have liked to do it if the bank account had any money left.
https://www.youtube.com/watch?v=ZYw3X4RZv6Y&feature=youtu.be...
It was Sun who refused to give the TCK to Apache Harmony.
It was Sun that had different licenses and price categories depending on which kind of Java and version one would be using in production.
Some people easily forget or act as if past actions/history aren't a metric, while others forget who actually did them in first place.
How did that have any effect on Sun? Their mobile strategy was dead with the death of Symbian
Gosling refers to it.
Ultimately the dev kit for android is a clean room implementation (submitters made legal commitments to that fact). The question pivots on how you feel abut the IP of the java language and the library interfaces being used by someone else.
A scenario that is only going to get worse as Java keeps improving.
https://blogs.apache.org/foundation/entry/the_asf_s_position...
> Unfortunately, Sun breached the JSPA in 2006 by licensing the Java SE Compatibility Kit under terms inconsistent with its prior representations to ASF and its obligations under the JSPA, and incompatible with ASF's development of Apache Harmony. ASF urged Sun to honor its agreements, but after Sun persisted in its breach for a year, ASF withdrew from the JCP. At the time, Oracle supported ASF's position that Sun was in breach of the JSPA. But after acquiring Sun, Oracle adopted Sun's policy, disregarding the limits of the JSPA that formed the basis for ASF's participation in the JSP and acceptance of the various TCK licenses.
> Some people easily forget or act as if past actions/history aren't a metric, while others forget who actually did them in first place.
Choosing a video to support half a truth or a part of timeline doesn't mean anything.
Here's another source:
https://arstechnica.com/information-technology/2010/12/apach...
> The heart of the issue is that Apache can't certify that its open source Java implementation—called Harmony—conforms with the Java language standards because Oracle refuses to supply the necessary test suites under a suitably open license. Oracle's position on the issue falls afoul of JCP policies, which stipulate that standards and other relevant materials must be freely redistributable and made available under terms that are conducive to enabling third-party open source implementations.
You're deflecting any and all of Oracle's action toward Google should have bought it and well Sun did it first. That's a terrible argument to counter my point of Oracle past actions being terrible toward open source. The "google should have bought Java" is such a hand wave especially when Google did a clean room implementation of Java and involve the issue of if API is copyrightable (https://en.wikipedia.org/wiki/Oracle_America,_Inc._v._Google...).
They just broke Java the same way as Microsoft tried first, but since its the "Do no evil" company it gets a free pass.
Had it been other company and the Google support team would have another opinion.
I care about companies that care about Java, not those that fork the eco-system.
There are several companies playing by the rules, selling JVM implementations since the Sun days, none of them ever had issues either with Sun or Oracle.
But I failed to see how your personal opinion on fork on java and ecosystem have to do with my original post on Oracle's FUD and past action against open source.
If none of your cases you've bought in this argument/debate is toward this topic then I think you're pushing a narrative and being dogmatic toward Java. I have no beef against the language but I am wary of Oracle. I'm merely refuting/adding toward the comment of the post I'm replying to.