Scala Native v0.1 | Hacker News Reader

Scala Native v0.1

scala-lang.org

642 pointszepolud240 comments

I would love for there to be a similarly thorough project with Clojure. It really bothers me there's no good native compiler. Apart from anything else, it means that Clojure lives and dies by the languages it compiles to, and while Java is used everywhere still, it probably isn't the thing the kids are learning these days. Besides, without going into any further rational arguments for why using the JVM (or another VM) isn't always great, something about it feels a bit icky to me. On an aesthetic level.

I maintain a project called Ferret [1], which compiles a subset of Clojure to C++11. No VM no dependencies you get a single cpp file that can be compiled to native code. Supports any platform with a C++11 compiler. (atmega, arm, x86 etc.)

[1] http://ferret-lang.org/

Doesn't using the GPL for the stdlib also mean that all programs written in ferret must also be GPL?

Scala (and Java itself) are much easier to build native compilers for than Clojure, where numerous values are actually small dictionaries, nothing has a fixed type, and a large amount of the code is expanded from macros that are expecting to executing in the environment of deployment (not some special compilation stage). It is a language that essentially needs a heavy runtime to be remotely efficient (as anything super small is going to be rather equivalent to a barebones/naive JavaScript implementation), and it is unlikely that an attempt to build out a nice runtime for it is going to somehow be extremely better than the JVM (or a JavaScript engine, which might be even better in some ways).

Forgive my ignorance, but wouldn't one of the existing Schemes be a better or as good option?

I thought the benefit/raison d'etre to Clojure is the JVM ecosystem.

JVM is the benefit, but great immutable collections data structures and respective functions are also very very nice.

IME the biggest thing is having the same, much-better-than-JS language on front and back end.

> isn't the thing the kids are learning these days

Yep, they learn Ruby and then they need to go JRuby when performance becomes relevant. :)

Or they tend to just Go. Interestingly, I've read a lot of Go users come from the Ruby and Python communities.

What is interesting is leaving the expressiveness behind of a language like Ruby or Python, for a bit of AOT compilation to native code.

I bet if something like RubyMotion wasn't a commercial product, the reality would be quite different.

Crystal and Nim can be good alternatives, no type system sucks anyway

I agree, but they lack the Google stamp, which is one reason for Go's adoption.

As a Scala/Java dev, I'd be willing to explore Go not because it seems like a better language (I have no opinion), but because I'd hope there's less design pattern/enterprise-overengineering approach among devs. Even in Scala, which is super-flexible, people tend to structure code like they do in Java - everything is decomposed into tiny classes to the point of absurd. One of the reasons is that the mocking frameworks expect to mock classes, so every bit of functionality needs to be it's own separate class, just like in Java...

Funnily enough, I was reading a Scala program the other day and I noted how much it looked like a high-level dynamic Java programming. I think there was maybe 8 lines of typical functional programming code and the rest just looked like imperative code. It seems a waste to use Scala that way to me, but maybe I'm missing something?

Even if you're writing Java-like code you get a lighter syntax, fewer awkward special cases (primitives and arrays aren't as special, == does the right thing by default, try doesn't need braces and nor do function blocks, if is an expression), and some very useful code-saving constructs (case classes, pattern matching). It's not using Scala's full potential but it's still a good language for that space.

I view mocks as a code smell. If you have good interfaces and make a clear distinction between services and values you should never mock - stub your services, and use real values in tests.

If Go ever gets adopted by the enterprise for full stack applications, not only devops related stuff, expect enterprise architects to come up with GoEE.

SwiftEE looks far more probable as IBM has already jumped on Swift on server thing. The no-GUI and no-framework Go is not helping Java style solutions.

> The no-GUI and no-framework Go is not helping Java style solutions.

Just wait and it will come, it only needs a bit of enterprise architecture caretakers.

JEE might have its flaws, but it is way better than any C or C++ enterprise stack I have used before, definitely better than CORBA for example.

I have no problem with JavaEE and I have used it many times. Though JavaEE developers/architects like Java EE solutions generates 10-20 times scaffolding in meetings before getting to the point.

I've read a lot of Go users come from the Ruby and Python communities

Don't believe what you read, especially when it originates from a biased source.

When i want productivity, i use python.

When i want speed, i use c.

When i want both, i use both.

I'm more confident with static typing... also easier to refactor. Go's more productive for me after a cut-off. I still love using Python for quick scripts.

So it isn't the case many Go users come from the Ruby and Python communities?

It's the Never Java crowd who I just don't get. Java is almost always the most practical answer. And if you don't like Java, then use C#.

If you hate yourself sure, Java is one of the worst working environments for a tech stack I've ever worked in. I've worked in a ton of others.

I am yet to use a development environment that matches Java and .NET in terms of tooling, including libraries, IDEs, GUI toolkits, profilers , application servers, integration with legacy systems, network monitoring of clustered applications,...

Im one of those. Anytime Python / Django doesnt cut it I use Go. Works very well.

Have you got a few examples where you had to replace the former for the latter? Really interesting.

Sure. Last one was a static asset server that had to perform or the users would notice a delay. Built one in Go in about a week and put it behind nginx on a vps. It was faster than required, stand alone, and used very little memory. Its still running to this day.

why Ruby's performances are low? I've tried ruby but sintatically makes my eyes sad. and i left

Java is still taught in plenty of places and is entrenched enough to stick around for the foreseeable future. The JVM would still be fine even if Java the language declined. Clojure is is a much less vulnerable position than most languages in terms of VM risk, with Clojurescript and ClojureCLR.

The real benefit of native Clojure to me would be for reducing startup time and the JVM overhead in containers. The best option for this right now is CLJS, but it would be nice to just not have a runtime.

You always need a runtime, even if it is a native compiled one.

one of Hickey's initial goals for Clojure was to be hosted language by design because <reasons>.

still i agree with you, having native clojure would be amazing. there is pixie which is transpiled by rpython into c++ but it's pretty much abandoned.

There's also https://github.com/pixie-lang/pixie - a Clojure flavoured Lisp on top of the PyPy JIT foundation.

There's [cljs + react native](http://cljsrn.org/) which isn't a native compiler, but does compile to native code.

The kids are learning javascript, as it runs everywhere. Plus there's a clojurescript, so you should be safe. Furthermore, I wouldn't be scared on Java dying.

This will be huge for getting Scala running on AWS Lambda. The cold-start times for JVM apps is just ridiculous and makes Lambda/API gateway essentially unusable for anything written on the JVM.

Actually, there are people using Scala.js to run Scala on AWS lambda, precisely for that reason.

* https://github.com/tptodorov/aws-lambda-scalajs

* http://underscore.io/blog/posts/2016/03/21/serverless-scale-...

Sure, but it'd be nice not to have everything in the known universe depend on JavaScript.

I disagree. Having a common language in many places has led to huge improvements over the board in the JS ecosystem, it's become the "C" of the modern era, and you can build a lot of stuff on the foundations that others have laid.

I still hate JS though.

Which is anything but positive, as it also shares C's style of unsafety due to its semantics.

Javascript performs unchecked memory accesses? Or are you alluding to their both being weakly typed?

No,but implicit conversions, including from operators into numeric values isn't much better.

The amount of page differences between "JavaScript the good parts" and the actual language reference speaks for itself.

That book was created before many alternatives came out; now if you are working in a team project you should look at JavaScript the same way you look at assembler, you know it will be the final thing but you shouldn't be creating it directly, but instead "compiling" from TypeScript, ELM, etc. (langs with no implicit conversions, strong typing, etc)

Those alternatives are nice in startup land, in enterprise consulting land, we get to use what the customer IT department sanctions as allowed programming languages and tools installed on computers for external consultants.

One might argue the same point based on the fact that there even exists a book called "JavaScript: The Good Parts".

Which is ultimately irrelevant here. Consider, C is rock solid safe compared to machine/assembly.

(Though, I tend to share the annoyance of javascript...)

C is just as unsafe as Assembly, as the weekly CVE entries prove.

The only differences between C and a powerful macro assembler like MASM, is that C is portable across CPU architectures and exposes less internals.

Rock solid?

With all the UB that Assembly actually doesn't have, and the types of memory corruption issues shared with Assembly programming, not really.

My point is that safety or otherwise of the target language is just not relevant. Even C has compile errors if you use the wrong type for an operation. Assembly largely does not.

Taken to the extreme, any language ultimately goes down to machine code. So if it was a relevant fact, it would be relevant for all languages.

> Even C has compile errors if you use the wrong type for an operation. Assembly largely does not.

There are strong typed assemblers, e.g. IBM i.

Also Assemblers don't remove code under your feet, like optimizing C compilers with their UB tricks happen to do, introducing security bugs.

Absolutely! I'm just pointing out that this need is so strong that people have reached for the slow throughput of Scala.js just to get its fast startup time. If we can have both, that's even better!

No intention of arguing with Scala.js' creator :-) But, I do not think it has a "slow throughput" at all. In my experience, it's had pretty good performance.

Oh it does have pretty good performance, on par with hand-written JavaScript. But that's still 3x slower than Scala/JVM on our benchmarks, and those benchmarks are ported from the Octane benchmark suite used by V8 (so supposedly that's where V8 shines).

Everything is relative. Scala.js remains significantly slower than Scala/JVM in terms of throughput.

^ From the creator of Scala.js :)

I am excited that now there will be an option for getting both quick startup and fastest execution time!

Saw the same for clojure. People used clojurescript/plank for live dev.

> This will be huge for getting Scala running on AWS Lambda.

Do you mean it might convince Amazon to add Scala as a supported language? You don't get access to run native code on AWS, just Python, Java, C#, or JS.

And, even with a robust, mature native implementation, I'm not sure that Scala would be the top priority for the next language.

You can run native code on AWS Lambda. It just isn't officially supported. See Apex for an example of how to get it working..

https://github.com/apex/apex

Go on Lambda https://github.com/eawsy/aws-lambda-go Rust on Lambda http://julienblanchard.com/2015/rust-on-aws-lambda/

etc. etc.

I don't understand. How does the ability to generate native code help getting something running on AWS Lambda? By using asm.js?

You can achieve this by executing the native program in a child process. Apex is doing something similar to support golang in AWS Lambda, here is more info http://apex.run/#understanding-the-shim

It seems to me like Scala's biggest benefit and biggest downside are two sides of the same coin: easy interop with the JVM and Java code. Scala Native just seems like you're paying all the price of that for none of the benefit.

I think the Scala community's aims have been higher than what you've suggested for a while. For example, from what I've seen, Scala.js has been wildly successful, yet according to you it should never have existed.

Scala.js lets you run Scala code on javascript-based VM's and provides full integration with the underlying platform and libraries. Scala native looks to be another attempt to expand Scala's reach into new platforms beyond the JVM.

Define "wildly successful"... I can't name a single company or project that uses Scala.js.

There is a small list under "Built with Scala.js" here: https://www.scala-js.org/community/. And that's only for the public facing apps of companies who have explicitly sent a PR to the website to add themselves to the list. Other companies use Scala.js for internal tools.

that's hardly 'wildly successful'

Don't crap in Sid's Cheerios. For a project mostly made by one dude, Scala.js is amazing.

It's hardly "one dude", though. Don't give me too much credit. There were two others (@gzm0 and @nicolasstucki) who helped a lot on the core. But even more than that, the reason it's so popular and adopted is mostly due to the amazing community who wrote great, relevant libraries, and communicated to the world.

It's at least a dozen people who should be credited, even if I'm currently the only one paid for this work.

Perhaps it could mean successful in wilderness not in typical companies etc.

Even if you're not using Java code, it's still the best language going IMO.

What are the alternatives? I couldn't live without HKT these days (once you're used to thinking in them it's painful to work without them), which rules out most languages even in the ML space, and Scala has better IDE support than anything that's left (indeed an excellent tool ecosystem in general in terms of e.g. profilers, instrumentation) except possibly Ceylon (which doesn't have anything like the library/developer ecosystem of Scala). Even assuming Ocaml lands their modular implicits functionality doesn't change this as far as I can see.

Haskell is an option, but eager evaluation makes reasoning about performance much easier. (And I do think there are legitimate use cases for traditional OO inheritance, though this is debatable). If and when Idris reaches a similar level of library/tool support to Scala then it might become a better alternative to Scala Native, but not that many people are willing to use Idris in production yet.

PureScript may soon be a serious contender in the HKT language category. It is more or less a strict Haskell for JavaScript runtimes with some new type system features like row types and more fine-grained control over I/O (see http://www.purescript.org/learn/eff/ and https://github.com/purescript/purescript/wiki/Differences-fr...). It has an experimental native (via C++11) backend.

I started working for a company that does a lot of Scala a few months ago. It's my first real exposure to it.

To me, Scala's biggest benefits are just the language itself. So many little things that I couldn't do in Java, I can do with little effort in Scala. I can write imperative code if I really want, but I have all the benefits of functional programming available too.

Java/JWM interop was a good starting point to make corporate adoption easier and Scala was/is pretty successful with that strategy. However, in my opinion, Scala developers are not necessarily interested in Java and see it more as the necessary evil.

Well, this is just targeting a different VM, LLVM instead of the JVM. You get easy interop with all LLVM languages with this, including C.

You should really check your facts before posting a statement like this.

While the VM in LLVM historically was short for Virtual Machine, it really has nothing to do with that. It's a compiler backend used by Clang (C++ compiler) and Rust.

LLVM IR apparently is not restricted to C/C++/Objc/Swift etc, it is a generally purposed IR; and LLVM itself is an infrastructure that contains many facilities to deal with compilation backends (mostly, analyses and transformations).

My understanding is that IR is designed as though there were a VM to run it, but in practice, IR is immediately used to generate code for a target architecture.

That might have been true originally, but I don't think anyone uses LLVM like a JVM/CLR-esque VM any more. As the parent states, the original Low-Level Virtual Machine initialism was even retracted, meaning the project's name is just the "arbitrary" sequence of letters LLVM, with no particular meaning assigned to them.

LLVM is not a VM in the same sense as the JVM is.

Yup.

Easy interop with C++ or Rust? I'll believe it when I see it.

It looks like it can!

http://www.scala-native.org/en/latest/user/interop.html

>Scala Native provides an interop layer that makes it easy to interact with foreign native code. This includes C and other languages that can expose APIs via C ABI (e.g. C++, D, Rust etc.)

From that page, it looks like Scala-C interop is decent, but that's a far cry from C++/Rust interop. For C++ at least, you more or less need to write a pure-C wrapper API to call from Scala, since it doesn't handle C++ types.

Interesting, I guess I understood that wrong. Looks like no "easy" interop, but it's there if you really need it and don't mind the extra work.

I've been playing with this and trying to convert a ~20 line helper script that I use at work (and would really like to benefit from no JVM warmup time), and I've already run into missing core library functions like parallel collections and regexes.

This thing will be really great when it's ready, but it's not even close yet.

You would need the same for Rust as well.

Rust's FFI is the same as C.

I'm not sure what exactly you mean by that, but what I was trying to say is that you can't expose Rust directly, you need to expose a C ABI. Which is totally doable, but is not just "drop in Rust code and it works."

> languages that can expose APIs via C ABI

That's what I thought.. another C-based FFI. Better than nothing, I suppose.

Nim has pretty decent interop with C++. In general, though, you are right that C is much much easier to interop with

The biggest benefit should be a powerful and expressive language, the biggest downside would be incompatibility with lots of existing (java-oriented) Scala code.

That's a great news ! We are exclusively using scala at work for back end and I wonder if it could be interesting to switch new projects to scala native.

Did you test scala native against well known and massive open source scala project ? Did the performance improved or regress ? Did you wrote a brand new scala compiler for native code ?

Never touch a running system ;) Scala on JVM is much more tested than the new shiny thing. Also don't expect improved performance... many people think that the JVM is bloated and makes programs slower (this is mostly not true). The downsides of the JVM are more memory consumption/footprint (when you have e.g. small servers or micro instances) and the cold startup time of the JVM itself (which is not relevant on a server in comparison to desktop Java apps). Would be interested to hear if any backend Scala projects like e.g. Play work on Scala Native.

> the cold startup time of the JVM itself (which is not relevant on a server in comparison to desktop Java apps).

I disagree somewhat with this.

We found that when we started writing microservices in languages that are not java, the short startup time changed how we did some error handling.

For errors where we say lose connection to the database, or rabbitmq, we much rather have the nodejs-process die and restart, than try to construct reconnect logic.

The problem with reconnect-logic is that it is code that (may) be tested very rarely. This in turn means it's easy to get strange long term problems there like very slow memory leak due to some listener being added to a connection object once the connection is initiated.

We did a 180 on reconnect-logic in our nodejs-processes and let the exceptions just bubble unhandled and take the entire vm down. With automatic restart script, the process will be back in seconds anyway, and with docker having built in back-off timers for auto restart, we don't necessarily overload the shared resources.

> For errors where we say lose connection to the database, or rabbitmq, we much rather have the nodejs-process die and restart, than try to construct reconnect logic.

This sounds like a very Erlang-ish way to handle the problem. Another advantage is that if the server/process is in some weird state that's causing problems, killing and restarting it lets you clear out the broken state, and get back into the state that it's most likely been tested under.

Yes. We're almost now risking it going the other way, that some bad programming goes unchecked for a long time, because overall, the process sort of does what it should. Even if it restarts like 10 times a day.

Well, you do still want good reporting, so you know when failures are happening and can capture a stack trace.

I guess I don't see the difference here if your VM startup time is 3 seconds or 15-30 seconds. If that's the difference between the site remaining stable and the whole thing collapsing then it seems like you're setting yourself up for a big outage one day when the nodejs process isn't able to come back in three seconds for whatever reason.

I think it depends a bit on class of errors. Certainly not everything is suitable for this treatment.

Lost connectivity to RabbitMQ or Elasticsearch would mean our site is dead anyhow (you can't do anything). So either of those errors should arguably result in some static 500 pardon-our-appearance page.

But say someone messes up the network connection or we get a brief problem.

Why wouldn't the nodejs process start quickly?

The most effective way to handle these kind of errors in Java unfortunately requires understanding class loading, thread contexts, wrapping connection primitives in the right kind of references, and then making sure that all resource deallocation/closing always use the same codepath. Even though you really only need to implemnt it once, it is both somewhat tricky and technically challenging.

It's a pity that so few Java projects have tried to use these mechanisms without building them as part of massive frameworks, sometimes apparently even without understanding what they have built.

Yes, I did spend large parts of my java developer looking at class loaders and class loading delegations in servlet containers etc.

I think it's a bit too hard to get it right.

Like, suddenly some third party library starts pulling in log4j and your whole logging setup goes wrong in subtle yet very bad ways.

Or you screwed up with that one reference to a ResultSet and even though it is closed, that reference keeps an entire class tree of Connection, PreparedStatement etc alive.

Isn't this also solved by just load balancing so that the customer ends up reconnected to a healthy node while the downed node is replaced?

We run our Scala apps on Aurora/Mesos behind a load balancer (hundreds of instances for just one app). If there's an issue that can't be handled within the app and error rates breach a given threshold, Aurora just kills the instance and creates a new one on another host.

A JVM boots in about 100 milliseconds. A difference of 100 milliseconds made the difference in how you do error handling?

On what box with what kind of disk for how big of app? You aren't getting 100ms starts on a 199 mb fatjar on Amazon EBS.

At the moment, there isn't direct support for multithreading [1], so I'm guessing it would be very difficult to run any of the common web servers or computing frameworks natively. It may be possible for libraries that have pluggable concurrency, for example by creating an `ExecutionContext` that wraps OS threads, but that's waaay beyond my pay grade.

[1] http://www.scala-native.org/en/latest/user/lang.html#lang

Cheers - this is the #1 item on my excitement checklist.

Long time Java lover here. I agree with all your points, but in the context of Java at least (does Scala support this?) there is no simple static binary that can be built and released, which includes the JVM. I think 1.9 will have this option, but this is something I didn't realize I missed until I started work with Rust and Go. It makes deployment so much simpler.

In the age of containers it's really not that much harder to build and deploy a JVM app.

Edit: thanks for the downvotes but you could at least tell me what's so crazy about my statement.

One app, sure.

At work I'm running 17 different containers- many of which require their own JVM. (That's 3 different JRuby apps, zookeeper, kafka, and ElasticSearch.)

Those JVMs get heavy when you're shipping container images compared to small Go or Rust binaries.

To my mind the JVM is where containers make the least sense. If you build an executable jar you can run with "java -jar ..." then that seems just as simple as "docker run ..." and gets you the single-file deployment, and you can control memory allocation via flags if you need to. You don't get virtual networking but IME that doesn't add value in the first place.

There are still some licensing issues. For instance, Atlassian has an official docker container to evaluate Confluence, but they don't support it in production since it uses OpenJDK and Confluence is still somewhat broken on OpenJDK.

Rather than fix Confluence to work on OpenJDK (I don't want to imagine what type of reflection garbage they've got going on down there that breaks so bad on OpenJDK), their instructions tell you how to make your own Dockerfile using the official Oracle runtime.

Actually, in that situation, if it won't run on OracleJDK it's probably not going to work via a native compiler either.

No downvote from me. But a explanation why e.g. compiled binaries are better. I had a hard time to get a normal non fancy Scala Play project running on a 512MB DigitalOcean instance. Mostly because it needs a lot more ram for building. I solved it with using a bigger swap partition. With single binary precompiled programs this problem is more a developer machine problem than a infrastructure problem. So I think the deployment step itself (and not looking at anything else) is easier with a small single binaries.

You shouldn't build it on the deployment server. You build a jar and upload/download that to/from the place you want it to run, just as you'd do with an executable. A jar is "not binary" but what practical difference does that make?

yes, you're right, thanks for pointing that out.

Does it really need to be a binary? Build executable jars (use the maven shade plugin), run them with java -jar foo.jar, that's about as simple as it gets.

I have never successfully deployed a Java application with defaults on the GC, etc.

I'd love it if I could compile those options into a binary.

Yeah you can't do that, and I don't necessarily agree with that design decision. But for the sake of the comparison it's worth saying that these "simple" compile-to-binary languages simply don't let you set those parameters at all - it's ridiculous to argue that Go (say) is better than Java because something that's impossible in Go requires fiddling with parameters in Java.

I didn't actually say Go is better than Java. I said that binaries were something I realized I missed because of those languages. That is, it's something I appreciate about Go and Rust.

But what's the advantage of a binary over a (shaded) jar? "java -jar myapp.jar" is a little more typing than "myapp", but only a little (and you can avoid that by prepending a launch script if you want); having the JVM installed on all your servers is a one-time cost.

See my above comment about GC and other runtime options. I always need a script to specify all the options to the JVM. It is never just as easy as a single jar with no options. That makes it better, no doubt, but it still sucks.

I don't understand how you need those other options in Java and avoid needing them with a binary? What's the difference that means you can get by with not passing any options in go or what have you?

(I've used "java -jar myapp.jar" in production and it's been fine; the Java mainstream may favour using lots of -Dblah but it's entirely possible to replace that with code)

Maybe some apps do not have classpath. Here is what I see of running kafka instance on one of my server. And it does not looks like as simple as it gets.

java -Xmx512M -Xms512M -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/opt/kafka_2.11-0.10.0.0/bin/../logs/zookeeper-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/opt/kafka_2.11-0.10.0.0/bin/../logs -Dlog4j.configuration=file:bin/../config/log4j.properties -cp :/opt/kafka_2.11-0.10.0.0/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/argparse4j-0.5.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/connect-api-0.10.0.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/connect-file-0.10.0.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/connect-json-0.10.0.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/connect-runtime-0.10.0.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/guava-18.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/hk2-api-2.4.0-b34.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/hk2-locator-2.4.0-b34.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/hk2-utils-2.4.0-b34.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jackson-annotations-2.6.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jackson-core-2.6.3.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jackson-databind-2.6.3.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/javassist-3.18.2-GA.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/javax.annotation-api-1.2.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/javax.inject-1.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/javax.inject-2.4.0-b34.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/javax.ws.rs-api-2.0.1.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jersey-client-2.22.2.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jersey-common-2.22.2.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jersey-container-servlet-2.22.2.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jersey-guava-2.22.2.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jersey-media-jaxb-2.22.2.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jersey-server-2.22.2.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jetty-http-9.2.15.v20160210.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jetty-io-9.2.15.v20160210.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jetty-security-9.2.15.v20160210.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jetty-server-9.2.15.v20160210.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jetty-util-9.2.15.v20160210.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/jopt-simple-4.9.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/kafka_2.11-0.10.0.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/kafka_2.11-0.10.0.0-sources.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/kafka_2.11-0.10.0.0-test-sources.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/kafka-clients-0.10.0.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/kafka-log4j-appender-0.10.0.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/kafka-streams-0.10.0.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/kafka-streams-examples-0.10.0.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/kafka-tools-0.10.0.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/log4j-1.2.17.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/lz4-1.3.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/osgi-resource-locator-1.0.1.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/reflections-0.9.10.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/rocksdbjni-4.4.1.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/scala-library-2.11.8.jar:/opt/kafka_2.11-0.10.0.0/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/opt/kafka_2.11-0.10.0.0/bin/../lib

If you use the maven shade plugin (or similar) you can replace the whole "-cp ..." stanza with a "-jar myfile.jar". The "-D" arguments can be set in code instead (though I'd ask why you are allowing remote management without authentication and without SSL?).

The rest of the arguments are about GC tuning and logging. How would you do those things in a language that gives you a "simple" static binary? Either you can't at all, or they'd require an equally complex series of arguments.

It's a few lines of code to get sbt to combine everything into a single jar. Do other languages not requiring including libraries?

Statically compiled languages like Go do not.

Well, there are two separate questions here:

1. What should the default be? Java build systems default to building dynamically linked, though it's a few lines to change. IMO dynamic is a better default for large projects, as you usually have more library modules than executable modules. On the other hand a large project is likely to already involve a fair bit of build config, so maybe the defaults should be optimized for small projects.

2. Whether you allow dynamic at all. To my mind it's always worth having the option, and I think Go will come to regret not having it if and when it ever gets used for large projects.

There are plenty of large projects like kubernetes/docker/rkt/influxdb/tidb/cockroachdb and so on. Go is providing quite large memory efficiency and sub-millisec GC as compared to Java.

As of Go 1.8 it also provide plugin support though I am not sure if they are any where near Java in term of dynamic libraries loading support.

Only works on Linux as of now.

Sure there is, all commercial JVMs support AOT compilation to native code.

That most don't want to pay for them is another matter.

> the cold startup time of the JVM itself (which is not relevant on a server in comparison to desktop Java apps).

With microservices run as containers that are started on demand it matters; with other architectural choices it may matter less.

I'm interested in building CLIs with this.

I think for Server-Application Scala on the JVM will probably beat Scala-Native. The benefits of Scala-Native over Scala JVM are:

   - faster startup time
   - (drastically) lower memory footprint
   - fine hand-tuning of you application

All these things are not super important in server-applications. For example Java trades memory for throughput (higher memory footprint, but also higher throughput. These usually go hand in hand.).

For me most important benefit is running without pre-installed VM. Not particularly important for backend, but huge win for user-space tools.

You could always bundle the jre, and with Java 9 one can even make use of the newly introduced linker to create a customized image just with the relevant classes.

True, and if you wouldn't have to worry about Oracle licensing either (although you can avoid that currently with OpenJDK as well)

Servers very frequently benefit from lower memory footprints, as it can also dramatically improve performance by improving cache efficiency.

The large memory footprint of the JVM is memory for classes, profiles, things like that. Those are used to create optimised code and to recover when optimisations were too optimistic. When your program is optimised and running in steady state, this memory isn't actively used and so doesn't contend with your application memory and so has no impact on cache efficiency.

This sounds like a plausible explanation, but is this verified/verifiable? Are there memory profilers that can show me the relative sizes of the young/old/permanent generation segments of the GC?

I'm always blown away at the memory usage of JVM apps. Part of it is the fact that java has encouraged insanity-inducing inheritance hierarchies...but also it is incredibly hard to do dead code optimization on for such a static (type and compilation model) language (I blame dymanic classloading, but that's more of a guess than anything). Maybe what you're saying is the reason we don't see noticable GC pauses until you start seeing large amounts of data...but it is still a huge pain for low memory environments like phones, embedded devices, IoT, etc. And while memory usage is always gonna be higher on a GC'd language, the JVM still consumes vastly more memory than other languages like OCaml, D, Go, etc.

Yes, it's called "perm gen cache," or something like that, on any standard JVM profile. This roughly represents the memory used by the type system. It can get pretty high if you are doing something like auto-generating types (GUI, build systems, etc)

perm gen disappeared on java 8 and i think the high memory demand for perm-gen was one of the reasons.

In general (not for server apps), two major benefits of Scala Native are:

  - Predictable latency if desired (optional GC)
  - Very low call overhead for C ABI

As to your point about memory use, Java trades memory for convenience, not performance. GC requires substantially more memory for similar performance. I read an IBM blog (which I can't find at the moment) within the last week which showed a Swift web service running slightly faster than Java, but using only half the memory.

The following comparison is also interesting, with a JSON serialization example in Swift outpacing Spring/Java by a factor of ten... This is also running on Linux instead of macOS.

https://medium.com/@qutheory/server-side-swift-vs-the-other-...

The medium post is really a stupid benchmark. Check this for a real JSON serialization benchmark - https://www.techempower.com/benchmarks/#section=data-r13&hw=...

To be fair, in that list, the spring entry didn't exactly run circles around the competition either. It's somewhere between the better PHP contenders and even behind grails (which can be pretty accurately described as spring with layer of slowness added on top). Really looks like there is something unfortunate going on with the idiomatic way to implement those examples on spring.

> Predictable latency if desired (optional GC)

Does Scala Native support not using a GC? It seems like it would be difficult to get Scala working without a GC.

It supports direct allocation via both the heap and the stack.

  type Vec = CStruct3[Double, Double, Double]

  val vec = stackalloc[Vec] // allocate c struct on stack
  !vec._1 = 10.0            // initialize fields
  !vec._2 = 20.0
  !vec._3 = 30.0
  length(vec)               // pass by reference

...and...

  @extern object stdlib {
    def malloc(size: CSize): Ptr[Byte] = extern
  }

  val ptr = stdlib.malloc(32)

http://www.scala-native.org/en/latest/

Otherwise it currently uses the Boehm GC.

One big area that needs more work:

  "Scala Native doesn’t yet provide libraries for parallel
   multi-threaded programming and assumes single-threaded    
   execution by default."

But can be important if you only have a 512MB DigitalOcean instance or using small Linux containers/Docker.

I used the JVM on the 512MB DO instances and containers and they run fine. I think for containers there are other issues (most likely you are going in a micro service-direction where latency is eventually going to be important, so picking another JVM GC-algorithm might be suitable). There may be applications for which the 512 instances and JVM are not suitable, but you can most likely just upgrade the instance.

The JVM itself doesn't add a huge memory footprint. I remember Charles Nutter of JRuby fame calling it the "20-30 MB memory tax" (see http://blog.headius.com/2008/11/noise-cancelling.html).

A lot of the extra memory usage of Java apps comes from sloppy programming and from depending on lots of heavyweight libraries and frameworks.

If you include idiomatic Java programing as part of sloppy programming I also agree with that.

https://www.cs.virginia.edu/kim/publicity/pldi09tutorials/me...

Many things pointed out in this article apply to just about every managed language runtime. Implement a TreeSet in any language and you'll see the same overhead from object headers, memory alignment, etc. Java has some oddities that cause it to waste extra memory, but off the top of my head the only I can think of is 16-bit character Strings. Java 9 is supposed to help with that by allowing Strings to internally store utf8 characters.

I do like the slide though showing that people tend to assemble abstractions together and completely lose sight of the performance costs of what they are doing. There's also the fallacy commonly held by many that because someone took the time to write a framework or library, they must have also taken the time to ensure it's optimized well.

Go uses quite a bit less memory than Java.

http://benchmarksgame.alioth.debian.org/u64q/go.html

As per the Specjbb benchmark, JDK9 compact strings optimization itself provides,

  * 21% memory footprint reduction
  * 27% less GC
  * 5% throughput improvments

https://www.infoq.com/presentations/java-se-9-cloud - check this presentation for more details.

Impressive numbers.

Due to value types support, which are part of Java 10's roadmap.

I'd imagine it will be at least 5 years away when most popular Java software use this feature and java users see the effect of this.

Don't forget the ability to distribute binaries. This could be really nice for in-house tools.

I'm not involved with this project in any way, but I would expect performance to be overall worse with Scala Native. The advantages of Scala native are likely:

1) Much faster startup times

2) smaller memory footprint for small programs.

3) Potential for easier installation since no dependency on the JDK (assuming binaries are statically linked)

So basically you could use scala native to cover some cases that are better covered by golang or rust right now. For large and long-running server-side processes, the JVM is still king.

Yeah go for it. Nothing could go wrong: https://github.com/scala-native/scala-native/issues/543

Scala is a language I desperately wanted to like - high level, statically typed pure OO language. But in practice I found it almost unusable. The type signatures were unreadable and I distinctly recall writing a 100 line or so program where the type declarations crashed the compiler. And the tools themselves were huge memory hogs - sbt was a particularly bad offender (though otherwise quite pleasant).

I also did not get on well with the community, which seemed to have a lot of people with the attitude - "they won't let me use haskell at work so I'll make do with this shit". They didn't seem to understand or be interested in OO at all, and were very fanatical about driving application logic with types, purity, and the like.

Regardless, a native variant would be something well worth investigating if it ever reaches "production ready".

> The type signatures were unreadable

I think a lot of people see a dense 1-liner and panic when they can't read it as fast as a line of another language, when actually the same operation would be 5 or 10 lines in that other language. It's ok to take a bit longer to read it; in my experience if you take a deep breath and read slowly, breaking it down into pieces, it's all very simple - the combination may be complex, but everything is very modular and compositional so you can understand each piece in isolation and then put them together to make the bigger picture.

> I distinctly recall writing a 100 line or so program where the type declarations crashed the compiler.

Do you still have it? Those are pretty rare (and almost always a type error rather than a correct type, though I appreciate that doesn't help you much at the time) and I'd be interested in minimizing it into an actual bug report.

> And the tools themselves were huge memory hogs

Yeah that does happen. I avoid SBT, but eclipse isn't exactly light. They're putting a lot more focus on performance in recent releases which will hopefully address this up to a point.

> I also did not get on well with the community, which seemed to have a lot of people with the attitude - "they won't let me use haskell at work so I'll make do with this shit". They didn't seem to understand or be interested in OO at all, and were very fanatical about driving application logic with types, purity, and the like.

The community is pretty awful. But I found that as people got more experienced, myself included, they tend more towards the pure, strongly typed side of things and make less and less use of OO. I understand OO but am no longer interested in it, and I think most experienced Scala folk feel the same way, so while the language does have good support for it, you'll struggle to find many people interested in helping you with it. I'm not sure anything can be done about this - you can't expect the experienced users to answer questions they're not interested in.

It looks like you dislike the haskell like usage of Scala, I wonder how could you then produce types so complicated/weird that you crashed the compiler in 100 lines (In 4 years I have seen it crash only once in an unreleased version of pickling a long time ago).

I never claimed my criticism was fair or consistent :P I think I was more interested in Haskell-like type driven programming at the time though. I honestly can't remember what it was.

And it's not so much that I hate type driven programming or purity. I just view them as tools in my toolbox, not as methodologies or worse - religions. I still think objects are an amazing concept if you look at them through the smalltalk lens. Or if you take a moment to step back and realise that an anonymous function is very similar to an object with an "apply" method, or that a constructor for an immutable object is like a functor that partially applies many related functions at once.

In my opinion you are mostly describing Scala. Mutability and impurity is trivially easy to do in Scala. You may see that as a positive or a negative. Local mutability is definitely easier. Complexity of Scala comes from mixing OO and FP worlds and I think it's an amazing engineering feat. It will undoubtedly look ugly after you do Haskell though.

As a Scala guy on a Scala team, I'd think this would be most immediately useful on smaller fill-the-gaps sub-projects where we have to integrate with native code.

Looks like a good fit for the CLI we have been planning for our API bases on a Scala backend.

This is extremely exciting. I can't wait to try this out.

On the other hand, I wonder why such an effort was never carried out with Java itself? Or maybe it was but just never took off?

Would this be a good time to mention that a few years ago I wrote an experimental Java AOT compiler which worked by converting bytecode into C++?

http://cowlark.com/cowjac

I used Apache Harmony as the standard library, and binaries were about 2MB, which back in 2012 I considered unacceptably large; benchmarks were adequate only. I never got round to implementing a garbage collector but the hooks were in place for a simple mark/sweep stop-the-world collector. The hard part would be to actually make all threads of the program stop so they could be collected; I had a plan, but no implementation.

Interested parties may also be interested in my experimental Java JIT which worked by converting bytecode into Lua and then running it with LuaJIT:

http://cowlark.com/luje

It actually beat OpenJDK Server for some unrepresentative microbenchmarks. It's very unfinished, and generates fairly pathological Lua code which LuaJIT has trouble with, but the biggest problem is that Lua and LuaJIT can't do shared-memory threading, so most Java code won't work on it.

(JVMs are surprisingly easy to write. If you're interested in interpreters or code generation, I encourage you to try it.)

(Man, I completely forgot I wrote these until now!)

It was. Several times.

It didn't take off for two reasons.

Firstly, it's not really any faster than a good JIT, and Java has very good JITs. The startup time is better, but the startup time of a JVM itself is actually pretty small ('time java -cp . HelloWorld' prints 'real 0m0.112s' on my machine), and for the uses Java is usually put to, irrelevant. I'd be very interested to see some benchmarks for Scala Native.

Secondly, there's enough dynamic code in the Java ecosystem - bytecode weaving, dynamic proxy generation, plugin classloading, etc - that in practice, your AOT-compiled application need to have an interpreter along for the ride anyway. That means you don't even save the space overhead and distribution complexity of deploying a JVM.

> Secondly, there's enough dynamic code in the Java ecosystem - bytecode weaving, dynamic proxy generation, plugin classloading, etc - that in practice, your AOT-compiled application need to have an interpreter along for the ride anyway. That means you don't even save the space overhead and distribution complexity of deploying a JVM.

The key advantage of Scala in general is that these things are rarely necessary - the fancy type system and generally expressive language make it much easier to address the use cases with "vanilla" Scala instead. So Scala Native will hopefully be able to bypass this issue.

For mobile all of these don't apply. The mobile version of Java doesn't include as much dynamic stuff and doesn't require dynamic loading so you can build a really efficient AoT compiler e.g. https://www.codenameone.com/

The core JDK makes those assumptions and is indeed a bad candidate for AoT OTOH so is scala and it will prove very difficult to provide any value beyond what the JDK already offers.

There was RoboVM[1], but it looks like it shut down. GCJ [2] was GCC's Java implementation, which was natively compiled, but I think has been removed from GCC. I'm sure there are other projects.

[1] https://robovm.com/

[2] https://en.wikipedia.org/wiki/GNU_Compiler_for_Java

Codename One predated RoboVM, was always larger and is still open source and well supported: https://www.codenameone.com/

Will macros work with Scala Native, as they do with ScalaJs? (I believe that compile-time metaprogramming is the way forward, especially if the target doesn't support reflection or dynamic code loading).

"We support the whole language including the more advanced features such as method dispatch via structural types and even macros."

D'oh, thanks! Can't believe I missed that.

If i recall correctly, they are planning to (or have already) remove macros in a newer version of Scala.

Yeah, the current macro system will be replaced with scala-meta, which is a different macro system, but it hasn't happened yet. Scala-meta doesn't have def macros yet, so it's not ready.

This is amazing. Scala is one of my favorite languages to work with, and getting it native-code support will finally help make it fast enough to justify using it everywhere.

Somewhat relevant: https://blog.plan99.net/kotlin-native-310ffac94af2#.ijzik0jx...

Title says Kotlin, but it is about JVM languages going native, in general. Or should they?

They should. The article basically throws in some FUD to convince people that they don't want what they think they want. Except, we do want it! The JVM ecosystem's general aversion to native code and native system integration is what gave C# the opportunity to take over as the closest thing we have to a WORA language. If I want to write code in a higher-level language than C++ that can run on any mobile device or desktop OS, I do it in C#, not Java.

The Java ecosystem is playing huge catch up here, and I don't think it's a bad thing that people are exploring alternatives, whether that's Avian, the now defunct RoboVM, or LLVM-based backends.

The JVM eco-system is full of options to compile Java into native code.

The only thing is that most ignore there is a JVM world outside OpenJDK.

And even those that know that world aren't willing to pay for commercial JVMs.

Thus propagating the myth that Java doesn't support AOT.

Which in a way is also understandable, given Sun's political decision that AOT compilation was tabu.

Luckily Oracle isn't Sun and has heard the industry. So if all goes as planned, by Java 10, those that don't want to pay for AOT compilers, will have one in OpenJDK.

I can't get hello_world to work, something about a unresolved dependency: org.scala-native#sbt-Scalia-native;0.1.0: not found

I'm not a regular Scala user.

Please drop by our gitter chatroom [1] and we'll try to debug the issue together. It's likely some sort of environment misconfiguration.

[1] https://gitter.im/scala-native/scala-native

I am a regular Scala user and I hate that message

Great! Any benchmarks on whether the Scala compiler runs faster when compiled to native?

We're looking into that, but nothing to report yet.

Nice work, I hope this will eventually be a serious alternative to the JVM route.

Quick question: does this compile down to DOT before going to LLVM? Or has DOT not yet arrived in Scala Native?

DOT [1] is a theoretical calculus that is used as a foundation for our latest front-end compiler called dotty [2]. Scala Native is mostly a middle-end technology that bridges front-end compiler and LLVM. We plan to support dotty as a front-end in the future. 0.1 release is based on older Scala 2.11 front-end.

[1] http://www.scala-lang.org/blog/2016/02/03/essence-of-scala.h...

[2] https://github.com/lampepfl/dotty

Just what I hoped to hear, thanks!

Question: what kind of frameworks can be practically migrated to Scala Native?

In Scala it's very normal to write libraries or frameworks in "pure" Scala (i.e. not using reflection, annotations, proxies or anything like that), and all of those should be fine to cross-build for Scala Native (assuming their upstream dependencies do first). The state of libraries for Scala.js should be a good indication - anything that's cross-compatible with that is likely to work just as well for Scala Native.

I think the dependencies (like Netty for a web framework) would have to be migrated first. Scala native has a subset of java core libraries (from e.g. io, nio, util) rewritten in Scala, which could be a blocker if there's anything missing.

Does anyone have any example binaries compiled with this? What are the sizes that you could expect?

They say one of their targets is using this for command-line tools (I'm guessing for startup speed and needing to be small in memory footprint) but it's not of much value if an "echo" or "grep" implementation takes up 15 to 30MB on the drive.

> but it's not of much value if an "echo" or "grep" implementation takes up 15 to 30MB on the drive

In this day & age? It might be of value if an echo or grep takes ~20MB and a full-blown HTTP REST server with DB access takes ~25-30MB. Why do I care again exactly whether there is a fixed portion of bootstrap/runtime core code in the binary, as long as I'm not writing echo or grep?

(Any helloworld app not written in asm (or lib-less C) will consist largely of "non-helloworld code" in terms of "bytes occupied in the binary", right?)

Dunno might be of concern eg with embedded/raspberry and such, but for code to be "moved from a jvm-kinda flair to a go-kinda flair" I'm not seeing the issue .. yet ;)

When you are writing that full-blown HTTP REST server with DB access you would probably be served better on the JVM. Scala native is just as garbage collected as Scala JVM and garbage collection is where the JVM shines brightly.

In my eyes, the main value proposition of Scala native will be "you don't have to learn a new language if you need to write an echo or a grep", with "you don't need to learn a new language if you need to run some of your code on a platform that does not have a reasonably good JVM" a distant second. The former would suffer a lot from oversized binaries, the latter depending on the specifics of the target environment.

A major value proposition is also "I want to run my code with predictable latency". This opens Scala to embedded and realtime development.

The LLVM AOT optimizations are also likely to be a lot stronger than JVM JIT optimizations - granted it's good to compile on the exact target architecture. Also granted that sometimes dynamic optimization helps a lot.

Nice handle BTW. lol

Binary sizes kill the assumption that it would be possible to use in embedded development. An unpredictable GC makes it impossible to use for realtime work.

For both you need:

   * An understanding of what kind of code will be generated
   * Predictable, and hopefully verified, GC
   * Reasonable binary sizes
   * Very good tree shaking

Currently the only things I've seen used for realtime developmend have been C, C++, Rust, raw assembly, and LISP. LISP and raw assembly are dead now. Rust is just starting and has a lot of progress to be made. C++ is showing some reasonable improvements. C is C.

If Scala Native could deliver on small binary sizes, "0 runtime" like Rust, and a way embed inline assembly like instructions then it would be a winner.

But how much predictability can you get in presence of garbage collection? Granted, the JIT can also be an element of surprise that is out of the picture with AOT and the statically linked GC can never be configured differently from what you expect, but it's still garbage collection and not Rust or a C.

There's a lot of examples of GC in realtime systems. It's just a different set of concerns. Rather then the programmer worrying about memory constantly, the compiler engineers are worrying about memory constantly.

This is a very fun read: http://www.flownet.com/gat/jpl-lisp.html

Is this actually the case? I'd assume that Scala has the same dynamic nature as Java, and so AOT won't be able to convert dynamic dispatch to static as frequently.

I've created once custom barebone JavaVM with binary size of around 100k. It was made as an executable jsmile.exe that was capable to read bytecodes cat'ed to the executable itself : http://www.terrainformatica.com/org/j-smile/index.htm

The goal was to create JVM suitable for standalone GUI applications. Project was abandoned when Sun/MS Java wars started in favor of the Sciter (https://sciter.com).

As of NativeScala ... I think that approach (binary with nano JVM + attached class files) may work better and with less effort. Scala needs JVM infrastructure, GC, etc. as far as I understand.

How do the compilation times compare to targeting the JVM?

There is roughly 1-2s (after sbt warm-up) penalty to perform compilation to native on iMac (Retina 5K, 27-inch, Late 2015) with 4 GHz Intel Core i7. The linker is highly parallel so you can throw more hardware at it as your project grows bigger. Barebones project includes nearly 1000 classes and 2000 methods in transitive dependencies that are being compiled due to closed-world nature of the toolchain.

so dream comes true!

P.S. I think this is related to rust, in a sense before Rust there was no serious competitor to C/C++, but after seeing what Rust doing to C/C++ I think there will be more native language to compete with in low level area.

Between the "scripty" GC agility of Golang and all the impressive low level benefits that Rust is enjoying from folding memory management into the type system, I suspect that Scala native will be a difficult sell.

On the other hand, the new targets (js and native) might make Scala more interesting to Java pragmatists who don't care much about going functional but would not mind writing Java in a more streamlined syntax.

Golang is pretty deficient as languages go, really. I find Scala to be far more appealing.

At first glance of this article's title, I thought it would be a terrible idea to use Scala to write native mobile applications. Imagine using Scala.JS to write a NativeScript/React Native app. shutters

> Imagine using Scala.JS to write a NativeScript/React Native app.

People do [1], and are very happy with it.

[1] https://github.com/chandu0101/sri

It seems like that would open up a nice way of using Scala on iOS also.

> This opens the door for Scala to be used in environments where full-blown virtual machine is usually an overkill

Not sure if I get this, don't Java VMs support this use case (J2ME) ?

Important bit: "The project has reached a point of feature completeness in terms of the coverage of the Scala language. We support the whole language including the more advanced features such as method dispatch via structural types and even macros."

It must be frustrating to work on a project like this, see areas where the language can be improved, but only be able to do the work to make it purely compatible. Hopefully some good comes out in the form of some good SIPs.

Or inspiring! I guess it depends on your outlook.

The generated binary for simple Hello World is 3.45 MB which is quite a lot for printing one line of text but it can be compressed to 326K using UPX.

All of our binary size problems are mostly caused by a single well-known issue [1]. We hope to address that in one of the future releases. In the meanwhile, compressing binaries with upx is a temporary solution that can alleviate some of the pain.

[1] https://github.com/scala-native/scala-native/issues/180

Rust's is almost the exact same size (3.46 MB). It's probably just that the libraries that are statically linked by default (jemalloc, std, etc.).

That's correct. We statically link all of the transitive Scala dependencies.

This is certainly an impressive piece of work. However, i think it's worth paying attention to the limitations, and the use cases they imply; overall, this looks less like "compile your existing Scala app to native code!" and more like "use Scala to interface with existing native libraries!".

On the other hand, it's also worth bearing in mind that this is version 0.1.0; over time, some of these limitations will lift. What i don't know is whether Scala Native will develop into a complete version of Scala which compiles to native code, or evolve into a variant of Scala more tightly adapted to a niche of talking to native libraries.

Anyway ...

(1) No threading [1]:

Scala Native doesn’t yet provide libraries for parallel multi-threaded programming and assumes single-threaded execution by default. It’s possible to use C libraries to get access to multi-threading and synchronization primitives but this is not officially supported at the moment.

So forget about using Akka for now.

(2) NullPointerExceptions are replaced with segfaults (hopefully) [1]:

A number of error conditions which are well-defined on JVM are undefined behavior: Dereferencing null. Division by zero. Stack overflows. Those typically crash application with a segfault on the supported architectures.

That's not so bad; where Java apps might let nulls flow around and rely on catching NullPointerExceptions to recover from them, Scala apps are much more likely to use Optional consistently.

(3) If you do want to talk to a native library, and you need to allocate memory to do it, you're on your own [2]:

Unlike standard Scala objects that are managed automatically by the underlying runtime system, one has to manage native pointers manually.

Scala Native provides a built-in way to perform stack allocations of unmanaged memory using native.stackalloc function: [...] When using stack allocated memory one has to be careful not to capture this memory beyond the lifetime of the method. Dereferencing stack allocated memory after the method’s execution has completed is undefined behaviour.

Scala Native’s library contains a bindings for a subset of the standard libc functionality. This includes the trio of malloc, realloc and free functions

Java's traditional JNI is a verbose, slow, pain in the stdout, but it was designed pretty carefully to avoid problems like this.

(a) Intermission! Check out how they do type-level numbers [1]:

Natural numbers are types that are composed of base naturals Nat._0, ... Nat._9 and an additional Nat.Digit constructor.

That's a new one on me!

(4) Incomplete JDK libraries [3]:

Scala Native supports a subset of the JDK core libraries reimplemented in Scala. Here is the list of currently available classes: [...] This is an ongoing effort, some of the classes listed here might be partially implemented.

The list has most of the fundamental stuff - a good chunk of java.io and NIO, the collections, java.lang, atomics. But no java.text, java.net, concurrency, regexp, date and time, JDBC, reflection, XML, etc.

They don't mention how much of the Scala libraries they support. I would imagine that they can build anything that's in pure Scala and depends only on JDK classes in that list, so you'll get the core language stuff and the collections. Not sure.

[1] http://www.scala-native.org/en/latest/user/lang.html

[2] http://www.scala-native.org/en/latest/user/interop.html

[3] http://www.scala-native.org/en/latest/lib/javalib.html

I wonder if Akka will be a part of scala native? It's sort of considered stdlib? When our team spiked on Akka, we liked it and got our near-reality proof of concept to work. It'd be awesome to have such a high level library like Akka compile to a binary.

Nice job, thanks! Seems that right now you had implemented some wrappers for libc and some java classes ported to scala. What plans do you have to further evolve API? Will you focus on reimplementing java.* or create your own set of classes?

Is Scala Native GCed?

Should be, because Scala doesn't have strong vs weak reference distinction.

yes (https://github.com/scala-native/scala-native/issues/128)

Does this mean the future of Scala is off the JVM? I ask because the post calls the JVM impl. a "reference implementation".

IMO the future of the JVM is highly multi-platform: JVM, native and JS.

Does that not mean that it's the canonical one which other implementations should be measured against? That would seem to imply just the opposite.

Does this provide a garbage collector?

Yeah it uses the Boehm garbage collector.

I asked a similar question on the reddit thread about this. I'm bummed that I didn't find this answer right away in this documentation. Probably the most interesting question for me

Boom GC is a dependency

Any Windows pre-built binary to try?

I imagine building this stuff on Windows will be challenging :/

Will this mean that we can get rid of type erasure when running native code?

Why is type erasure necessary anyway? Surely this can be worked around somehow. You could manually do what you do with array length in C - pass the information along manually. Why can't the compiler do this in a transparent manner for us? Is it speed?

It's more likely to make type erasure more extensive. The main effect of type erasure is to stop people doing naughty things in generic methods. When one doesn't have to support JVM reflection, erasure can be pushed further.

Will language support destructors for manual memory management ?

Please file a feature request on http://github.com/scala-native/scala-native, we can discuss the details over there.

good idea!

Most things called native are not.

Unfortunately, this requires an existing Scala compiler to build, so it won't be useful as a bootstrap compiler for Scala on the JVM. Does anyone here know of an alternative implementation of Scala that could be used to build the libraries and tools of the reference implementation from source?

It is a problem that many compilers cannot be bootstrapped from source without a trusted binary of a previous release.

It's a universal truism that all compilers cannot be bootstrapped from source without a trusted binary. It's true that in the world of standardized languages (which mostly means "C" in practice) there are compilers (mostly just gcc and clang) that can bootstrap themselves with a trusted binary of a previous release of some other compiler.

Is that such a big deal?

I wrote "a trusted binary of a previous release" not just "a trusted binary". There is obviously a difference between having a small set of trusted binaries to bootstrap and having a bootstrap binary for every language or build system.

There are several compiler implementations that enable bootstrapping from alternative implementations, which shifts the problem to a simpler language, which may already have a bootstrap path.

how big of a problem is it, really?

is it a theoretical problem or a practical problem for industry users?

It is a practical problem for people who want to have a correspondence between source and binary. Some users would like to have to rely on as few opaque binaries as possible. There are efforts underway to build a minimal C compiler in a subset of Scheme that can be implemented on bare metal. The goal is to reduce the set of binaries that needs to be trusted or audited manually.

Reply on news.ycombinator.com