Hacker News Reader

Comment by densh | Hacker News Reader

Hi all, I'm the author of the project and would gladly answer any questions.

How's the managed runtime implemented (GC, memory model, etc.)? How did you get rid of JVM and JRE class library dependencies of Scala? What's the debugging story? I'd be great if this kind of info found its way onto th website. Is this in any way related to vmkit?

Excited about this, keep up the great work!

Looks like it's currently using the Boehm–Demers–Weiser conservative garbage collector. In scala-native/rtlib/src/main/resources/rt.cpp it includes gc.h and allocates memory using GC_malloc, i.e. not in precise mode.

Ouch.

The garbage collector is typically the most difficult part of a language implementation, so I'm not surprised. On the other hand, a simple GC can be sufficient for many projects.

They can still replace it later once they're further along with the rest of the implementation. I doubt they'll keep it.

> How's the managed runtime implemented (GC, memory model, etc.)?

In a first release we're going to use Boehm GC. That's just a stepping stone, not a long-term strategy. Stay tuned for news on this front.

> How did you get rid of JVM and JRE class library dependencies of Scala?

We're reimplementing / porting from Apache Harmony all the things we need.

> What's the debugging story?

We're going to integrate with lldb.

> Is this in any way related to vmkit?

Not related in any way, apart from being based on LLVM.

> We're reimplementing / porting from Apache Harmony all the things we need.

Doesn't that create licensing issues? You already seem to have code from Harmony in the project, but the project's license is neither Apache nor does it mention it.

Additionally, won't that cause trouble with Oracle, considering the lawsuit against Google?

We're are going to have an official guide on licensing in next few days.

> We're reimplementing / porting from Apache Harmony all the things we need.

Would GCJ be any use? They had native compilation over a decade ago.

(Although developers moved on to IcedTea, iirc)

Hans Boehm's GC is actually really, really good. Before you embark on a massive engineering effort that will likely lead to little gain and a lot of pain, I would measure the overheads very carefully. I am not at all convinced precise GCs buy you that much over the Boehm GC for most apps, and they fundamentally kill interop with C code.

(I can't recall for sure but I believe I had this conversation with Martin in a taxi ride not too long ago :)

Mono was a long time on Hans Boehm's GC and despite feeding it a lot of metadata never got satisfying results with it. Eventually they replaced it with a custom GC (SGen).

This is exciting but I've been burned in the past. Your commit graph seems to show you're serious so I hope you stick with it.

Will probably have more questions after I browse through the code but some basics: Q: How much of the collections library is covered with full support? Partial support? NYI w/ ETA? Q: Cross-platform support? If I wanted to run this on iOS and Android today, what do I have to do? What about on Windows or OSX? Q: How does GC work? Is there a way to take more control over GC for Scala objects? Q: Is it possible to ‘link’ in external dependencies, i.e. Joda Time. How do I do this right now? Q: Does the compiler use a translation layer? If so, is it consistent with Java 7 or 8?

On another note, the c extern stuff is excellent — exactly what I’d like to see for portability and performance (without the headache of something like the JNI).

> Q: How much of the collections library is covered with full support? Partial support? NYI w/ ETA?

Non-parallel collections should work in first developer preview (to be announced.) Parallel collections may took a few release to get working.

> Q: Cross-platform support? If I wanted to run this on iOS and Android today, what do I have to do? What about on Windows or OSX?

If you can compile and run C code with Clang on some platform, it's reasonable to expect Scala Native to be run there eventually as we're based on the the same underlying compiler toolchain. Currently we mostly aim at POSIX environments.

> Q: How does GC work? Is there a way to take more control over GC for Scala objects?

Boehm GC for now, but that's not our long-term strategy.

> Q: Is it possible to ‘link’ in external dependencies, i.e. Joda Time. How do I do this right now?

We can link with libraries via C ABI. Java libraries need to be ported over to Scala.

> Q: Does the compiler use a translation layer? If so, is it consistent with Java 7 or 8?

We strive to keep the same semantics as Scala/JVM.

Ah, so since Scala Native is not built on the JVM you can't just take a pre-existing Java library and make it interop. You need it to be real Scala. The only options then are to use a convertor or manually port the library from Java to Scala?

Yep, or drop the dependency. E.g. For joda specifically, you should probably move to Java 8 date API, which is more likely to be ported.

Bare in mind that scalajs has the same issue and a lot of work has already been done!

Is this actually affiliated with the facility that is behind Scala for JVM? I saw the banner of the university on the Scala Native site, is that for real?

I have been so extraordinarily impressed with Scala, and with the contributions its made in the PLT field, I'm very excited for Scala Native to start picking up steam.

Can we expect the same level of excellence from Scala Native as we've received from Scala for JVM?

Yes. Scala Native is developed at EPFL for real.

Will Lightbend be supporting this project in any way?

The reason I ask is that I see on GitHub that you are the only contributor so far. This project is quite a large commitment for a single person, I'm hesitant to play around with Scala Native without knowing more about future plans for support, milestones, etc.

Having said that, this project looks awesome. Nice work so far!

Agreed. I see the statement "It's being developed at EPFL by one engineer" as a very high risk that once this student graduates or moves on to a different university, the project will be abandoned and die.

It's worth noting that there was an LLVM-backed Scala project that was abandoned and died, so I definitely hope this avoids the same fate.

There was also a Scala.net project that was abandoned and died.

That's very common for academic projects.

Scala.net didn't die because it was academic. It died due to lack of interest, as nobody cared about a Scala for .NET

ISTR a significant reason it died was the .Net platform's reified generics, which made making a language with generics and a type system more expressive than the underlying platform problematic, and especially made it difficult to have something that would be fully compatible with Scala-on-the-JVM.

Exactly.

It died for technical reasons because .net supports reified generics, which made it pretty much unable to support Scala's type system.

People who keep saying that erased generics is a stupid idea have no idea what they're talking about.

scala-native is facing insurmountable difficulties, the kind that will certainly not be conquered by a single student who will stop working on it as soon as he graduates.

That's not really true. Yes there are challenges with reified generics and yes Scala's generics don't fit. However you can work around it by doing the type erasure yourself. You can always consider a List[Int] to be a List[Any] and be done with it. And yes that's going to generate inefficient code, but ClojureCLR doesn't seem to mind.

No, the reason for why Scala.Net didn't happen is because nobody cared. To find proof of this, you only need to look at Clojure. Its .NET implementation is well maintained by David Miller, yet it's very unpopular. And the reason for why .NET developers don't care is because they don't have an open source culture. Or in other words, if it doesn't come from Microsoft, then it doesn't exist.

If you do erasure, then you can't also have the level of platform integration Scala has on the JVM: Scala on .NET either ends up as a different-but-similar language that isn't 100% compatible with Scala-on-JVM on a language level (as well the library differences), or its a second-class citizen that doesn't integrate well with the .NET platform, in which case, why have it?

So, in a sense, no on cared -- because neither of the available options was anything anyone wanted.

It's worth noting that Scala.js started this way, and is now mature and used by thousands of developers. I foresee a similar path for Scala Native.

There have been so many successful projects started by one person. Shooting it down before seeing what will be done is a bad idea.

I'm interested in it breaking away from the JVM to become more performant, since that has been one of my concerns about Scala; there is a lot of good about JVM-based languages, but when in the ring with other languages like Go, you need to be quick to win.

I hope that others join him, but there's no reason to believe he can't do it if he sets his mind to it.

Note that I am not concerned about the fact that the project was started by one person, nor am I dismissing it altogether.

The author of this project is most definitely a smart and capable individual. However, at some point this project will reach a point where it requires more work than one individual can achieve, hence the question regarding official support - is there any financing, plans for new contributors, etc.

I understand your concern, but, as I said, a number of authors have not have financing, plans for new contributors, etc. in the beginning and have been successful; these are not requirement for success in the beginning.

For example, here is the story of how Python got its start from Guido van Rossum (quote from 1996):

"Over six years ago, in December 1989, I was looking for a "hobby" programming project that would keep me occupied during the week around Christmas. My office ... would be closed, but I had a home computer, and not much else on my hands. I decided to write an interpreter for the new scripting language I had been thinking about lately: a descendant of ABC that would appeal to Unix/C hackers. I chose Python as a working title for the project, being in a slightly irreverent mood (and a big fan of Monty Python's Flying Circus)."

And the history of how Ruby got its start by Yukihiro Matsumoto (quote from 1999) is similar:

"I was talking with my colleague about the possibility of an object-oriented scripting language. I knew Perl (Perl4, not Perl5), but I didn't like it really, because it had the smell of a toy language (it still has). The object-oriented language seemed very promising. I knew Python then. But I didn't like it, because I didn't think it was a true object-oriented language — OO features appeared to be add-on to the language. As a language maniac and OO fan for 15 years, I really wanted a genuine object-oriented, easy-to-use scripting language. I looked for but couldn't find one. So I decided to make it."

Denys Shabalin is working under the author of Scala and corresponding regularly with the author of Scala.js, so he has more support than either van Rossum or Matsumoto did, and neither of them had the plans you spoke of- they just wrote the language because it is what they wanted to do.

Well, Ruby or Python would not have been suitable for many use cases in those days, and probably weren't as widely announced as scala-native is being.

I'm currently employed as research assistant at EPFL at Martin Odersky's lab and closely collaborate with Martin and Sébastien Doeraene (the author of Scala.js) on technical design and implementation of Scala Native.

Can you please post your slides from Scala Days?

EDIT: Thank you!

https://github.com/densh/talks/blob/517b20c30dd4aaf390785039...

Is there a video online for it?

In the past the videos from the Scala Days talks are usually posted online through Parley's, I expect that would happen again this year. Though it is usually a few weeks to a month or two after.

What are your mobile plans? :) I realize this is probably a ways off, but I've dreamed of being able to write core logic in a language like Scala & then using that code from ios & android.

Will scala native generate some c-linkable files?

iOS/Android support is the most requested feature. Stay tuned for updates on that front.

Scala Native generates LLVM IR that can be compiled to C-linkable code but for now we focus on "one statically linked application at a time" use case.

How do you plan to deal with the large amount of metadata required for some stuff? Like the timezone database for calendars, the CLDR for formatting stuff, or various Unicode tables for handling case folding/comparions/etc.?

Will you be using native libraries/the data blobs installed on the host system? Or will you ship these things with each binary?

What are the plans with tail call optimisation in this version? or do you get that for free from LLVM?

The slides mention that proper tail calls "just work", even mutual ones. :-)

We get that for free from LLVM.

Do you see any possibilities to undo/mitigate JVM-specific design decision which handicap the language? For example could you not erase generics or would that cause too much problems incompatibilities?

Idiomatic Scala doesn't care at all about erased generics. With implicits and macros and libraries built upon them like shapeless, you can do everything you'd want to do with reified generics and more and it'll be done and statically checked at compile time instead of dynamically at run time.

I know there are ways like typetags to work around it, but on the JVM this is never going to run without extra work:

    def erased[T](xs: List[T]) : String = {
      xs match {
        case ns : List[Int] =>"Natural numbers"
        case fs : List[Double] => "Floating point numbers"
        case _ : List[T] => "Something else"
      }
    }
    System.out.println(erased(List(1,2,3)))
    System.out.println(erased(List(1.0,2.0,3.0)))

Which is purely because of a JVM runtime limitation. Will native scala behave identically to JVM scala in this case or will you be able to improve?

The part that you're missing is that code like this in Scala is totally unnecessary and it never happens:

    case ns : List[Int] =>

Worst case scenario, you can always do the following and it would be more idiomatic, no type erasure standing in the way:

    val listInt = listAny.collect { case x:Int => x }
    val listDouble = listAny.collect { case x:Double => x }

But even that is totally unnecessary. You know why? Because your function does not make sense. What could you possibly do with a function that takes a List[T] and returns a String?

The only way this would make sense would be if you are talking about a List[Any] deserialized from somewhere (like some shitty JSON library). But then in Scala we don't really work with Any, that should never happen and if you see libraries that use Any in their API, drop them like they are hot ;-)

> Which is purely because of a JVM runtime limitation

No it's not. Talk to compiler authors. This is actually a freedom, because by introducing reified generics in the runtime, then language designers either have to deal with inefficiencies due to boxing because of extra conversions, or to limit their type system, so for example you can say goodbye to higher-kinded types. A runtime that has reified generics is not a multi-language runtime. In fact many languages are doing type erasure. Haskell is doing type erasure. And why shouldn't it, I mean, type casting and isInstanceOf checks make no sense in Haskell.

That's a very interesting aspect that's also being explored in our lab under Dotty [1] project and it's Linker [2] subprojects. It's not a direct goal of Scala Native.

[1] https://github.com/lampepfl/dotty

[2] https://github.com/dotty-linker

In point of fact, I think its not so much a handicap: http://stackoverflow.com/questions/20918650/what-are-the-ben...

On what OS does it run right now?

Mentions using LLVM, so all OS should work. Not sure about the stdlib.

All OSs which LLVM supports should work. Not quite "All OS".

Scala Native is currently being developed on Darwin/Intel/x64.

Fantastic work!

* Any plans to strengthen the Scala core lib so one doesn't have to reach out to JVM for mundane tasks like I/O or threading?

* Is there a simple way to use raw C / C++ libraries from Scala native?

* Is there a http://search.maven.org/ equivalent?

1. That's our long-term goal, but don't expect that to happen overnight.

2. @extern objects are an easy way to call C code. There is no C++ interop at the moment.

3. Not yet.

Heya, your project looks interesting I must say. I was curious, is this mainly a 1-man gig or do you have support from the existing community (e.g. Odersky).

I heard some rumors that someone (you) was doing it as a graduation thesis under Odersky, is that true or complete bollocks?

Just wanted to say thank you for your amazing work!

Hi,

What is the rationale for doing this, given that the Graal / JVM guys are already developing an AOT compiler which can compile Scala to native binaries?

Is this motivated by performance? The desire to write C libraries in Scala? What?

Given the high level, dynamic nature of Scala I am skeptical LLVM+Boehm GC will result in faster code. I expect it would result in slower code.

EDIT: OK, I just saw the link to your talk below. I feel you should know a few things:

1) The Hotspot team are adding AOT compilation to the JVM. There is a talk on it here. This mostly eliminates warmup time:

https://www.youtube.com/watch?v=Xybzyv8qbOc&list=PLX8CzqL3Ar...

You can't eliminate warmup time entirely because profile guided on-the-fly optimisation is actually quite powerful and an AOT compiler that doesn't have profile data to work with can't generate code that's as good. But they support tiered AOT, where the AOT compiled code does basic profiling of itself, and then it can still trigger profile-guided JIT compilation if you want to.

2) Interop with native is being heavily worked on in project Panama: another JVM upgrade project. With the current prototype you can feed it arbitrary header files that are parsed with clang, and it generates Java interfacing files that are efficiently compiled (no overhead). There's a Pointer<T> type and so on.

3) Stack allocation isn't so easy to do unless you are willing to pretty wildly violate the safety of the language. It's also been found to not help much when very adventurous JVM implementors tried it anyway (Azul), because a good generational GC makes short lived allocations so cheap. Valhalla is doing value types on the JVM but that's about more than just stack allocation.

4) You can do manual memory management on the JVM already, via the Unsafe class. I'm sure a Scala DSL would make it have a more convenient syntax.

5) Java 9 will have a static linker equivalent that generates stripped, standalone, optimised application packages that don't depend on any separate JVM. The AOT compiler will be a plugin to this static linker.

So I don't mean to discourage you if this is a fun research project, but you should be aware that the daydream is not one that the actual Java team are ignoring for mysterious reasons. They're working on it too, and are better funded and taking a more general and compatible approach. By the time you have a tool that's usefully complete it'll take years and by then there might not be much difference between the approaches.

What Oracle does is cute, but I will put my bet on people who have a solid track record at shipping.

Reply on news.ycombinator.com