Dotty: a next generation compiler for Scala
dotty.epfl.ch
dotty.epfl.ch
Scala 2.12 is late, and not having easy java8 interoperability is becoming more and more of an issue.
I love the language but it is becoming harder to recommend it for new things due to the ambiguous roadmap.
And I think a bit of simplification and formalization is exactly what Scala needs, in the long run.
(I don't mean to be terse, I just don't know how to ask that with more words...)
Are you based in the UK? It would be interesting to compare notes. :)
Look me up in the GAL (My username is my name).
Are you saying two new compiler engineers were hired by Lightbend?
Because if they're working on Lagom that's a different (mostly Java) story https://github.com/lagom/lagom
Different projects require different approaches. The introduction of the community process added 5 new maintainers and the merge of Slick and FreeSlick caused a lot of activity and contributions, so I'm pretty happy.
> Are you saying two new compiler engineers were hired by Lightbend?
Yes.
> Because if they're working on Lagom that's a different (mostly Java) story https://github.com/lagom/lagom
No.
I do think 2.12 (and 2.11 before it) either had unrealistically optimistic deadlines or suffer from poor project management. 2.12 has slipped by well over a year at this point, which is sad for frameworks like Spark which are going to stick to 2.11 for their next MV cycle.
https://groups.google.com/forum/#!topic/scala-internals/lv4s...
The fact that the 2.11.x series had a number of "dead on arrival" minor releases also confirms: better not to hurry to adopt a new Scala release.
While the theme for 2.12 is easily summarized (Java 8), I assure you the changes behind the scenes go all the way to the core. Compiling lambdas and traits to the best Java 8 bytecode we can think of, unifying type checking of Single Abstract Method types and Scala's built-in function types, as well as rewriting the whole optimizer and byte code emitter was no small feat! It's been very satisfying to rework these parts of the compiler, and I'm excited to see 2.12 going live soon!
We're cutting the last milestone (M5) this week, with RC1 scheduled for the second half of July -- assuming M5 is well received. We're pretty confident, as we build over 1MLoC of OSS Scala every night using dbuild (see scala/community-builds on github).
We at Lightbend care deeply about language stability and compiler robustness, as we believe they are key for driving Scala adoption by the community and the enterprise alike. Being PL researchers, we are also excited about simplifying and evolving Scala! This is why we're working on a smooth & steady migration path towards the ideas being incubated in Dotty.
Scala's process is a dialogue, which means coordinating with Martin Odersky and his teams at EPFL and the Scala Center, as well as the whole Scala Community. Ultimately, we at Lightbend set the Scala 2.x roadmap based on this consensus (since 2.10), and take on OSS development and maintenance (as well as commercial support for our customers). With the founding of the Scala Center, we're happy to share governance, while remaining as committed to the day-to-day engineering and representation of the pragmatic robustness and careful evolution voice :-)
This dialogue comes naturally to us, as half of our team are alumni from Martin's lab (Lukas & I have been working on the Scala compiler for almost 10 years, ranging from adding support for type constructor polymorphism in Scala 2.5, rewriting the pattern matcher in 2.10, implementing named & default arguments, while working on the theoretical foundation of an effect system for Scala as well as the core of Dotty), while the other half are long-time contributors to all aspects of Scala.
Sure, releases sometimes slip a bit -- we are a small team (we're a small company -- our team size is no outlier), and while we get to spend the vast majority of our time working on OSS, we also have some commercial obligations inside the company (e.g., Scala support tickets are handled by us). The other reason is that we have worked on 2.11 for 6 months longer than we usually spend on a major release, because we wanted our Java 6 users to get as many of the 2.12 features as possible on their platform (2.12 requires Java 8), as well as providing a preview of the new ASM-based back-end and optimizer.
The Scala team at Lightbend takes care of half of all PRs, with the other half being contributed by our awesome community (including EPFL, though they mostly focus on their research and Dotty, which is how things are supposed to be). We've worked hard on making it easier to contribute to Scala, and I'm very glad to see the rate of community contributions trending up!
The Scala and Dotty teams talk regularly (we just had a Dotty & Scala summit before Scala Days), to exchange tips on compiler performance as well as to work towards convergence for Scala 2.x and Dotty. For example, the 2.12 trait and lambda encodings were tried first in Dotty, with our real world adoption of these ideas feeding back into the Dotty compiler. For 2.13, we're planning feature flags to implement the first wave of Dotty features and restrictions (and vice versa for Dotty emulating Scala 2), so that you can start migrating your code bases. For some more background, here are our roadmap updates: http://www.scala-lang.org/news/2016-schedule/, http://www.scala-lang.org/news/roadmap-next/, http://www.scala-lang.org/news/2.12-roadmap/.
It's becoming easier for me to recommend it due to the slow moving roadmap. So are you going to tell people to stay away from the JVM for its ambiguous roadmap (openjdk projects being delayed, etc)?
Scala's Road Ahead - by Martin Odersky https://www.youtube.com/watch?v=_2oGY8l67jk
CBT is very developer friendly, fast, simple, easy to use. Join us on gitter https://gitter.im/cvogt/cbt or check out my Scala Days talk about it :) https://www.youtube.com/watch?v=5COKEp7ItZk
Also the simplifications coming with sbt 1.0 like removing the Build.scala and dropping support for multiple build.sbt files makes it a bit cleaner in my mind at least. Before everything was spread around in multiple files (and possible 2 different build definition styles with the .sbt and .scala files) instead now everything is in pretty much one single file with a clean enough syntax for anyone to learn. (as someone said you only really need to know :=, += and +== of which the latter 2 function the same as they do in the standard lib for collections)
edit: Sorry was wrong about the multiple build.sbt support. Its still in but working with a single build.sbt has been made easier over the 0.13.x versions and we reverted all of our multi project builds into that and I just assumed it was being dropped (We also had a lot of Build.scala stuff so I guess i got mixed)
<groupId>com.foocorp</groupId>
<artifactId>foo</artifactId>
<version>1.2</version>
"com.foocorp" % "foo" % "1.2"I generally prefer gradle's method of 'com.foocorp:foo:1.2' but that isn't acceptable statically. scala string interpolators could make it a thing, though.
libraryDependencies += "com.lihaoyi" %% "scalatags" % "0.4.3"
libraryDependencies += "com.lihaoyi" %%% "scalatags" % "0.4.3"Far and away the biggest blocker to learning sbt is the shaky IDE support for the build project. Apparently that's coming up next in 1.0; will be nice to click through to source from sbt build files and instantly grok what X symbol means. Have wasted a lot of time over the years slogging through documentation...
ModuleID(organization = "com.foocorp", name = "foo", revision = "1.2")I won't go back.
If you stick to defining dependencies maven works perfectly. If you encapsulate whatever other crap you want to do as proper maven plugins maven works perfectly. It's the people who decide they absolutely need to output to a different folder on Tuesdays but can't be bothered to write a proper tested plugin for doing that who get themselves in trouble.
Your comment is accurate. If you want to do a very specific set of things, Maven works fine. If you want anything outside that, it gets much harder.
Maven is not a really like any of the other build systems. I would say most build systems are glorified Make. Maven is not.
SBT, Gradle, and even old school Ant seem far less verbose than Maven but Maven really shines when you have hundreds of projects in an organization. You can get extreme consistency.
You don't have to have everything checked in one gigantic source tree which makes open sourcing projects subsets of your code base easier. Yeah Gradle and SBT can sort of do this but not nearly as decoupled as Maven does it (generally with Gradle and SBT you have to do some path magic).
Just about every PL and editor can readily edit Maven pom files. That means with out loosing comments or spaces you can have scripts go and rewrite hundreds of pom files or even just do a sanity check to make sure that they are consistent.
This is the complete opposite with build systems where basically the build system is turing complete scripting language. Projects have major drift and you loose lots of consistency.
I'd rather just change a single file instead.
It's a good way to differentiate football-team partisanship from actual principles (not that any principle applies 100% of the time).
Not only that, but virtually no-one uses this Turing-Complete facility. In Gradle at least, I never see if- and for-statements. By using dynamically-typed Groovy in Gradle, you not only lose the ability for massive rewrites, but often also don't get any of the TC benefits. Perhaps it's different using statically-typed Scala in SBT, or Kotlin in Gradle 3.
What subset of Scala? what libraries? what build system? apparently if things worked well, there would be at just one true - and well respected - answer for these.
So now I ask, honestly, why not go with Kotlin (https://kotlinlang.org/)? a language that IMHO didn't try to rule the world from day one (a Good Thing), but scores high on aesthetics, abstraction, power, IDE, compilation speed, and Java integration.
I have been following (and playing with) Scala since a bit before the Yammer thing exploded, I've been following Odersky as a student listening to the OOPSLA podcasts, and I never got to convince myself Scala is worth it. Can someone convince me to still use Scala when Kotlin is around?
The domain dictated that we needed a 'heavy lifting' framework like .NET or the JVM.
We did also work with Ruby, nodejs, and Go for certain microservices.
So now, decision making time:
- Do you reimplement in .NET ? - Do you live with both .NET and JVM bases? - Do you migrate existing systems from .NET to Java?
The .NET system was aging in any case, people needed a fresh breath of air, and it was nice to get rid of all of the Microsoft licensing and closed garden. It is then when I decided the new-gen stack would be Java8 and Kotlin, side-by-side. Kotlin was (and still is) amazing.
You mean the horrors of MIT[0] license?
[0] - https://github.com/Microsoft/dotnet/blob/master/LICENSE
Since when has that ever been true of any language where it wasn't dictated from above as a requirement, or people weren't funneled through norms and making alternate methods at a minimum hard (e.g. python)?
Expecting there to be one true and respected answer is to assume that you're either entering the ecosystem at a point where things are temporarily stable, or the ecosystem is stagnant, or someone has developed perfection and no advances can be made. CS and technologies and tooling hasn't stopped advancing, so I'm not sure why you would expect any particular language to have, or see that as a good thing.
I probably can't, but several things about Kotlin have annoyed me. The main thing, coming from Scala, is lack of library-based Option as opposed to compiler null-checking enforcement. Sure Kotlin talks about the lack of overhead compared to an Option instance (or Optional in the Java 8 case) but the fact that I have to use imperative tools (if/else, etc) to do some of the functional things I might want to do (flatMap, treat as single-element collection, etc) drives me nuts.
Also, for the longest time I couldn't even have a sealed data class hierarchy (i.e. ADTs) but I think that is being remedied. I have also run into bugs where their insistence on limited operator overloads causes ambiguity with overload resolution [0]. I could go on and on having written both a lot (but less Kotlin lately) but not sure this is the best forum for comparing. I might not convince you that Kotlin's downsides compared to Scala outweigh its upsides, but we can't pretend downsides don't exist.
Most on our team would consider programming in Kotlin a huge regression, as it's not really geared for functional programming.
Here's a previous post where I talk about the pros and cons:
Kotlin spends a lot of time promising things and talking about things they plan to ship – and then never ship it.
Scala devs are honest, tell you exactly what's planned and then ship it. Tooling is great, the library system is amazing, and people get things done.
If you are happy with Java, keep using it (or try Kotlin – which is nothing more than a slightly less verbose Java). If you want a better language–use Scala.
I have a question regarding slides 29 to 31.
The author says an untyped tree is represented as Tree[Nothing]. This leads to undesired variance behavior which is in turn overriden.
But there cannot be any instances of type Tree[Nothing] because there are no instances of type Nothing. Wouldn't an untyped tree be better represented as Tree[Unit]?
class Tree[T] { def elem: T }
Basically, Tree[Nothing] allows you to indicate that a Tree is empty via the type system while still allowing you to use that same type to build an actual tree from. If Nil was List[Unit] instead of List[Nothing], 1 :: Nil would be a compile time error instead of a List[Int].
Oh, and Unit does have an implementation in scala called "()", so if you had a Tree[Unit] you would not be able to tell if it was actually empty or not. You can see this in scala with () :: Nil. It produces a non-empty List[Unit]. Nothing has no such implementation. As for the exceptions, there's not much you can do wrt that. What should happen if you call head on an empty List? That's a straight-out error. headOption or a match block is the safe way to deal with lists if you don't want to have exceptions ever.
The good thing is that Scala is a typed language, so they can
a) provide tooling for migrating code
b) compile things and get clear information about how much differences there are
I think somebody said that necessary lines of change from Scala 2 -> Scala 3 should be below 0.1% to be acceptable.
But Python changed Strings to use Unicode, and that's something everybody uses and that is insanely hard to refactor automatically even with types.
For Dotty, instead, there are some hard-to-refactor-for changes, but they're in really advanced Scala things, so I expect they'll affect fewer users — mostly the kind of users who can write shapeless-like stuff.
Also xml support as the scala-xml library will continue. Though as the xml literals are removed you have to use string interpolation (xml"<your><xml/>here</your>") which isn't as nice to use. (hopefully someone comes up with a nice DSL or something instead)
[1] https://www.youtube.com/watch?v=_2oGY8l67jk
[2] http://www.slideshare.net/Odersky/scala-days-nyc-2016 (slide 21)
(slide 52)
Looking forward to their solution.
I hope Swift will eventually stop piling up features and will try to get back to the right fundamentals too.)
> Kotlin is a pragmatic programming language for JVM and Android that combines OO and functional features and is focused on interoperability, safety, clarity and tooling support.
> Being a general-purpose language, Kotlin works everywhere where Java works: server-side applications, mobile applications (Android), desktop applications.
Extension methods, properties, special syntax for everything, final by default, short constructor syntax, trying to put band-aid around Java's broken collection types, ...
"Android Studio should currently not be used for development with sbt and Scala. Chances are, however, that it's a good fit for a Gradle + Scala setup."
http://scala-on-android.taig.io/editor/android-studio/
Using InteliJ instead of Android Studio means not being able to use the latest tooling even from the stable version, as InteliJ is always some versions behind.
Using Gradle + Scala implies a few configuration steps that tend to break at each new Android Studio release, versus the out of the box experience from Kotlin.
Which issues did you experience with Android Studio?
I expect no more hurdles than just using Java official Android tools.
Ceylon is what you should be looking at.
I thought, that's a big claim -- then saw who is working on it.
A lot of very smart people seem to be very excited about that, I can't help but find it ominous.
Will it break a lot of codebases? Yes, it will.
There's a tradeoff between breakage and improvement, but I'd like not to wait 10 years before I can use dotty (like python 2 to python 3). We can always have another dotty in 5 years time for another round of rationalisation/simplification.
No, it means that person is not working only on Scala, or rather the current implementation of Scala. People can work on multiple projects over the course of a year/years.
> on a language that will "eventually" become the future of Scala
You changed "Is it the future Scala? Yes, it will be - eventually" (I think can best be reworded "eventually be the future Scala") to "eventually become the future of Scala." To me, those have different connotations. The former means that it's not finished yet, but when it is it will be the base of Scala at that time. The latter implies, to me, that at some time in the future this will be the base of Scala at some time further in the future.
> which means that, right now, dotty is not the future of Scala
And here's where that different meaning leads you wrong. dotty is the future of Scala right now, but it's not that future Scala yet.
> A lot of very smart people seem to be very excited about that, I can't help but find it ominous.
People are generally happy to have to the creator of something they like working on its future.
Scala is being maintained. Improvements and releases are being made in the meantime. More than one person can work on Scala, and a single person can work on multiple aspects of Scala. Big changes like dotty are being carefully planned and executed at the same time other changes are made. People are excited that Scala is continuing to evolve and becoming a better language, and that a beloved innovator/creator is playing a significant role in that.
On the other hand, I stand by my "not working on Scala" comment. This is based on commit history, which I admit might not give the whole picture but is certainly a fairly strong indication: https://github.com/scala/scala/graphs/contributors Odersky's activity has drastically diminished towards the end of 2012, with his last Scala commit in July, 2014.
I've no issue with him doing exactly what he wishes, he owes the community (or at least, me) absolutely nothing and if he's more excited to work on Dotty than on Scala, then there is no doubt that's what he should be doing. My concern is that the more I look at Dotty, the more it looks like a very promising language that looks a lot like Scala, but that adds and removes some features to it - that is to say, not actually Scala. I've no trouble imagining a future where the differences grow enough that they stay / become entirely distinct, and seeing a large chunk of the brains behind Scala busy on Dotty doesn't feel me with confidence in the former's future.
I'm not saying Scala is in bad hands or a dead language. But when a project's creator looses interest and moves to different things, well, it's usually not the best sign for that project. And again, I'm not saying that this is happening right now, or even that it necessarily will, just that Dotty definitely has the potential of leading to that situation.
"looses interest" ... "moves to different things" ... WTF? How can people come up with this FUD?
Scala2 and Dotty are two dialects of the same language, just as ScalaJS and Scala Native are dialects of the language.
Keep calm and carry on.
What I said, and mean as my opinion and not an absolute truth, is that a non-insignificant chunk of the brains that used to work on Scala now seems to be busy on Dotty, and that the resulting work, amazing though it may be, might not make it back to Scala.
I have nothing to back this opinion up. Everybody involved with Dotty seems absolutely committed to it being the future Scala, and I've no reason to believe they're not in earnest. But I can't take the switch for granted until it's happened.
from the linked thread that takes about 1/100 of the time to read as you've written casting doubt on the Scala to Dotty transition ;-)
The first is that the linked post answers a lot of points, but not the ones I made. It makes Dotty sounds really exciting, which it is. It makes it sound as if everybody working on Dotty believes it to be the future of Scala, and I believe that to be true, too. What it does not do (that I could see) is give reassurance that Dotty and Scala will not drift appart and become two similar but incompatible languages. I felt it'd be rude to point this out, since there really is no way to provide such guarantees (just like there really is no way for me to back up my worries with anything concrete).
The second part is, I'm not casting doubts, but expressing worries. I'm not saying it will happen, I fear it might. I'm sorry if this offends - I really am, my goal is not to insult anyone or demean anyone's work. I have no hidden agenda, and I'm not trying to be one of these anti-Scala trolls that sometime crop up. I love Scala. Look at my github profile, most of my public work is in Scala, and some of my libraries represent a lot of work (which obviously does not mean they're actually any good). I really, really hope everything works out the way people mean it to - as I said, Dotty sounds really exciting. I've seen too many "let's rewrite this from scratch and then make it backward compatible, it'll be amazing" projects to be able to believe it blindly, though.
We take continuity very seriously. No one wants a Python 3-style transition. In addition to the desire to avoid this, we also have a type system and a community build (think Google Blaze for Scala) that builds > 1MLoC OSS Scala code.
What I fear in that moment down the line where one feature goes over the arbitrary threshold you set yourself for community breakage, but is just too great to pass up on. Since the threshold is arbitrary anyway, it's fine to adjust it a bit... and a bit more later...
Not saying it will happen, and it would be dishonest of me not to say that I don't think you could possibly do more than you're already doing to prevent this kind of issue.
Scala 2.12, the next version that's currently at milestone 4, is going to be a huge release, having a new backend, optimizations, Java 8 interop, improvements for the type system, refactorings, basically the real deal.
But Scala needs to be production quality and stable enough for real work TM. Like when they were pushing experiments in Scala, people were complaining that Scala breaks too often.
Apparently now people are afraid that the experiments have moved to Dotty. You can't please everybody.
I'm not afraid that the experiments have moved to Dotty. That's an ideal scenario, if successful ones are indeed ported to Scala. I'm just not convinced they will be.
It's easy to verify for yourself who's working on Scala right now: https://github.com/scala/scala/graphs/contributors (work on Dotty started around 2012, when I moved from EPFL to Typesafe).
Myself, retronym & lrytz are on the Scala team at Lightbend. I'm proud of what we've accomplished with 2.10, 2.11 (8 minor releases!) and 2.12 (RC1 coming soon -- see my comment above), and excited about what the future holds for Scala! We work together closely with Martin's Dotty team, exchanging ideas about compiler performance and convergence of language features in Scala 2 and Dotty (the incubator for the future of Scala).
Compilers are huge beasts, but good ones are worthwhile, and Dotty is a better Scala compiler, so this criticism seems misdirected.