Typelevel Scala and the future of the Scala ecosystem | Hacker News Reader

Typelevel Scala and the future of the Scala ecosystem

typelevel.org

135 pointsLiveTheDream76 comments

Background: I saw Miles' tweet on this last week but didn't follow the apparent brouhaha that followed. All I know of him was his (hilarious) talk at Scala Days in June.

As someone who uses Scala daily, I don't think this is a good thing. I fall squarely into the Typesafe side of the camp—we don't use Scalaz or Shapeless or much of anything from the Typelevel project but Akka and Play are core technologies for us. We don't have a Typesafe subscription, we just use Akka and Play, a lot.

> As I observed earlier, there are already several variants of the language in existence and it has been clear for a long time that different sections of the community have different interests.

I don't think the author's examples really hold water. Nobody is using the Dotty compiler commercially, Scala.js is still in its infancy and while I know large organizations have at times run on minor Scala forks (I think Twitter used to but is now on 2.10.4), I'm not sure I believe that this is a major constituency. The official Scala is the Scala.

If this fork gains steam, it's going to be a huge opportunity for FUD around Scala and will hurt its adoptability, especially if existing Scala teams now have to choose sides.

That said, if the Haskell side of Scalaland wants to hang out somewhere else, that might be good too. Too many Scala questions on SO are answered (and upvoted) with obscure Scalaz solutions that make no sense to someone coming in from Ruby or Java.

I like functional programming. I like how monads can simplify things. But I also have to write code that someone without a lot of Scala (or Haskell) experience can maintain.

> I like functional programming. I like how monads can simplify things. But I also have to write code that someone without a lot of Scala (or Haskell) experience can maintain.

I agree with this, but I think a same replacement for the type lambda syntax goes a long way towards making those scalaz solutions maintainable.

Twitter has not used a fork of the scala compiler/library since the pre-2.8 days (and maybe not even then, since we have always had a fair fraction of open source code.)

To people worried about fragmentation, retaining binary compatibility means it really doesn't matter. Scalaz can be written in the Typelevel fork and you can use it without issue from code compiled with the standard Scala compiler.

Also, from the announcement:

"As a conservative fork, our aim is to be merge compatible from the Typesafe Scala compiler, and a key objective is to feed pull requests (many of which will resolve long standing Scala bugs or feature requests) back to the Typesafe Scala compiler."

So we aren't looking at radical changes, and hopefully these changes will get into Typesafe's compiler.

So basically it's an opt-in better Scala compiler with no effect on compatibility. What's not to like?

So the elitist community component of Scala that is constantly trying to turn it into Haskell has decided they need an island of their own. Great, they can have it. Now I'll know where to go when I need monadic and monoidal libraries for serving HTTP requests using nothing but type level hacks.

In all seriousness though I don't really know what this is going to accomplish. The article mentions the academic and non-academic components of Scala and it is clear the author falls on the academic side but if that's the case then why not just go and hack on something like Haskell (or Coq), a perfectly unapologetic academic language that has tried to avoid success at all costs. If push comes to shove I'm with team Odersky.

>Now I'll know where to go when I need monadic and monoidal libraries for serving HTTP requests using nothing but type level hacks.

Monadic libraries with classes like Option, Future, and List? Used in elitist features like for comprehensions? Anyone who writes Scala on a regular basis deals with monads constantly. Why not take 15 minutes and learn how they work?

Oh no that is all a little too pedestrian for me. I've moved my sights much higher now. These days I use template haskell and applicatives to generate HTML forms: https://github.com/yesodweb/yesod/wiki/Multi-select-field-po....

I'm all for applying theory to enrich the practice of programming but lets be reasonable here. Lists and Options existed way before anyone added monadic sugar on top of it as did mapping and filtering collections. I like category theory as much as the next guy but lets just pull back a little bit. Not everything needs to be point-free, functorial, and shapeless.

>I'm all for applying theory to enrich the practice of programming but lets be reasonable here. Lists and Options existed way before anyone added monadic sugar on top of it. I like category theory as much as the next guy but lets just pull back a little bit. Not having needs to be point-free and functorial.

I agree that Spire/Shapeless/Scalaz tend to go far beyond anything I would use in a production app, but Options/Futures/etc are barely usable without for comprehensions, which are just syntactic sugar for flatMap. I see the ability to write

  for {
    x <- f1() // returns Option[T1]
    y <- f2(x) // returns Option[T2]
    z <- f3(y, x) // returns Option[T3]
  } yield foo(x, y, z)  // for expression returns Option[T4]

as a huge part of Scala's success

Sure that's nice and so is LINQ but there are thousands of programmers using LINQ without worrying about monads. It's like I said, I really don't have any issues with applying theory to enrich the practice of programming and the new crop of languages that are more fully grounded in theory is really a breath of fresh air. The problem I have with some people that enjoy these type level hacks is that they are not in it to build things that thousands of programmers can use to make their lives easier. They're usually in it to build monuments to their own egos. Forking the language is just another instance of that hubris. They're basically saying what Odersky and friends are doing is not good enough even though Odersky and friends have poured thousands of hours into making it something that in my opinion is actually something good.

If this was something that was driven by anything other than ego then they'd look at library popularity and contribute fixes to the most used and valuable libraries. Instead, they've decided to fork the language because 3 libraries, one of which is the author's own, don't have enough syntactic sugar to express certain type level constraints. Does that sound reasonable to you?

I think it is misguided to call this an act of ego, like Miles says in the announcement Typesafe has different goals, and milestones that don't necessarily serve the whole community. There are actually tons of nice small changes that haven't been implemented for various reasons, and I see this as an incubator where people can try out experimental extensions of the language and reasonably package them together, ideally porting them (if they are good) back to scalac itself. This isn't much different than people reimplementing Python or Ruby, or adding experimental extensions to Haskell (go look at how many different Haskell compilers there are). Personally I'm always interested to see what directions people can push software in when not motivated by money, or publications.

I think it is also important to remember that these people work on stuff for free in their free time because they enjoy it. They are free to pick designs and approaches they like, and have zero obligation to contribute to popular libraries, regardless of their supposed merit.

How many people write anything in Python or Ruby that is not compatible with CPython and MRI Ruby? The lowest common denominators in those communities are still the reference implementations and any production grade library conforms to that lowest common denominator. As for the Haskell compilers, I'd like to hear from someone that uses anything other than GHC in production.

I agree they can work on whatever they want but this is a bit less benign than writing another library that no one is going to care about. Language forks have never been successful. Please show me one example where a language fork has succeeded and I'll concede. This is a whole lot of drama that is going to turn more people away from Scala which in my opinion is one of the better languages that runs on the JVM.

Haskell's actually got a pretty great (if informal) standard which has been used to write a number of different compliant Haskell compilers. GHC is obviously the most popular open source one, but Standard Chartered Bank has its own Haskell compiler it uses a great deal of in production, for instance.

What is the point of a proprietary, in-house, custom compiler?

The Excel backend of course.

> How many people write anything in Python or Ruby that is not compatible with CPython and MRI Ruby? The lowest common denominators in those communities are still the reference implementations and any production grade library conforms to that lowest common denominator.

If the library is binary-compatible with that reference implementation (as the typelevel guys say they will be), why would you as a user care which compiler was used to compile it? A lot of typelevel code already uses macros or compiler plugins to generate source (I've even heard that at one point the Shapeless code was generated using Freemarker templates) - but as users we just depend on their released jars and don't have to worry about any of that.

Why all the hoopla then? Just go ahead and do it. Why make a public stand and generate so much drama?

We've tried to avoid creating a "drama"; that's not the intention at all. But we want to attract other developers to our experimental fork, so that if you want to use new and exciting features, or contribute your own, you don't have the entire burden of maintaining your own fork. The hope is that the Typelevel fork will gain enough critical mass to encourage people to develop new language features where they previously wouldn't have bothered, due to the high costs involved (in terms of maintenance) and limited returns. It really is meant to be a community effort.

a) It's a useful project for other library authors; of course they want to announce it.

b) I think the part tucked away at the end about a foundation is actually the most important part. Scala's currently owned by one university and also partially by typesafe, and while on the whole they've done a good job there are a lot of stakeholders who aren't really represented there. For a language that so many different organizations are depending on, I think it's time there was something a little more formal and dependable; many other languages have similar foundations.

Who cares how large their egos are if they're right?

Edit: to elaborate, you're mostly arguing about the personalities involved, which I consider irrelevant. Convince me that they're wrong based on the merits of their proposal.

Right about what? That their precious libraries don't have enough syntactic support in the current Scala toolchain? I mean sure I guess they're right in that sense.

My argument is not just about the personalities involved but I admit that is a large part of why this leaves a sour taste in my mouth. I think these libraries are a really nice way to bone up on theory and see how certain abstract concepts get expressed in something that is computable. None of that is the issue. The issue is that there is now going to be an unnecessary choice involved in what toolchain you are going to use even though the choice is going to be completely irrelevant. The enterprise guys are not going to use anything that is not backed by service/stability guarantees and the group that is going to use the alternate toolchain is potentially going to diverge more and more from the stable branch. It just feels like a whole bunch of wasted effort. There are many examples of this happening already. The biggest example being D and the fork of the standard library in the early days. The other examples are python2 vs python3 and perl5 vs perl6. Going by historical examples this is going to be a complete and utter failure.

Don't focus too much on the relevance of the Typelevel fork to the Typelevel libraries. The people involved (myself excepted) work on the Typelevel libraries, and consequently have particular interest in the features they benefit most from. But the fork, and the infrastructure around it, is for the benefit of the entire community, not just Typelevel. We'll happily entertain PRs for any useful language feature, provided it meets the compatibility requirements.

Counter examples Egcs xorg Oderskys Java compiler

My biggest frustration with Scala is the exact opposite of their frustration with Typesafe's Scala: it moves too fast. Scala's backwards compatibility problems are obnoxious for the people who don't feel the need to reinvent everything. I can find a version of Foo that works with 2.11 but the only version of Bar I can use only has artifacts for <=2.10.

The play framework exemplifies these problems. Documentation doesn't keep up with releases, blog posts and tutorials don't either. Sometimes I'm looking for example code for some plugin or library, and the code only works with some other version of another dependency that I can't use. I totally understand that people prefer Agile development, but you can't combine Agile development with non-Agile documentation, unless you want to create a framework that only experts can use.

You are misunderstanding. The issue you have observed is caused by binary incompatibility between Scala versions. Java has an almost one-to-one mapping to JVM bytecode. Scala, having many features that Java does not, does not have such a direct mapping. Sometimes the way Scala is compiled to JVM bytecode changes, to fix bugs or to improve performance (e.g. compilation of lambdas will change now that Java 8 supports them). This leads to binary incompatibility between Scala versions, though the problem is decreasing with time.

This effort is aiming to keep binary compatibility, so it will have no effect of users of Typesafe's compiler. To quote: "programs which are valid when compiled with the Typesafe Scala compiler would have the same meaning and binary representation when compiled with the fixes in place."

Yeah I've wasted so much time dealing with code for the wrong version of Play. It's horrendously organised: case in point, the JSON parsing (WTF?)

The documentation for JSON parsing in Play is hilarious.

You read the "The Play JSON library Basics" document, it looks like a lot of code for something that should be simple, but you go ahead and implement as explained. After a while, the verbosity starts bugging you, so you read the "JSON Reads/Writes/Format Combinators", followed by "JSON transformers" and finally the "JSON Macro Inception" pages.

The punchline in the last section can be summarized as "forget all that other stuff, just do "implicit val personFmt = Json.format[Person]". I was happy to be able to delete hundreds of lines of code from my project after learning this, but couldn't you just have told me this upfront since it probably covers 99% of all cases? The documentation also has an aura of "Isn't this all very clever?" feeling to it. It's also quite hard to search for anything, as outdated versions of the docs keep popping up in Google as the first results.

Look at this one: http://stackoverflow.com/questions/1722726/is-the-scala-2-8-...

Scala has a lot of really clever stuff and the "isn't this all really clever" attitude is definitely there.

EDIT: Come on. I meant clever in a good way as well. I'm just a bit ambivalent, is all.

I think if anything that example shows the cleverness is a good thing. It's from almost 5 years ago, since when Scala has gone from strength to strength, so the collections rewrite turned out quite well, and fears that it would stop new people coming to Scala, or that companies would stay on 2.7 and the community would schism, were definitely unfounded. The "use case" simplification of the documentation has been implemented and has shown itself to work well, suggesting that this kind of complexity can be managed. And a few years later I've seen Chris use equally complex techniques in his own code (check out his "practical scalaz" talk).

Yeah, I think there's a strong possibility that the Play JSON docs constitute a much more worrisome suicide letter.

I had this exact same thing experience, thinking, "wow, they want me to write marshal/unmarshal code for every type... is it 1999?" and then, finally, stumbling across the horrifically written "JSON Macro Inception" articles--as if no one had ever used reflection before for serialization--and feeling very, very stabby.

And lets not talk about all the wrong/missing import statements in code examples...

The documentation also has an aura of "Isn't this all very clever?"

This attitude needs to die. Forever. Why the fuck should I need know about contravariance and parser combinators to write a basic web app? I hope this Scala forking will kill it in Typesafe. It has no place in actually building products.

I notice a lot of difference in quality between libraries released by enterprises and those by users. It's the difference between https://www.paypal-engineering.com/2014/02/13/hello-newman-a... and Dispatch. We need way more of the former.

Wait, so it can figure out case classes on it's own?

Yes. The implementation uses a macro, which some people have (sometimes legitimate) concerns about, but you can tell it to make a format for a case class and it will Just Work.

We've been bitten by this as well. We started several projects within a few months of each that also used what we believed to be similar enough versions of Play but there are just enough inconsistencies between versions of play (and Scala language versions as well) to make it exceedingly frustrating to work with.

I'm conflicted. Type constructor partial application would be great (the current type lambda syntax is just awful), but I really don't want to see Scala fragmented.

Agreed on conflicted-ness.

The thing that concerns me the most is that from observation it feels like many of the high muckity mucks of the Scala world would probably be heading in this direction. There's nothing inherently wrong with that, but IMO that increases the chance of serious fragmentation.

I think it's fantastic news - even if the net result is just forcing typesafe to address concerns about scala that have been festering for some time.

I'm worried about this fork. Scala is far away from being a wildly used industry language. Yes, there are some well known companies using Scala, but how many of them develop a really big part of their software with Scala? Not so much. If you look around you'll see Java, C#, C, C++ (depending on the business) and actually not so much Scala. The ones who think forking Scala does help the community are in a reality distortion field.

Take a look and Linux desktops to see that diversity is not always good, especially if you're in a niche.

I think there is a lot of Scala usage you perhaps don't see. As an example from the corporate world, at least six major international banks (probably more) are doing a significant amount of their development in Scala, some with teams in the hundreds. That's a solid market for Typesafe right there, which isn't going to go away overnight, and won't be stirred by the existence of the Typelevel fork of Scala. Furthermore, they should benefit from its existence because it represents a means for more people to be able to contribute to and enhance the language, without negatively impacting the stable, reliable releases from Typesafe.

What's the history behind this fork and what is it likely to mean in terms of fragmentation? In particular what are the irreconcilable beliefs on each side that have led to this?

I haven't tried Scala yet, but my gut reaction is I should stick with Clojure until the dust settles. A fragmented community around a relatively niche language is never good.

As someone who's been using Scala for the better part of a couple years:

This seems more like a more ambitious version of the Shapeless and ScalaZ libraries (with some of the same crowd/authors).

I (and I suspect most people who program in Scala for a living) don't use those libraries. Not because they're in any way bad. But there are trade-offs using them IME. The biggest being compile-time. So I get by fine without.

I think this is probably generally good news. In the same way that (I believe) JRuby pushed MRI forward and popularized the case for a multi-threading VM, more advanced type features in this fork probably benefit the Scala community as a whole.

I certainly don't see how it could be harmful. Typesafe isn't going away. The vast majority of businesses will still use their compiler. If you didn't use the advanced-type-based-libraries before, this news probably isn't going to impact you anyway.

So from where I'm standing, having made a late 2.9 to 2.10 transition, and now having made several 2.10 to 2.11 transitions, Scala seems a very mature language/platform to me. Libraries are generally upgraded fairly quickly, and source wise, those releases broke very little (2.11 being the biggest culprit just because of the modularization effort, but that's pretty trivial to resolve).

I dunno. It's not Java. But it's not Ruby either. It's gotten new features since 2.9, but it's broken or changed very little. I find working with it, even working through upgrades, very stable and mostly frustration free.

As someone involved in the scala community (and scalaz) community, I'd say about 60% of the companies I talk to at the various meetups use at the very least #scalaz. I was pleasantly surprised.

Could this be a biased sample though? Not saying this is the case but it seems logical that the people at the scala meetups are cut from a cloth that would make them more likely to be using things such as scalaz.

I know that my local scala meetup seems to be more heavily oriented towards the functional end of things than the bulk of the scala developers I interact with. OTOH, perhaps the bias is with my sample.

The fork means nothing dramatic. I'll make a short summary of both positions:

Typesafe: we need to offer our clients a good guarantee on stability and compatibility with Java 8 or they won't take Scala seriously. So, for the moment we won't introduce new features or make big changes in the compiler.

Typelevel: we are OK with compatibility and stability, but there are some features that we really want which either don't exist or are a bit quirky with the current compiler. So we will make a fork in which we can add these things while staying binary and merge compatible with yours.

If they make it work, this is great news. There wouldn't be fragmentation as "bleeding edge Scala" would be binary and merge compatible with "stable Scala". And nobody should have hurt feelings, both positions are completely sensible.

It's not a niche language. Household names including Twitter, Meetup, Foursquare, and Coursera run on Scala.

According to Cedric Beust, it's still a language that "pretty much hardly anyone uses". http://beust.com/weblog/2014/05/11/more-about-language-popul...

Yes, let's cite this guy who is "so uninterested in Scala" that he spends an considerable amount of time maintaining multiple user accounts to spread FUD about the langauge in various places and makes up stories about how "his team/group/company/... was trying to use Scala but abandoned it due to all the horrible issues with it".

You know that he is really persistent (despite claiming that he doesn't care about Scala) when the community starts maintaining a list of his sockpuppet accounts.

Anyone who tries to make the claim that "pretty much hardly anyone uses" is standing on shaky ground. It definitely has not surpassed Java in popularity but it seems to be quickly gaining traction at lots of large companies and is being used more and more. I was recently at Scala by the Bay in San Fransisco and it seems each year more and more companies are on board. If LinkedIn, Twitter, PayPal, Cisco, Verizon, and more are all using Scala I would hardly call it a niche language anymore.

There are a few really big and impactful things already implemented in Scala, and some companies may be using it without even actually writing any Scala code (for they provide Java APIs): Play, Kafka, Spark, Akka.

By analogy its not terribly far off from Debian issuing a "testing" branch off "unstable" a couple years or decade ago.

Its a little more extreme than that, but not much.

Usually forks are more dramatic. This isn't MariaDB or something like that.

For PR reasons they probably should have stolen the Debian "testing" vs "unstable" names instead of getting all forky which is scarier sounding.

>>In particular what are the irreconcilable beliefs on each side that have led to this?

I'm not familiar with this one. But in general most of these forks are generally out of personality clashes where one person just refuses to take another as the boss.

Scala is still not among the top languages used in the industry. This kind of my way or the highway attitude will only fragment the already fragile ecosystem. Imagine any serious software shop wanting to use Scala reading this. If I want to be writing an enterprise application which I plan to keep running for the next decade, should I be choosing a language which isn't just not widely adopted, but the existing ecosystem itself has forks because a few members don't give two hoots about stability(Which is far more important to 99% of the software shops out there, than adding new features).

Instead of doing something like this, and putting your broader community of users in perpetual confusion as to which ecosystem to use, you could rather contribute to something meaningful.

Sometimes it's useful to read the blog post before commenting.

I did, and its the same template story always. We don't agree with technical decisions made by Set A people to be in best interests of the community, therefore we plan to fork Y, with intelligent Set B people.

The actual story is always about control and ego.

You really didn't read it.

> What's the history behind this fork and what is it likely to mean in terms of fragmentation? In particular what are the irreconcilable beliefs on each side that have led to this?

What I've managed to put together, corrections welcome:

Typesafe has been moving to stabilize Scala, primarily to make it more attractive to enterprise clients. Prior to 2.10, even point releases broke binary compatibility; now compatibility is maintained for those and they're making efforts to make minor releases binary compatible. The tooling they've produced and the testing framework that builds all the major libraries in the ecosystem and checks for binary compatibility means they now have quite possibly the best binary compatibility regime of any language.

Typesafe recently published a roadmap for the next few releases, saying that the next release would focus exclusively on Java 8 support and remain compatible with the current release. This means work to merge any new features won't even start until January 2016, and those features are unlikely to be in an official release until 2017.

Less frequent and more compatible releases are what a lot of people have been asking for, and it would be very awkward to maintain Java 8 and pre-Java 8 releases of Scala. So the typesafe position that there can be no new features in the release that introduces Java 8 (so that people will definitely be able to switch to Java 8 without making any code changes) makes a certain amount of sense. At the same time it's meant Scala's getting "frozen" in a bit of a weird state; macros were a recently-introduced experimental feature and the official version is kind of half-finished (there's a branch with a more complete version). And there are a couple of features that are quite small and simple but would be a tremendous help for the kind of code the typelevel guys work on (e.g. partial application of types would be a very simple syntactic rewrite, and there's already an implementation as a compiler plugin), and there's long been a feeling that they were coming as soon as we figured out the little details - so to then be told that we won't see these features until 2017 at the earliest was a bit upsetting.

So there was a bit of consternation among the typelevel folks - a group who maintain some quite abstract and powerful libraries that tend to push the type system a bit further than most user code. Some of them started seriously talking about porting their libraries to one of the smaller, more experimental languages with a more elegant type system (Idris, Ermine, or I forget the other option). At the same time, the Scala community now has a lot of expertise in maintaining binary compatibility across quite complex changes, so at some point Miles came up with this idea. I don't think a fork would have happened otherwise - most of the goals can be achieved by compiler plugins (at the cost of being harder to maintain), and if they were going to break compatibility they might as well have jumped all the way to another language. But if the typelevel guys can pull off this "conservative fork", where they'll use their own compiler to compile their libraries but those libraries will then be binary compatible with "vanilla" scala and usable by ordinary scala users, then they get the best of both worlds. It's ambitious, but I hope it works out.

> I haven't tried Scala yet, but my gut reaction is I should stick with Clojure until the dust settles. A fragmented community around a relatively niche language is never good.

If they're maintaining binary compatibility then that should keep fragmentation to a minimum. Honestly I don't think this changes much compared to how it was with compiler plugins - if you're building some super-duper-typey project with these advanced features then maybe you'll muck around and use this forked compiler, but if you're a normal user then you can ignore it and carry on using their libraries without noticing any different.

Plus even if the Scala community were to split in two, I think either half would be bigger than the Clojure community.

> Typesafe has been moving to stabilize Scala, primarily to make it more attractive to enterprise clients.

I work primarily for startups and have been using Scala for the last 3 years. Typesafe is entirely in the right to want to deliver a more stable Scala, not just for some mythical enterprise, but for all people using Scala for real-world projects. And this is because people complained about the lack of stability. This isn't just because Typesafe has products to sell, it's also because if you want a language to succeed, then it needs to cater to real world concerns.

Lets compare the situation with Clojure for example - with which you can usually use a library created for Clojure 1.1 and that hasn't been updated in years and that's still usable on top of the latest Clojure version.

Personally I'm happy that this fork happens, because:

     1. if it adds any good features, they'll be merged, 
        and that's what open-source is about

     2. if it fails, then there's no harm done

On the other hand I do not have faith in the ability of the Typelevel people to pull this off. Like every time I tried playing with Scalaz, I ended up with stack-overflow errors - not to diminish their skill or efforts, as yes, they are amongst the very best developers, but these people are too idealistic to address the needs of projects that solve real world problems.

Miles himself speaks with contempt about Play and Akka and their Java compatibility - but other project leaders have a lot to learn from these projects, as they provide a blueprint for how to achieve success - a documentation first effort, reasonable stability, gradual learning curve, a sane deprecation process, mindfulness about the platform they run on and so on. Compare with Lift. Or Scalaz.

Ok, time for some personal opinions. I agree that binary compatibility is important, and that Typesafe's recent focus on stability has been correct. I do however think that the planned focus on Java 8 to the exclusion of all else is misguided.

* Scala on Android users are very worried by this direction, because Java 8 doesn't and won't run on Android; Typesafe has not put forward a good narrative about Android. If Typesafe is simply abandoning any support for Android that's... possibly a reasonable commercial decision, but rather upsetting for those of us who were using it. Either way, more clarity would be better.

* Scala users, including users who use Java libraries in Scala, are mostly unconcerned about what bytecode their code compiles to - if it does the same thing either way then it doesn't really matter whether it's Java 7 or Java 8. Performance and code size improvements are always welcome, but not worth freezing the language for 2 years for.

* The only users who really benefit from the Java 8 support are users who want to use Scala libraries in Java (since they want to be able to e.g. pass Java 8 lambdas as callbacks). This is evidently a constituency that's commercially important to Typesafe, since they put a lot of effort into making Play and Akka usable from Java. But I don't think this reflects the wider Scala ecosystem at all; most Scala libraries are written to be used from Scala, with little if any concern for Java support. IMO the primary (not exclusive, but primary) focus of the Scala compiler should be supporting Scala programmers (some of whom desperately want features like partial type application), not providing libraries for Java programmers.

* So who are the users who are actually benefiting from this? Only those who are both cutting-edge enough to be moving to Java 8 (and to use Akka/Play), but conservative enough to not want to use Scala for their code. That seems like a fairly small set to me.

I agree with the value of documentation and deprecation cycles, and that there are Typelevel projects which should make more effort in these directions.

> Like every time I tried playing with Scalaz, I ended up with stack-overflow errors - not to diminish their skill or efforts, as yes, they are amongst the very best developers, but these people are too idealistic to address the needs of projects that solve real world problems.

Huh? That's not a question of idealism, that's just a bug - and "practical" programmers introduce plenty of those. I've never hit a stack overflow in Scalaz FWIW, but sure, sometimes programs have bugs. I've found the typelevel projects to be relatively high-quality code though.

On Android - I agree it is worrisome.

On Java 8 - it's not just because of the purpose of interoperability, but also for the runtime improvements. Things like invokeDynamic, ConstantCallSite and others are not in Java 6. Upgrading the requirement means opening the door to performance improvements and is a real concern of many. How much it can be achieved in Scala I don't know, but these improvements in the JVM have made a dramatic difference for other JVM languages.

Going back to Android - it's not right for Android to hold back the Java ecosystem. I've been rooting for Google in their lawsuit with Oracle, but they should also get their act together and improve Android's runtime and toolchain to keep it up to date with the latest Java.

As this is not going to be a problem just for Scala developers - this is going to be a problem for the whole ecosystem, as Java libraries will eventually have good reasons to upgrade from Java 6 too. For example value classes are coming and possibly generic specialization for those value classes in the next versions (like 9 or 10, so it will happen, it's on the roadmap) and there are libraries that desperately want to avoid boxing. So what then? Will we stay with Java 6?

In fact I think this upgrade to Java 8 is sort of anti-enterprise. How many companies have you see with projects running on anything newer than Java 6? There are still companies running with Java 1.4 for that matter. Java 8 is bleeding edge and it will remain to be bleeding edge in 2016.

> For example value classes are coming and possibly generic specialization for those value classes in the next versions (like 9 or 10, so it will happen, it's on the roadmap) and there are libraries that desperately want to avoid boxing. So what then? Will we stay with Java 6?

Well, Scala already does value specialization even on Java 6, so that's less urgent. You are right of course, but I think most libraries will simply release new versions that require Java 8, rather than having an intermediate version with no changes except a Java 8 port. But I could be wrong.

> In fact I think this upgrade to Java 8 is sort of anti-enterprise. How many companies have you see with projects running on anything newer than Java 6? There are still companies running with Java 1.4 for that matter. Java 8 is bleeding edge and it will remain to be bleeding edge in 2016.

I agree - which makes the Typesafe stance rather perverse. Are there really going to be people in 2016 who want to move their Scala to 2.12 so they can call it from Java 8, but would be put off if 2.12 contained any new features? Who won't have anyone available to fix even the most minor of syntactic changes? (or not even that - Typelevel is talking about a compiler that accepts all valid Typesafe-scala programs, with all the new syntaxes being things that are currently errors). I really don't think including partially applied types, byte/short literals, or singleton types in Scala 2.12 would be a dealbreaker for anyone (though I guess Typesafe has to draw their line in the sand somewhere).

What I expect/hope will happen is that Typesafe will accept pull requests for those pieces and some fixing up of macros, which will be enough to make a Scala I'm happy using until 2017. Then this fork can either become a place for further experimentation like Meta, or quietly die; important libraries will be able to continue to support Typesafe Scala, Typesafe will have accepted the proven features that users want without opening the floodgates to every experimental piece under the sun, and the community can hang together.

I think the idea behind keeping 2.11/2.12 as similar as possible is to enable people to cross-build things more easily, which would drastically improve the life span of the last Java6-compatible release (2.11) until people have found better solutions.

(I can imagine that there will be generic translators from Java8 bytecode to Java6 bytecode, so why should they keep maintaining two backends, if they could maintain one and just leverage a common tool to do the rest?)

Well I have those hopes too - this can be a very positive thing, since it can attract more contributions to Scala.

After all, the Scala core probably has their plate full and could use some help. It's one thing to make a proposal for a feature to a group of developers with their hands full, it's quite another to submit a pull request saying "look, this is useful, people like it" - it changes the equation.

When I see all the various Scala compilers out there (Typesafe Scala compiler, LAMP/EPFL Dotty compiler, Scala.js, Scala Virtualized compiler, Scala.Meta system, and several research and private variants of the Scala compiler, IDEs own Scala variants) compared to the lone version of the Groovy one, I realize just how detrimental to the Groovy ecosystem was Codehaus Groovy's practise of ignoring creator James Strachan's JSR-241 standardization effort and redefining "Groovy" to mean the org.codehaus.groovy implementation only. At times it seemed they were scuttling others attempts to build a variant compiler, and even cannabalizing the annotation-based addon market like when they duplicated the Groovy++ static compilation "speed booster".

I'm somewhat surprised that everybody missed the fact that Scala 2.12 has more to it than just Java 8 compatibility and stability.

We're working on replacing the whole backend in scalac. We'll ship faster and easier to maintain optimizer that fixes lots of long-standing bugs in scalac. In fact, it will allows us to fix the second most voted issue ever: https://issues.scala-lang.org/browse/SI-4633?jql=project%20%...

More details: http://magarciaepfl.github.io/scala/

-- Grzegorz, member of Scala team at Typesafe

I don't understand what features they want to add beyond type lambda's, singleton types, and literals for bytes / shorts.

1. I understand if you spend all day making shapeless / scalaz the lack of sugar for type lambda's is annoying. Why not make a SIP to add some sugar there?

2. Singleton types would have use in ScalaJs as well for encoding. There is already a SIP for this.

3. Byte / Short literals, seem like again, a SIP could be made for the next release.

So all this boils down to getting SIPs into the language faster. So I think the real feature is not having to wait 3+ years for language improvements that significantly improve the QOL of some library writers.

That is a feature I would like as well. I really miss being able to open the mailing list and scan for exciting new features, Instead I know even if someone has a great idea, it'll be years before it sees the light of day. This sucks and I've moved onto to getting my neat idea -> wow I can use this in my language fix to ScalaJS[1].

I think (thought?) about Scala as ycombinator for language features. I see this enterprise mindset coming in now trying to tie down what Scala is and ultimately I fear that it will kill all the things that has made Scala popular in the first place. Pushing to a more infrequent release model might help enterprise customers but it absolutely crushes library developers in the vanguard working to make the Scala of tomorrow by fighting to get the language features of today. They need turn around on the improvements listed above and can't wait 4+ years to get them.

So does Scala take all the strength it has built up from acting as a feature incubator for the last 10 years and switch and lock down what works, it is in a prime position to do so? It has a virtual monopoly on it's space.

Or does Scala continue to act as a language feature incubator giving its coders early access to cutting edge CS idioms in a mainstream language. There are a lot of CS ideas waiting to get their shot.

Some people would see this situation and say Scala is facing a mutually exclusive set of outcomes, it must decide now. Nonsense I say! When confronted with tough choices I think of the famous Yogi Berra quote "When you see a fork in the road, take it."

I ask myself is there some way we can have our cake and eat it too? Can we go on trying out new features while still oferring stability to those willing to wait for it?

What if we moved to model of more frequently releases + less frequent releases? A controlled fork within the ecosystem, similar to what Ubuntu / Fedora does.

Publish 2 major short term releases a year that cutting edge library writers can target and build their libraries against. Vend these are properly tagged releases in maven central. Stick any old SIP in there, the enterprise overlords don't have to know about it. Rapid iteration, with rapid fall-off in support, a fun Scala for those of us that love using new features, bugs and all, for us the language is about growth.

Then have an LTS version released every other year that picks up the features that worked and everyone seems to love and that will be supported for years into the future. Relatively slow iteration with slow fall off in support, vend this Scala to enterprise users with a coherent multi-year support policy that just oozes with that 'this is rock solid and will work for eons to come' professional vibe that makes them happy.

1: I am using ScalaJS and man it feels great, it's like Scala of 6 years ago, I love opening the mailing list to see if there is a new release with some awesome new features / improvements. There is a lot of green territory over there because it's targeting a new platform and that will settle with time, but already it beats the socks off all the other competitors in the statically typed compiles to JS camp, just no one knows it yet.

+1 on ScalaJS. It's pretty amazing piece of technology (Thank's team!) using it in combination with David Barri's bindings for ReactJS[1] and Li Haoyi's uPickle[2]. It makes for elegant solutions with a shared codebase.

[1] https://github.com/japgolly/scalajs-react

[2] https://github.com/lihaoyi/upickle

do you know of any large apps using scalajs-react in production?

Speaking of byte/short literals, we can already have a syntax for them with string interpolation. In fact, that's what is already implemented in Spire: https://github.com/non/spire/blob/ac12fd02180e39d2d482217319....

> I ask myself is there some way we can have our cake and eat it too? Can we go on trying out new features while still oferring stability to those willing to wait for it?

Honestly, it seems like this is precisely that.

There's a general feeling in the community that the future of the Scala language and toolset is closely tied to the business goals of one specific company, at the expense of other members of the community. As long as the fork stays close to the existing mainline -- and it seems they intend to -- this seems like it should work out well for both sides: the community gets to fix some longstanding pain points with their tools, and Typesafe gets a fertile source of pre-battletested improvements to roll in to their slower-moving distribution.

Don't compiler plugins provide a way to provide a way to experiment with language changes more quickly?

Compiler plugins are powerful, but still too limited. We can't change parsing rules with a compiler plugin, for example. There are also potential compatibility issues with using some set of compiler plugins, and there's a performance penalty of implementing new features with plugins (which, by their very nature, are optional), rather than in the core compiler.

What kind of performance issues are you having in mind?

> I don't understand what features they want to add beyond type lambda's, singleton types, and literals for bytes / shorts.

We have more ideas on the table beyond these three, but Miles will go into more detail about these in his next blog post.

How is ScalaJS's output size lately?

170 KB to a few hundred KB for typical applications. That's will full optimizations which take a few seconds to apply.

The fast dev cycle version weighs between 1 and 2.5 MB.

Regardless of the outcome of this initiative (which has little chance of succeeding, but best luck to them), this is pretty bad for the Scala ecosystem overall. There's a general lack of confidence from the community toward Scala and Typesafe and this certainly contributes greatly to the general uncertainty about Scala's chances of success.

I guess it was only a matter of time until the "Scala will surely die!!! Really!!! This time it will go down!!!" crowd would show up ...

Reply on news.ycombinator.com