Scala's version fragility makes the enterprise argument near impossible | Hacker News Reader

Scala's version fragility makes the enterprise argument near impossible

lift.la

99 pointsLiveTheDream46 comments

This is the largest practical issue I've had with developing a Scala project.

One major cause of the problem seems to be that Scala is stuck with the "precompiled bytecode in JARs" model of software distribution; while this may be appropriate for Java (where core developers go through great pains to maintain compatibility between versions) it seems like a poor fit for Scala. Not only do the core library APIs change between versions, but the conventions used by the compiler to "link" Scala code together are also altered, for good reason.

An alternative distribution mechanism that's more resistant to these kinds of issues might make sense. Even just distributing source code and build files (like, say, Ruby) could be a better solution. Building dependencies might be slow at first, but centralized caching could address this. Another solution might be an intermediate representation (after type-checking and other compiler phases, but before "linking") that gives the Scala compiler the knowledge that's missing from compiled class files.

Even namespace-mangling or class-loader magic could be helpful here. Loading multiple versions of a single dependency should in theory be possible, since OSGi manages it.

Sadly, there seems to be a pretty large investment by the Scala community in the whole Java/Maven/JAR ecosystem. It'd take a concerted effort by many people to create a more robust solution.

I think you've hit the nail on the head, that having a centralized package manager tool/host, a la the haskell hackage / cabal-install or the like, would substantially simplify the problem, but because of desired JVM library compatibility this path isn't explored as much as might be ideal

Thanks for this comment, as not yet having got into Scala myself, I had to read between the lines to figure out what the problem was here. I thought "Ok, you don't have binary compatibility, so recompile your dependencies. Sure, it may extend your build time somewhat, but it shouldn't make it impossible to use dependencies or test pre-release builds..."

Oh, you don't _have_ the source? Well, there's your problem.

A good reminder of what some of us take for granted in our development environments, I guess.

You have the source, but the Maven model used by the whole Java ecosystem for distribution/management of packages doesn't work like that. And replacing it with something else isn't a small feat.

> You have the source, but the Maven model used by the whole Java ecosystem for distribution/management of packages doesn't work like that.

Oh, I don't know about that. Clojure does this; shipping source is default, and shipping bytecode is only done in cases of certain Java interop forms.

But Maven has a notion of "classifiers" that seem to be a good fit for this. I've seen them used to specify things like "this is the JDK 1.4-compatible version", so I wonder why Scala library authors haven't used them to publish the same artifact compiled against multiple versions of Scala.

Does clojure use maven for shipping source around?

Yes, the source is just stored in jar files, which can contain any kind of files.

'Jar' files are actually zip files with a '.jar' extension. On my Windows machine I have .jar associated with a zip application, very handy to quickly inspect its contents.

David carries quite a lot of weight in my book (we use Lift in production). Despite the negative title he makes a very good point and points out that this is entirely solvable. This falls under the "growing pains" category for Scala. I hope we get this right soon.

Go is also moving very fast and they often make backwards-incompatible changes to the libraries (and sometimes the core language) but upgrading code can be done automatically 95% of the time using gofix, which programmatically rewrites code to conform to the latest language and library updates.

http://blog.golang.org/2011/04/introducing-gofix.html http://golang.org/cmd/gofix/

Wow. So it just rewrites it to the latest version, with (according to what you just said) about 95% accuracy?

Does that make other people feel queasy, or just me? Like, that feels like a really bad idea to me. What happens if the result only mostly works?

It's a language undergoing rapid development. The developers are planning sometime in early 2012, to release a "Version 1" of go, to which no backwards incompatible changes will be made.

http://blog.golang.org/2011/10/preview-of-go-version-1.html

Typically gofix just works, well that is my experience of running gofix three times over the six last months. The release notes document when the language has changed and when gofix can help you migrate, and sometimes it can't help and it is documented.

Additionally, go exposes the complete AST for the language and a pretty printer as libraries, so you can write your own fixes if you don't trust gofix.

Python has been doing this for awhile, hasn't it?

Python has broken backwards compatibility with Python 3 and tools were written to translate 2.x code to 3.x but that is not at all a regular thing.

I would say that Python breaks backwards compatibility in small ways for minor releases semi-regularly, such as adding new keywords (with, yield), removing little-used standard library modules, and removing support for raising strings as exceptions, but always provides deprecation warnings for a release first and (for new keywords or syntax) allows early opt-in with "from __future__ import"

http://www.python.org/dev/peps/pep-0387/

> What happens if the result only mostly works?

You fix the other parts, or revert to a previous version of your source tree.

You missed the "automatically" part. The other 5% of the time, it can bail out with a message, and you go fix it yourself.

Python also has 2to3.

That is different than the fragile bytecode problem scala appears to have though. Go's issue is that the API's are changing fast so source code has to be modified frequently.

Scala's issue seems to be that even when the API's haven't changed and there is no need for a sourcecode change a compiler change could break you.

Frankly go's problem seems more manageable.

That Go does static linking also helps. And in Go-land source code is pretty much the only way things are distributed because it is so fast to build.

For example in Go on AppEngine you upload the source for your app, and it is built on Google's servers.

In a way Go's source is its 'bytecode' :)

Actually, it sounds like Scala's would be more manageable, if it were handled properly. In fact, you can find a kind of precedent for Scala's problem in C++, which was wildly successful in spite of itself and its many implementation problems that included routine ABI breakage.

I can live with having to rebuild my stack for a compiler upgrade, if the ecosystem is setup such that rebuilding the stack is a natural and easy thing to do. The problem seems to be that Scala's ecosystem doesn't work like that.

Go, on the other hand, is, in its current stage of development, requiring frequent source-level changes. I consider that much hairier, as it risks altering the actual meaning of the program even without possible compiler bugs (which Go, like every other language, can still have, I don't know why you seem to think Scala's situation there is special).

Given their apparent state, I would accept neither Scala nor Go as production ready in a general sense. That said, Go gets points for not pretending to be such.

A recent Typesafe blog post was written in response to the last Scala-complaint furor: http://blog.typesafe.com/getting-down-to-work

One of the points mentioned there is binary compatibility. The Typesafe people are working on it, so hopefully we'll see some improvement over the next year or so.

>> An attribute of Scala is that the Scala compiler generates fragile byte-code.

Can anybody tell me if this would be true to any JVM targeted language, or is it a specific issue of the Scala compiler?

I'm asking because I just recently decided to abort learning Scala in favor of Clojure.

As both a Scala and Clojure user, I believe this is primarily a Scala-specific issue. The one caveat to that is that, if you're using Clojure and AOT-compile your code, you can run into some issues if you are trying to use AOT-compiled code from different versions (or using a different version of Clojure than the one used to AOT-compile the code).

Now, there have been some source-level incompatibilities, especially going from Clojure 1.2 to 1.3, and this seems to have been a bit of a headache within some parts of the Clojure community.

As long as you're not AOT-compiling Clojure (i.e. producing class files, which is generally entirely unnecessary in Clojure except perhaps as the last step just prior to application delivery / deployment), you'll generally be very happy.

The 1.2 -> 1.3 transition has had a couple of bumps, but nothing compared to the pain I recall from my time across Scala versions years ago. FWIW, I have code I wrote years ago against pre-1.0 Clojure that still sits in jars in my Maven repository that is loaded and run in 1.4 alphas today. Backwards compatibility is mostly quite good.

In the JIT world, going .java->.class is not AOT. You may call it that, but it's essentially just source->source conversion.

What AOT really is, is when you first run a java program with certain flags, and some compilation starts, a special AOT version, which is usually a low optimization level which is saved to disk with some relocation markers.

This helps with startup time in subsequent runs as the compiler can just load its baseline pre-compiled classes straight into memory instead of having to do a bunch of grunt work during startup. Once things get going, the compiler can much more quickly work its way up to the higher optimization levels on the sections of code that need it.

THAT is what AOT means from a jitted language perspective.

Java bytecode is what we mean when we say 'compile'. AOT is 'ahead of time' with regard to clojure, since it compiles to bytecode at run-time normally.

"AOT" in the clojure world means compiling your clojure source down to JVM bytecode, which also means compiled-in calls against a particular state of clojure's internal java api. By shipping clojure source, one avoids any concerns over incidental changes to internal code.

Pretty sure this isn't even language specific. If you are using a library that drastically modifies how they implement something between versions, then you will run into this. Even in pure Java. Now, the ecosystem in Java is to usually not make large changes. At least not at the public API level.

I think Scala is slowing down in this area, but have not stopped.

It's not true of Java - the bytecode generated by javac is very robust in the face of new JVM versions. This is one of the reasons that Java has had to move so slowly in terms of language features.

Neither Scala or Clojure's compilers generate byte code that is expected to work with that from another version of the respective compiler or with a different version of the language runtime.

In the case of Clojure this is mitigated by the fact that (since it is a dynamic language) that applications and libraries are generally delivered in source form and "ahead of time" compilation is seen as an optimization rather than a code delivery mechanism.

I've raised this issue one year ago

http://codemonkeyism.com/scala-unfit-development/

There are two forces at play here:

a.) Binary fragility of Scala

b.) A community that upgrades libraries (your dependencies) to every new RC candidate - even when only fixing bugs. So if you have a show stopper bug you need to upgrade to the newest RC breaking all your other dependencies.

Either a. and b. would be no major problem, combined it throws you into a dependency hell for days hunting for dependencies that work.

Hope they resolve this issue. People promised a solution in the blog comments, then promised a solution a year later as the issue surfaced again on the #typesafe blog.

Really, it is just becoming increasingly difficult to see how proponents of Scala can credibly claim any kind of developer productivity gains. I can only imagine how much time your average Scala developer spends solving problems of accidental complexity that simply do not exist in more mature languages.

As a Scala developer, let me shed some light on this. In the last few months I have spent exactly zero time on this problem. It was a minor issue when 2.9 came out. No one forces you to upgrade, and we didn't till all our dependencies worked on 2.9.

And your organization has much more than 3 teams? The article specifically says this is a large multi-team coordination issue.

noelwelsh is responding to donjigweed, not the article.

Time spent: zilch, perhaps because most of my dependencies are java libs not scala ones. Like many Java devs starting out using Scala as a 'better java' which means adding more scala libs over time. It's definitely very pleasant to code in Scala vs Java, def more productive, to a surprising extent even. The compatibility issue needs to be addressed though.

Same here, zero time spent on this. Scala is a joy to program in. It's good to see these kinds of postings with complaints and suggestions. They'll move the language and the ecosystem forward.

I spent zero time on it. And I regularly run my code against 2.9 and 2.10 trunk. No problems so far.

Zero. The 2.8 to 2.9 upgrade took exactly one extra build cycle, which we had already committed other code to test anyways.

I expect this to be exactly the sort of thing that a commercial vendor will support: here is the language version and library versions that we support. Don't upgrade any of these until we tell you that you can, or you're on your own.

I'm just starting to experiment with Scala for scripting on top of existing Java, so the current state is not a hardship for me. The article is helpful in pointing out the task (technical as well as P/R) to be faced by any would-be "golden standard" vendor, though.

A response by J. Suereth to this post http://suereth.blogspot.com/2011/12/scala-fresh-is-alive.htm...

How about taking some time off from Lift development to implement the ideas that you and other worked on?

That's pretty much what I think every time he spends time on his anti-Scala rants.

Maybe he should switch the language? Looking at Lift's source code/API this is pretty much a abuse of the language.

He's one of the few true Scala experts around (how many can claim they've been using Scala for more than five years?).

The code he writes is probably as idiomatic Scala as it can get.

Yet another reason why people should be wary.

No he is a scala early adopter. There is a big difference. I have less and less respect for the man the more I look at Lift. In fact after I started working with Play and their scala support I have and even better sense that he is just a java developer who found a new way to manipulate XML files.

We're not discussing Lift nor his design skills.

We're talking about his level of expertise in Scala, which is probably one of the highest in the community.

Reply on news.ycombinator.com