Building Scala Projects: Maven vs. SBT
hammerlab.org
hammerlab.org
* SBT features its own DSL (Scala is, after all, the Scalable Language), which also includes infuriating operators such as "<++="
* There's a affinity towards elegance at the expense of basic understandability. A good example to illustrate this is the design of SBT is recursive; that is, your SBT build is an SBT program that must be built before your program is built. Then you may write a meta-SBT-build program to build your SBT build, and so on. I'm very sure SBT developers are quite proud of this architecture, but my team members universally thought this was pointless and confusing.
* It's slow, and trying to speed it up is an engineering spike in and of itself.
* As Scala doesn't bother to maintain binary compatibility across point releases, the SBT and plugin ecosystem has to use an older version of Scala. So now your SBT plugin code is probably using a different compiler than your Scala application code.
* In exchange for the steep learning curve, you are rewarded with a powerful feature set that was definitely innovative: incremental, parallel, and continuous builds, a REPL, a great SBT plugin community, etc.
After learning and using Make, Maven, and now Gradle, I have mainly learned one lesson: never write a build system.
* The "<++=" operators are much-maligned but I've seen them ~0 times in 6 months; I may have come in to SBT-land just as they were being finally phased out, I think I did see them in the beginning a bit.
* Agreed that the recursiveness of the build is presented somewhat confusingly and navel-gazingly!
* Speed-wise, I imagine you're talking about reloading projects, either in the SBT CLI or IntelliJ? That's pretty fair, I've had to adjust my IJ workflow a bit to not trigger full-project-refreshes as often (e.g. disabling auto-import). If you're talking about e.g. compilation or other tasks being slow, I'd be surprised and interested to hear more, though.
* re: SBT plugins being stuck on 2.10, I was also a bit daunted to read that early on, but at this point I've written several SBT plugins and it's basically never been an issue or introduced any friction; curious to hear if it's caused problems for you.
* Agreed that there is a steep learning curve and some payoff that reasonable people may value differently, and also to not write a build system :)
This is almost certainly in the negative column for me. SBT has the dubious distinction of being the only build system that takes me hours to figure out how to change a setting. Inevitably I have to fall back to reading the source, and given all the macro magic that can be difficult to unwind.
I've been using Pants, mostly because it's polyglot and has better multi-project support. The oss community is a lot smaller, and it has a fair number of problems around bugs, but I find the code much more approachable.
where are you using Pants?
I would really love to be able to use Pants as well, but alas, once I got running on SBT and was faced with a bit of rough-edges in Pants public-facing adoption/documentation story, I had to abort :)
I'd use SBT again for a locally contained, single project setup. Once you learn the arcana, it works well, has a ton of plugins, and the repl is nice. I don't think it scales well with new engineers or number of projects though.
I've used Blaze at Google and then Pants at Foursquare, so I've seen this sort of thing done, but yea, Pants seemed like it was going to take a higher level of commitment to set up / maintain, due to smaller OSS ecosystem around it, so I had to short-circuit that thread.
Also, putting everything in a monorepo is not really an option in my current OSS-focused setup, and I've come to have grave doubts about its desirability overall, after years of believing that it was the ideal way, but that's another discussion :)
"Build/Package/Release logic is complicated, and deserves sophisticated tools and abstractions just as much as the code it builds."
This seems correct until you realize that the #1 priority for your build system is to not be spending time screwing around with your build system. The benefits of using Maven to stay with the herd ends up dominating any benefits power you get to come up with clever build mechanisms. Then, yes, maybe you have an ugly shell script you have to run in your CI--but for all the crappiness of that, it's still composed on top of the thing that you basically never touch.
I've heard first hand from one big marquee Scala company (you could get it in a couple guesses) that this was basically why they were going to move away from SBT. That was years ago, so do with that what you will.
The "stay with the herd" argument is fair… if you're using/aping projects that have working Maven builds, it might be easier to mimic those than try to use a totally different tool; in fact, this is how my coworkers and I ended up starting on Maven: we effectively inherited the decision from Spark and ADAM.
"The herd" is also great for larger tooling/plugin support, but IME Maven folks more often than not think that SBT is less at parity than it is.
A related thing you called out is the "years ago" bit; a lot of people I talked to in the course of writing this had a bad experience with SBT N years ago (including myself!), so I think there's probably been a lot of improvement in SBT-land in the last few years, based on my finding it to be much more attractive now than it was last time I looked.
Finally, "spending time screwing around with your build system" is not unique to SBT :) in fact, that's partly why I became fed up with Maven ("I can enable these properties with profile A, and these other properties with profile B, now, is there one switch that can turn on both these profiles, which I frequently want to do?" comes to mind).
Anyway, we can ofc disagree about which tool to use for a given situation, I just wanted to express that I wish someone had put me on SBT sooner; good to know that others have had the opposite experience!
The ability to factor out repeated logic and configuration in SBT, by dint of using Scala instead of XML, removes a lot of complexity. Of course, arbitrary Scala can get more complex than arbitrary XML.
As some points of reference, not saying these clearly argue for one side or the other:
Here is the Maven POM for ADAM's core module:
https://github.com/bigdatagenomics/adam/blob/adam-parent_2.1...
and its parent POM:
https://github.com/bigdatagenomics/adam/blob/adam-parent_2.1...
For comparison, build.sbt from my fork of ADAM:
https://github.com/hammerlab/adam/blob/e4bee3227b5979e65a73d...
And a plugin that it inherits that factors out a lot of functionality I reuse across projects:
https://github.com/hammerlab/sbt-parent/blob/1.7.5/src/main/...
Spark's POMs (e.g. https://github.com/apache/spark/blob/v2.1.0/pom.xml) vs. SBT project folder (https://github.com/apache/spark/tree/v2.1.0/project) are another point of comparison.
It's not hard for me to imagine how someone used to staring at one or the other of these kinds of configs, maybe for years, would feel that it made more sense than the other, but having spent some significant time with both, I think the points about having the option to express more sophisticated logic – that SBT provides – are pretty important, and the lessons extrapolate-able beyond building Scala projects :)
~35 LOC per module for .sbt files plus classes in /project
300+ XML lines per module in Maven projects
And we do a lot more with SBT.
Plus compilation with SBT is ridiculously, laughably slow. Modifiying your build.sbt essentially causes a rebuild of your whole project. The complexity and expressiveness of SBT is a BAD THING in terms of compilation and troubleshooting.
Maven suffers from a different issue. Build processes are fundamentally procedural, not declarative so sooner or later you have to shoehorn your shell commands into a maven plugin section in your xml instead of just executing them as a line from within your build script.
In a perfect world I would have discussed Gradle here as well, but I've never actually used it and so that fell out of scope (same with a more thorough treatment of Pantsbuild that I hoped to do).
Your articulation of Maven's issues also resonates :)
I do experience slow project reloads in IntelliJ (not as much on the SBT CLI), so that's fair. Not a dealbreaker given how much easier (read: more possible?) it is to express the logic I want to express in SBT, but reasonable people can disagree about those tradeoffs :)
That being said, I've tried Gradle (knew a guy who worked for them too) and it's really nice. I could see using it over SBT. The trouble is sbt has a lot of nice plugins (sbt-native-package, sbt-docker) which, last time I checked, didn't have equivalences in the Gradle world (they might now).
we have noticed major improvements in stability, in the api, and in the community plugins from say 0.11 to now.
shortcomings aside, there are a few things sbt does much better than maven and gradle, and which we rely for our builds.
cross-compilation: One of them relates to binary-incompatability between major scala versions, hence the need to cross-compile. Sbt lets specify this direction in the build definition ('crossScalaVersions' is a built-in setting). The dependencies for each version automatically get picked up
quicker development cycles using sbt's interactive environment; compiling Scala source is time consuming; this latency can be pretty substantially reduced by working interactively, eg, one way to take advantage of this is to enter the sbt console mode and then preface tasks with a tilde '~' which will re-execte the task when the source)changes.
Disagree. It should not be complicated. We dumped Maven for Gradle. Huge mistake IMO. Instead of trying to simplify build and get rid of the million build exceptions and different conventions, we persisted on them, now with a more powerful tool.
We used to chop our legs off with a knife. Now we have a chainsaw.
FWIW I'm at a Scala shop where we use SBT exclusively. There's things about it that suck, it's definitely symbol-happy, but for our purposes it sucks less than Maven. Haven't looked into Gradle, all the other comments here have made me curious to try it out.
> operator-overloading enabling eDSL-creation
As far as I'm concerned, this is a bug not a feature. At least sbt seems to stick to ascii characters ;)
yea… that's a longer discussion. I've been influenced on this by some coworkers (cf. http://www.hammerlab.org/2015/01/21/introducing-ketrew-0-0-0...) and conference talks (including one at NEScala 2wks ago, https://github.com/nescalas/proposals-2017/blob/master/long-... ); one way to think of it is that there are blurry lines between a DSL and any plain old API… they're kind of the same idea, though conventionally DSLs have tended to involve more punctuation marks, and I agree that that can be taken too far, and that SBT did in fact take it too far in the past!
On the other hand, XML with a given schema is a kind of DSL as well, and where it's possible to make e.g. Scala DSLs overly concise to the point of inscrutable trickiness, XML APIs are frequently considered to be inscrutably verbose, as we've all heard and experienced ad nauseum.
However, given Scala's powerful type-checker and flexible syntax, more sane middle grounds can be explored. Maybe a less controversial example in SBT is the "/" operator for constructing filesystem paths (https://github.com/sbt/sbt/blob/v0.13.13/util/io/src/main/sc...), which I incidentally just mimicked in a java-nio-Path-wrapper library this past weekend (https://github.com/hammerlab/path-utils/blob/1.0.2/src/test/...). The alternative here is Java's Path.resolve(), which is a bit clunky to say the least.
Also some of the JSON-DSLs we dealt with at 4sq come to mind as nice bits of ad-hoc syntax that still benefit from all of Scala's type-checking wondrousness :)
(edit: fixed broken link)
Any specific examples you have in mind?
As I mentioned in the post, the first… 5 Maven plugins I looked for all existed in about as good or better form in SBT-land.
Come to think of it, the dependency plugin in SBT, https://github.com/jrudolph/sbt-dependency-graph, seems to be missing some wildcard features that I used to like in maven-dependency-plugin. OTOH, it has some things that afaik don't exist in maven-dependency-plugin, like dot-graph output.
That said, I remember trying it and not sticking with it years ago, but I also don't use it in SBT. I remember if it was particularly broken in any way that caused me to abandon it in Maven.