`go build` and `go test` do work, at limited scale and complexity. In Scala there's Scala-CLI which is excellent. If they work for you, you probably aren't the target market for these build tools. Once you start layering on bash scripts, layering on make, layering on Python scripts, layering on manual steps written down in a readme.md somewhere, that's the time when you should consider a proper build tool. And if that has never happened to you in your career, count your blessings :)
Why not just write some boring, pure code to handle the build? Why not write my build system in vanilla LYAH Haskell? It turns out that builds do have some specific requirements that most programs do not need to care about: caching, parallelization, introspection, and so on. Check out the following blog/talk for more details:
* Blog Post: Build Tools as Pure Functional Programs https://www.lihaoyi.com/post/BuildToolsasPureFunctionalProgr...
* Video: Mill: a Build Tool based on Pure Functional Programming https://www.youtube.com/watch?v=j6uThGxx-18&list=PLBqWQH1Miw...
Thus "naively" building your project "directly with code" ends up not working, so you do need some additional support. While most build tools end up constructing a complete bespoke programming environment from scratch, Mill tries to leverage the Scala language and JVM as much as possible, so you can re-use all your expertise and tooling (e.g. IntelliJ, VSCode, Maven Central, etc.) almost verbatim while getting all the necessary build-tool stuff (parallelism, caching, introspection) for free. Check out those two links if you want to learn more!
If your USP is to solve those layering cases, then why not target other ecosystems like golang/rust as well? Your design philosophy certainly seems to be language agnostic. By calling yourself a build tool for Java and Scala, it gives an impression that this is solving problems specific to those environments, and your adoption also indicates as such. Is it that these communities do not like to adopt such tools or is there something about the JVM ecosystem that tends towards having complex build requirements?
But you are right that it is not JVM specific! In the docs there is an example of adding Typescript module support, and an initial strawman implementation takes about 100 lines of code. I'm hoping others can extend Mill to places where I do not have the time and expertise
I also opened up a 500USD bounty to add a strawman Python example, so if anyone wants to try their hand at writing the 100 or so lines necessary, here's the link :) https://github.com/com-lihaoyi/mill/issues/3862
The issue is that most people in jvm land are on a closed bubble and haven't seen anything else. This is true for build systems as is for non OO design for example. Most simply don't know better and the rest of us are simply stuck.
Ant and then Maven started simple enough but people always find a way to justify adding more stuff. Gradle already started complex enough and they keep adding more stuff...
Gradle's complexity comes from at least two places:
1. The original vision of solving complex multi-technology/language Enterprise builds.
2. Poor early design decisions that they never recovered from.
Builds that require ad-hoc functionality is the default. It’s extremely rare that everything fits nicely into “cargo build” or other single language build tools’ model. And while these often have escape hatches, at that point you have to write imperative code with no caching and parallelization that is literally the job of a build tool.
161538 crates do not agree with you ;)
What about a large project built over 6 years by 50 people, and that has to use some obscure technology to communicate with company A, and another one that has an idiotic build step?
Which is how software should be generally made - from small and simple things, not from huge behemoths that contain the kitchen sink and brew coffee.
For your outlandish use cases you can use Bazel.
Not all Java builds are simply compiles. There are several projects that rely on processing steps during the Java build. EAR files are jars within jars.
Then, of course, there's all of the dependencies.
The modern Maven based repository based dependency manager is a blessing and a curse. Drag and dropping an artifact into your project that inevitably downloads the entirety of the internet. Now you may wish to cull your dependency tree, so that needs to be expressible as well.
The primary benefit of Maven and the pom.xml file is that for a vast majority of applications it just work. Even better, its become a universal "project" format that many IDEs directly support. It well handles "dependency hell" in a cross tool way.
I wish Maven were a bit faster, but, simply, it's as fast as it can be for what it does. A good Ant build just flies, but Ant "doesn't do anything". It's just a bag of steps that it follows (for good and ill), in contrast to Mavens declarative style (for good and ill).
I have no experience with Gradle other than I've never run into enough problems with Maven to justify trying something else. On its surface, it doesn't really appeal to me. I was comfortable with Ant (I have no problem with XML), I'm mostly comfortable with Maven. I've not been unhappy enough with Maven to try and jump back to Ant w/Ivy.
Other build tools and plugins just compete/fill in for:
* improved build speed / test speed: using background daemon to reduce strtup speed, intelligent caching / task reordering to avoid redoing, etc..
* extra functionalities like code generation, publishing or deployments. As code generation is really big in JVM world, and there are many ways to deploy an application: jar + libs in a zip file, uber jars, container image, etc...
I guess it's better than the nightmare over in the front-enders' world...
Granted, the constrained abilities do tend to keep folks from writing one-off snowflake build customizations, which is nice. But it still leave a hell of a lot to be desired. It was however, leagues ahead of Ant, which wasn't a high bar.
Maven gave us two things; good dependency management and convention driven builds (removing the horrible scripted build stuff in Ant).
Gradle from my point of view took the second one away again and it feels like it was just because people didn't like XML and couldn't be bothered to learn how Maven's build lifecycle actually worked!
Like I say, I'm a curmudgeon...
% mvn
[ERROR] No goals have been specified for this build.
Uh.
% mvn build
[ERROR] Unknown lifecycle phase "build".
Uh, nope, that wasn't it....
% mvn compile
BUILD SUCCESS
Now trying to run the app... Error: app jar not found.
Reading the README.md. Aha! So I need to install it!
% mvn install
16:59:35,075 [INFO] Building <blahblahblah> [1/32]
...
16:59:38,483 [INFO] -------------------------------------------------------
16:59:38,483 [INFO] T E S T S
16:59:38,483 [INFO] -------------------------------------------------------
^C
// me searching on google how to not run tests
% mvn install -DskipTests=true
// now good...
Ok, let's run some tests. The FlakyTest broke again in CI. Let me run it locally:
% mvn test FlakyTest
17:02:19,481 [ERROR] Unknown lifecycle phase "FlakyTest".
Aaargh, ok, googling it again:
% mvn test -Dtest=FlakyTest
17:03:16,102 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.3.0:test (default-test) on project blah blah: No tests matching pattern "FlakyTest" were executed! (Set -Dsurefire.failIfNoSpecifiedTests=false to ignore this error.)
The original idea behind maven is nice.
But the defaults are so bad, that the whole experience of convention over configuration has been ruined.Cargo and go build systems took the original maven philosophy and implemented them right, with good UX.
Gradle, SBT and friends took a step back to the times before maven, and went fully the Ant way, doubling down on "configuration over convention". Where "configuration" is actually "programming in a DSL on top of another language on top of Java".
[ERROR] No goals have been specified for this build. You must specify a valid lifecycle phase or a goal in the format <plugin-prefix>:<goal> or <plugin-group-id>:<plugin-artifact-id>[:<plugin-version>]:<goal>. Available lifecycle phases are: pre-clean, clean, post-clean, validate, initialize, generate-sources, process-sources, generate-resources, process-resources, compile, process-classes, generate-test-sources, process-test-sources, generate-test-resources, process-test-resources, test-compile, process-test-classes, test, prepare-package, package, pre-integration-test, integration-test, post-integration-test, verify, install, deploy, pre-site, site, post-site, site-deploy. -> [Help 1]
I am open to the fact that maybe catastrophically old versions of Maven did not include that help text, but certainly since 3.0 from 14 years ago https://github.com/apache/maven/blob/maven-3.0/maven-core/sr...> [ERROR] Unknown lifecycle phase "build".
And somehow missed that the error message also lists the valid lifecycle phases. (could have instead complained that it list all lifecycle phases which is a lot)
> // me searching on google how to not run tests
How do I know for other build tools how to skip tests? Skipping tests should not be necessary and is done to regularly. I would not consider a tool which immediately points out how to ignore those stupid tests as good UX.
Go projects normally just tend to be self-contained server-like software that doesn't need a lot of external libraries. But once you step away from that, you're on your own.
I guess my problem with Gradle is that app building should be way simpler than it is. Apps are not something niche anymore, but the tooling is still similar to the embedded software for microcontrollers.
My hot take: JVM build tools, especially Gradle, are a soup of unnecessary complexity, and people working in that ecosystem have Stockholm Syndrome.
In Golang, I spend about 99% of my time dealing with code.
In JVM land, I'm spending 30% just dealing with the build system. It's actually insane, and the community at large thinks this is normal. The amount of time it takes to publish a multi-platform Kotlin library for the first time can be measured in days. I published my first Golang library in minutes, by comparison.
Half are ignorant. Other half are like me and just stuck with no options.
But the tooling ecosystem on the JVM truly is horrific.
Anecdote: I wanted to publish a ~100LoC multiplatform Kotlin library -- just some bindings. I publish these sorts of things for Go with just a "git push".
Steps were: 1. Spend a few hours trying to understand Maven Central/Sonotype, register and get "verified". They're in the middle of some kind of transition so everything is deprecated or unstable. 2. Figure out signing, because of course published packages must be signed. Now I have a secret to keep track of too, great. 3. Discover that there is no stable Gradle plugin for publishing to the "new" Maven Central, it's coming soon... Choose one of the handful of community plugins with a handful of stars on GitHub. 4. Spend a few hours troubleshooting a "Gradle build failed to end" error, which ended up being due to signing not finding a signing key. 3rd party plugin didn't handle errors properly, and a bug in Gradle meant that my secret wasn't picked up from local.properties. 4. Eventually discover that because Kotlin Multiplatform can't be cross-compiled, there is no way to actually publish a multiplatform library without spinning up a bunch of CI runners. And you can't just publish code -- JVM packages have to contain compiled artifacts. 5. Realise this now involves maintaining GitHub Actions and Gradle, which is an ongoing cost. 6. Give up.
The harm that this kind of complexity must be causing to the ecosystem is immeasurable.
Even with all the headaches around modules versus CJS, and JS versus TypeScript, NPM is a lot easier than Gradle. Notably, you have a choice of alternate tools (eg pnpm, yarn, bun) that interoperate pretty well.
I guess my point is, Gradle and Maven are specifically and outstandingly bad.
EDIT: another comment making this point: https://news.ycombinator.com/item?id=41969847
In practice, Gradle really seems to fall down on the basic task of just being able to build stuff in the first place. It feels like you’re constantly fighting version hell just to find a Gradle version and plugins that work together, let alone your actual code dependencies.
And if you actually do need to do something slightly more complicated, like code generation, it’s very difficult to work with and the docs are really bad.
I'm surprised there is no source-only dependency solution for JVM -- it'd solve this issue. Pull down the source and build on the fly. Perhaps there is and I'm unaware?
Comparatively, Go does “only” 16k lines per second based on some HN comments.
But anyway, you may be right. I just ran mvn install for the second time with no source change on my current project. It took 57 seconds.
To justify this statement consider this blog post I wrote a while ago about porting GitHub Desktop (an Electron app) from its prior build/deployment system to Conveyor [1]. Conveyor is a tool for shipping desktop apps and is implemented as a single-purpose build system. The relevant part is this commit:
https://github.com/hydraulic-software/github-desktop/commit/...
The amount of code that can be deleted is huge! Some of it is in-process code that isn't needed with Conveyor (setting up Squirrel etc), but a lot is just shell scripts that happen to be written in JS. Replacing that with a real build system not only simplifies the codebase but means the build steps are fully parallelized, fully incremental, easier to debug, portable (the build can run on any platform), progress is reported in a uniform way and so on.
So whilst the JS ecosystem's approach to build tools may be "simple" in some way, in the sense that there's no dominant build tool like Maven or Gradle, that simplicity does cost you in other ways.
[1] https://hydraulic.dev/blog/8-packaging-electron-apps.html (Disclosure: Conveyor is a commercial product made by my company)
It's a bit Apple & Orange comparison: publishing a JVM only Kotlin library is quite easy, it's the multiplatform part that takes time.
In go: `git push` to a public repo
In js: `npm publish` after making an NPM account
In fairness, GitHub also finally got on board the train, too: https://docs.github.com/en/actions/use-cases-and-examples/pu...
1: https://docs.gitlab.com/ee/user/packages/maven_repository/
I don't think a high barrier to entry is overall good, in fact I think it encourages larger more complex packages to justify the maintenance burden
1: err, modulo that go.mod stuff that secretly adds a version slug to an otherwise normal github URL -- I'm looking at you, Pulumi: https://github.com/pulumi/pulumi/blob/v3.137.0/sdk/go.mod#L1
For what platform(s)?
Or did you really just push the source code?
I myself wrote a dependency resolver that matches Maven in functionality, and even a large project that uses Spring Boot and its dozens of dependencies can be resolved in a couple of seconds. About 10x faster than Maven or something like that. If you look at Maven's source code you'll see why. It's the worst kind of Java Enterprise overengineering you can imagine, complete with its own dependency injection framework, everything is pluggable (for no reason, really, do you really need to replace HTTPS for your protocols?? In Plexus you can), to the point that all the de-coupling results in lots of things duplicating functionality everywhere. I am not sure but I would bet Maven parses your POM at least 10 times to do anything due to the de-coupled nature of it.
I actually have no idea why these other resolvers are so slow, or why Coursier is so fast, but this slowness is very much a "maven" or "gradle" thing rather than a "jvm" thing. And Mill using coursier does significantly better!
This is so annoying that I prefer to use Rust over Java even in areas where things like better performance or better type system don't matter. But being able to start a fresh project with one `cargo init` and a few `cargo add` invocations to add any dependencies... well, this is priceless.
1: https://maven.apache.org/archetype/index.html and https://maven.apache.org/archetype/maven-archetype-plugin/us...
It’s like LEGO vs Playmobil. I want LEGO. ;)
* Spring Boot (it has a UI to create projects where you pick Java version, DB, build tool, some libs etc): https://spring.io/guides/gs/spring-boot
* JHipster - the nuclear option, pick what you want a la carte: https://www.jhipster.tech/
* JBang - a cute CLI for this: https://www.jbang.dev/
* Maven Archetypes - the old fashioned way (existed before "create-app" kind of tools appeared): https://maven.apache.org/guides/introduction/introduction-to...
And most IDEs also have "new project" wizzards.
Also, why does it matter how long it takes to publish a library for the first time? It sounds like a non-issue to me. I have written dozens of libraries and published them to a local artifactory instance because it simply doesn't matter if your company specific code is accessible to the world or not.
The key feature that justifies their existence is parallel and incremental execution of DAGs of world-mutating tasks. This is an awkward fit with most programming languages, hence the prevalence of DSLs. But people don't want their build system to become a general purpose programming language, because they don't want to think about build systems at all and because programmers don't buy programming languages anymore, so, this causes a big design tension between generality (people want to use build systems to automate many things) and deliberately limiting the expressive power to try and constrain the design space and thus tooling investment required.
Java is in an awkward place because the JDK was born in the 90s on UNIX, by people who thought make is a sufficiently good solution. You still see remnants of this belief in the official Java tutorials, in JEPs, and in the fact that OpenJDK itself is compiled using an autotools based build system! (fortunately it's one of the nice make based build systems out there).
The problem with make is twofold:
1. It assumes a CLI that's both powerful and standardized provided by the host OS. Windows violates this assumption but Java is meant to be portable to Windows.
2. "Plugins" are CLI tools or scripts and so make implicitly assumes that subprocess creation is cheap. But process creation on Windows is expensive, and starting up JVM programs is also expensive due to the JIT compiling.
Therefore make just doesn't work well in the JVM ecosystem. At the same time, the Java project wasn't providing any competing solution, so the wider open source community was left to fill in the gaps. These days language developers provide build tooling out of the box as part of the base toolset along with the compiler, but Java still doesn't.
So - you ask, what are people doing with Gradle/Maven that requires all those features. The answer is: everything! Gradle builds frequently orchestrate dozens of different tools as part of a build pipeline, build documentation websites, do upload and deployment, download and manage dependencies, run security scanners and license compliance checkers, analyze dependency graphs, modify compiler behaviors, and so on.
Additionally Gradle isn't specific to Java, or even JVM apps. It can also be used to compile C/C++ programs, run native code compilers like Kotlin/Native, and it abstracts the underlying platform so Gradle builds aren't tied to UNIX.
That's why it's so complicated.
Configuration and workflow execution.