Bazel can now build Haskell | Hacker News Reader

Bazel can now build Haskell

tweag.io

135 pointsmboes53 comments

I wonder if Bazel takes off across many languages. It wouldn't hurt to have a modern autotools & co. replacement across ecosystems. I don't think it's doable, considering the fact that we have language specific package managers and build tools, but you never know.

By "replacement", I mean picking up a random open source project and being able to do the equivalent of:

./configure

make

make install

To be able to build and run the app directly from sources, when needed.

Right now, it's a mess, the "replacement" is:

curl whatever.io | sudo bash

> By "replacement", I mean picking up a random open source project and being able to do the equivalent of

For that, Bazel needs someone to write all the necessary configs, and make them available out-of-the box, with sane presets baked-in.

You can look at Angular team's working on integrating Bazel into their build process, here's the repo: https://github.com/alexeagle/angular-bazel-example

There's:

- build.bazel

- Workspace

And then there are rules that have to be included and/or developed:

- nodejs rules, https://github.com/bazelbuild/rules_nodejs

- sass rules https://github.com/bazelbuild/rules_sass

- typescript rules: https://github.com/bazelbuild/rules_typescript

etc.

And those rules have to be not only loaded, but specific steps from those rules have to be invoked (see WORKSPACE).

I'd say there still still a lot of configuration involved in setting up all the moving pieces. Will it get better? Hopefully (there are just not enough good/decent build tools).

That doesn't seem to be an issue. bazelbuild/rules_typescript is able to work with NPM and Yarn, for instance

https://github.com/bazelbuild/rules_typescript

I don't see why this wouldn't be extensible to other languages with similar build systems. In practice, I find it useful to think of Bazel as a scripting languages that works when commands generate reproducible output files.

I don't think this is very far off, most languages with a package manager are relatively easy to finagle into bazel. That said, c/c++ dependencies are not, and every bazel user needs to put in some extra effort there. Eventually bazel-packages won't be empty and might solve that problem, but time will tell.

Right now there are a handful of (mostly Google) projects that you should be able to clone and bazel build after only having installed bazel.

To build bazel itself: # install bazel w/ system package manager git clone https://bazel.googlesource.com/bazel && cd bazel && bazel build //src:bazel

Doing “./configure && make && make install” is no better than doing “curl url | bash”.

Actually no, there is a difference, the former never works properly for any sizable project, while the later does, because for the later the user just wants to install the app so all the script usually does is to download some binaries.

Having a common build environment is impossible because you need a common way to manage dependencies and that will never, ever happen, for better or worse.

Man, this might be a dream, but it's one I'd love to see come to some form of fruition.

As someone who's learned programming through self-study, the broad swath of tooling out there can be a bit of a barrier to picking up new languages and concepts, especially when the concepts related to building and delivering software are occluded by some of the opinionated decisions all the different tool sets make about how building projects should work.

Learning a new language and new concepts already takes a good deal of time, which may be a valuable resource if you're not in any official program of study and making use of free moments. This is only exacerbated by the frequent need to learn a whole new tooling ecosystem along with any new language you decide to pick up. A sort of rosetta stone of package management and build tooling would be a godsend.

That's the idea.

Use bazel to describe and build the projects incrementally. The more languages are supported the less project-specific build systems have to be learned.

Use Nix and nixpkgs to pull external dependencies (and bazel).

I don't think Bazel is the right tool for that niche. I don't think Bazel shines in any context other than a monorepo.

I cannot agree with you more. The last time I look for build a bazel project with external dependency it was messy and undocumented. It was to use Tensorflow and Opencv in a standalone C++ code, tensorflow use bazel by default.

By external dependencies, do you mean things like arbitrary Git repos? There are standard rules there to accommodate things like that.

As well, this may be new documentation since you last looked, but there's https://docs.bazel.build/versions/master/external.html to read now, at least.

Presumably they mean non-Bazel external projects, which that link doesn't even touch on ("The other repository uses make... just rewrite their build system!") - an external project with the same build system is the "easy" case for things like this and CMake; any new contender has to accept that there is a wide ecosystem of other solutions it's going to have to interoperate with.

...unless you are in a corporate environment with a build team to integrate complicated external projects into a monorepo and where things like build system can be mandated.

That's always the feeling I've gotten from Bazel - you either do things the "google" way, or find something else.

This is one of the reasons I find people put too much emphasis on programming language features and not enough on tooling. If I want to contribute to your project, learning a new language (well enough to begin submitting patches, anyway) is often a lot easier than figuring out how to build the stupid thing anyway.

Scripts for dependencies, scripts for building, scripts for packaging, and often with their own special snowflake syntaxes for each.

Go has room for improvement to be sure, but I love that I can jump into any project and run `go build`, `go test`, `go install`, etc and they do the right thing. Even if you don't know the language, it's easy to figure out what's happening because the language itself is basically just a vanilla imperative language with a GC.

Not enough languages devote enough care to their tooling, in my opinion.

Yes, look at how git basically triumphed over all other version control systems. We need a revolution like that in the space of build-tooling.

I'm not sure I like the analogy, I wish a system with a saner UI had won :)

Magit? (joking, I agree with you :)

I doubt Bazel will take off for most open source projects due to being written in Java, but I do share the sentiment.

I've been meaning to try and write a somewhat saner build system to replace the Auto* hell, at least for C and C++. I'm focused on other things right now but I've been studying and collecting information about the architecture of many other build systems for a while, and some of them are so much ahead of those used in open source projects it's not even funny. At the very least I'll be writing a lengthy blog post with my findings.

The java dependency is my biggest hangup wrt to incorporating it into my own environment.

So far, I've avoided having to incorporate the JDK into my stack. Having to bring it in just for a build tool, keeping it up-to-date on developer configurations, just feels like a royal pain.

I realize it's never gonna happen, but having bazel rewritten in Go or C++ or anything else that produces a single binary would be awesome.

They just need to use Java9 modules so they can use jlink to produce native binaries. No need to use another language.

We have done more or less that: https://please.build / https://github.com/thought-machine/please

It's written in Go so doesn't need any external dependencies unless you want to target that language (i.e. it only needs a JDK if you're trying to build Java).

Disclaimer: I'm the main author.

Wow. Looks good! Thanks for the heads up.

Skylark (the language used by Bazel) is being ported to Go. Maybe a good place to start?

https://github.com/google/skylark

So, not a build tool that works across languages, but a generic mechanism for bootstrapping and running a build tool that works across languages. That actually sounds pretty reasonable.

Okay, i, the original developer, somehow obtain Tool X, and tell it to do a number on my project. It works out or is told what build tool i'm using, and generates a portable shell script in the project root. Later, you, the developer who has picked up my random open source project, can run it:

  ./xmake

Or on Windows:

  ./xmake.bat

That then uses some almost-definitely-available HTTP client (curl on Linux, some PowerShell access to a .net client on Windows) to obtain the binaries necessary to run the project's build tool, caches them somewhere sensible so it won't need to download them again next time, then sets up whatever environment is needed to run the build tool, and runs it.

For example, if i have a Gradle build, it will download a JDK, unpack it somewhere, export JAVA_HOME, and run ./gradlew build. That will then download Gradle itself and run it, which will download the dependencies and build the project.

If i have a Cargo build, it will download the appropriate version of Rust, then run Cargo.

The top-level script could support a number of standard targets - 'all' (the default), 'assemble' (just compile and link the main code), 'test', 'package' (put everything in a zip file somewhere), etc. In each case, these would translate to the appropriate invocation of the real build tool. Where a build tool doesn't have the concept (does Cargo do packaging in a zip?), the top-level script could provide a shim.

Does that make sense?

If a target language already has tooling to install specific versions of its SDK, then the top-level script should chain to that, instead of duplicating its functionality. That means rustup for Rust, ruby-install for Ruby, and probably SDKMAN! for Java (obscure, but it exists).

Where a language doesn't have such a tool, perhaps it could be shoehorned into an existing one. SDKMAN! is nominally language-agnostic, so perhaps that.

Probably the biggest stumbling block is C/C++. They don't even have a standard build tool, let alone a standard SDK version manager. What tools do exist often assume that they can use system versions of compilers, libraries, etc, rather than ones from a particular installation. Still, i've written enough C++ build scripts to convince myself that this would be possible, even if was a bit painful.

This is exactly what we're doing at my startup. The vision is to let anyone check out any repository and just run one command that works irrespective of the actual build tool to compile those sources in the right environment and get you binaries in the end.

Like Nix?

Given that Nix is already popular in the Haskell ecosystem and it's also polyglot, why not build the monorepo with that instead of Bazel?

I am a Nix user, but one advantage that Bazel has is that it can do more fine-grained incremental rebuilding (nix as commonly used can only do per-project rebuilds). Also while there is some overlap, Nix usually relies on language-specific package managers to perform the actual building, whereas bazel handles that itself. Nix is more of a packaging tool, whereas bazel is a build tool.

Can Nix build incrementally? One random collection of notes I just dug up says that it rebuilds packages from scratch on minor changes. It seems like Nix is addressing a different problem.

I thought that Bazel's notion of "incremental building" was at the project level of granularity too. Is that incorrect?

Bazel recompiles at the source file level, and it's very good at only recompiling only what's necessary; for example, it caches object files for C++, and will only rebuild the object files of changed source files, and then relink.

When suggesting that rust do the same, I learned that nix doesn't work on Windows.

Seems like SCons: http://scons.org/

> Built-in support for C, C++, D, Java, Fortran, Yacc, Lex, Qt and SWIG, and building TeX and LaTeX documents. Easily extensible through user-defined Builders for other languages or file types.

Godot https://godotengine.org/ uses SCons and it's great; easy, quick and reliable.

What I don't like about scons, is that when it fails, it's inscrutable. Happily for scons detractors, it seems to be fading from use.

A significant hurdle for the universal adoption of Bazel is its too many dependencies.

sudo apt-get install openjdk-8-jdk

sudo apt-get install pkg-config zip g++ zlib1g-dev unzip python

That's a fair few things, but it doesn't see that awful. None of those dependencies seem exotic. I wonder if they could get rid of Python by using Jython on top of the JDK.

It's way more than that. It bundles a lot of jars --- building those from source, recursively, is a big task.

(I tried building Bazel and its dependencies completely from source for Guix.)

I think any big Java app is a mess for distributions. They usually include a ton of jars, since they just get them "for free" when using Maven/Gradle/Ivy.

Building the jars should be trivial in theory (especially for Maven), but the different between theory and practice is smaller in theory than in practice :)

Yeah, a lot of Java things require extensive patching to cut dependency cycles or to make the code compile with the current JDK.

Say what you want about Autotools, but at least the user won't have to install Autotools to build the software. A build tool that depends on Java and dozens of Java libraries that have dozens of dependencies of their own, each with their own set of bootstrapping problems ... it's a tough sell.

True, but as someone who has been part of that ecosystem: your standard Java developer doesn't rebuild everything. They don't want, need or have to do it. The Maven repo is immutable and very reliable. So everybody builds on top of the binaries.

Of course, there's this implicit system of trust, which might be misguided, but there's many billion dollar companies built on top of it.

Right, and a proper packaging "in the box" reduce it to a single install command on most any Linux. Right now it's pretty close to that on a Mac also if you already have brew.

The current long pole in the tent is Windows. Bazel brings along its own JVM - so there isn't any Java or Java dependency difficulty (as some other posters have suggested). But the install instructions direct you to provide some dependencies first, by hand. Of course in the long term a little while spent installing is no big deal... but each new adopter has to get through the short term on the way to the long term.

Hopefully a future Bazel installer will "just work" on Windows as a single installation with no instructions.

Maybe they could use this for the JRE bundling: https://docs.oracle.com/javase/9/tools/javapackager.htm#JSWO...

Python comes with your distribution. It's Java that is the bigger obstacle in some shops, I expect.

Can you expand on why that is an issue? The other argument I've seen is that just requiring java is already too much.

Java is kind of a bitch to bootstrap on anything that is not x86. We've been trying to get openjdk9 to build on arm for months now in nixos. This means we cant package any bazel-based projects for Nixos. Also the dependency chain is huge. So building from scratch also just takes a lot of time. It's very hard to package in a source-based distribution