Please – A cross-language build system
please.build
please.build
Glad it's getting some discussion on HN. Can't wait to hear other's experiences.
It's consistent between all languages and supports a lot of functionality. For example the proto_library and grpc_library rules make building microservices extremely easy. You can build an entire service in like 3 lines of boilerplate.
It also makes it easy to do cross-language deps. For example a c library included into a Java library that uses native bindings to wrap it.
The build logic and CLI experience looks to be well thought out. I really like the native sandboxing support and the intentional exclusion of the calling environment. There's a lot of polish here.
To the maintainers of Please: please make cases and user stories, also comparisons with other build tools.
Many visitors will jump on each "comparison with tool X" where X=each of the current tools I'm familiar with. Only "Bazel, Buck or Pants" are mentioned. Okay no JVM, looks like Please is a better thought out Blaze/Buck.
Please mention CMake and others, like the C/C++-oriented http://floooh.github.io/fips/ (a high-level build system "similar to Rust’s Cargo or Javascript’s NPM, but for C/C++ projects.").
Changing build systems is hard, and getting buy-in from the team even harder - so you need to demonstrate value up front.
With that being said I have put a good amount of work into the QuickStart.
The code labs are designed to get you up and started in an inviting way. If you have any specific feedback about your QuickStart experience I’d love to hear it.
Have you considered sponsoring, or accepting donations, to work on IDE integration? That's pretty much the only reason I migrated my current employers codebase over to Bazel instead of Please.
Also, the docker support in Bazel is pretty "different". It's not bad, but it's not immediately obvious to newcomers how this is meant to work.
It's getting better every day though.
(I'm one of the original developers, but these days do less maintaining of it than tatskaari does)
We'd also tried Gradle previously but that was pretty awful for anything non-Java which made it a non-starter.
To be clear, I'm one of the original implementors, although I imagine that was obvious already...
People interested in type annotations can check this discussion: https://github.com/bazelbuild/starlark/issues/106
For most purposes, Bazel dependency on Java is an implementation detail (users don't need to install a JVM), although you might notice it if you need to bootstrap Bazel.
(I co-designed Starlark and I used to work on Bazel)
My rubric for documentation is "how many clicks to code". I found the installation instructions, and started to look for usage, but didn't find it. Too many clicks. That's where I'd start.
PS - IMO the gold standard for this evaluation method is Sinatra.rb, which is zero clicks to code.
[1] http://sinatrarb.com/intro.html [2] https://github.com/sinatra/sinatra#readme
So much wasted time goes into diagnosis of incorrect results from incremental builds that most people who implement CI systems never use incremental builds, and always build from scratch. Developers are far too used to having to do things like 'make clean' because their build didn't work quite right. Efficiencies are then gained by doing things like ccache/Gradle build cache, which trace their dependencies better than most naively written build systems do.
Edit: zig is a beautiful exception (https://ziglang.org/download/0.3.0/release-notes.html#cachin...)
My perspective here is specifically from that of a maintainer of large builds where the code base is always under active development. Murphy's law becomes your enemy at scale.
This is not a criticism of Please - just an observation that as an industry we are not there yet. But, I'm happy to see things moving in the right direction.
genrule(name = "oops", cmd = "date > $@", outs = ["oops.txt"])
Again, I am not picking on Bazel here. Correct builds are difficult enough to get right even when everyone involved wants to keep the build correct! But, the reality is that a lot of real-world build conditions can be downright hostile. (did you know that ext4 vs xfs can change the order JARs in the same directory are loaded off the classpath? I wish I didn't!)
You hopefully have tried out many other build systems by now, been frusted with all of them, and only then made your own. And with all that experience with the status quo, marketing should be easy.
If you haven't done that, this project is a net-negative on the world, because the proliferation of build systems exacerbates Conway's law and balkanizes our software commons.
NIH in FOSS is not free.
This isn't a paid product, after all.
(I'm also a traditionalist that thinks writing good prose and writing good programs are related skills.)
For me, writing code is a way of solving problems for people. That involves listening to them to see how I can help. Then collaborating with them to iteratively make something that fits their needs. Which includes helping them adopt it and get productive with it. To me, all of that is part of software development.
And that's most of the "marketing" that's needed for a free, open-source tool. When that's lacking, I have to wonder to what extent the tool was actually made to solve somebody's problems. As opposed to being created because the developer was excited to build a thing as a technical exercise. Either path is fine, of course, but it's a mistake to confuse the two.
Growing up, one of the jobs I took was as a system's administrator at the same company my father worked at. It was a small company, maybe six people or so, doing contract work for aerospace. One of the older men who worked there held his Ph. D. in aerospace engineering. He was kind, warm and brilliant... but he struggled with things like sending coherent emails to clients. But that was okay - other people were able to take that on.
Whoever made this clearly knows about slick presentation --- they've got all those colors and shapes pizzas --- so they either capable with slick communications, or work with someone who is. My patience for lack of explanation is therefore a lot thinner.
I’m sorry, what?!
Do you realize the “sell something to people” industry has been at least as big as the tech industry for many decades and is the one growing faster and faster?
Also marketing has historically been a problem for people who make stuff.
Also, better have the Balkans of software than the USSR of mega corps.
When we created Please we were currently using Buck from Facebook. As has been covered elsewhere at the time it had some limitations which meant that it was getting more and more expensive to workaround (no multiple outs from rules so doing things like sourcemaps was hard etc). We had previously migrated from a set of disparate build tools (Gradle, Gulp etc) to Buck after evaluating it against Pants (we actually trialled our repo with both).
We all had experience with Blaze (Bazel did not exist in 2014/2015) and wanted to get closer to the experience we had in Google, hence the final decision to build our own. At the time we rolled it out we had full Buck compatibility (and actually most of our engineers did not notice initially). Once we were happy it worked, we migrated in full to Please.
Overall it allowed us to provide much better tooling for our developers (queries on the build graph for smaller CI/CD footprints, coverage in all languages, and for rules that can output multiple languages (like protocol buffers) we only build variants in the requested languages rather than all languages).
If Bazel had been open-sourced and easy to extend at the time we would certainly have looked to adopt and improve it (as we did with Buck at the time). But by the time they open-sourced it the two system had diverged in their approach and use cases.
I (obviously somewhat biased) think having multiple principled build systems which prioritise the needs of different communities is good for the ecosystem overall (and allows the sharing of good ideas such as the Remote Execution Interface).
As a concrete example, one frustration at $DAYJOB is that building certain Google-released OSS is hard to integrate with our existing build system because Bazel is pretty opinionated about compiler-provided headers, output paths, etc. We're calling Bazel as a subprocess during our build and making various files available to it to make our compiler build visible to it, and then we pack up the results. I'm pretty sure if we somehow adopted Bazel for our entire build this would be much better - would it also be better if we adopted one of those other systems?
(Maybe one way of asking this is, are any of the corporate sponsors of Buck, Pants, or Please building third-party Bazel code from source? Or vice versa?)
A broader question: would it be worth defining a compatible subset of BUILD file syntax and library calls (i.e., not just Starlark but also the rules themselves for common things like building C libraries or JARs)?
Are we in the build-system equivalent of the vendor-specific C / C++ / UNIX implementations, and will a cross-vendor standardization effort emerge one day?
[0] https://opensourcelive.withgoogle.com/events/bazelcon2020/wa...
On the BUILD file level, Buck is using the Starlark. Pants and Please are close enough that some tooling (e.g. buildifier) can be shared. But that's about it.
If the goal is continued maintenance, then encouraging new engineers to explore and read the codebase with tools they can comprehend is critical.
disclaimer: have not used these particular tools, but the domain is nice and polite
Here are some real reasons why Make isn't the end-all:
- Make doesn't allow platform selection in a nice way. - Make doesn't work on Windows natively (no, NMake doesn't count). - Recursive make doesn't work well at all. - Make doesn't track byproducts or deleted artifacts. - Make doesn't have auto-clean. - Make doesn't facilitate out-of-source builds. - Make itself isn't a scripting language (arguably good/bad). - Make doesn't facilitate compiler selection cross-platform, so things like MSVC are nearly impossible to integrate nicely without the use of the developer tools prompt. - Make can't (elegantly) handle a number of rule cases, such as single-input multiple-output rules (it can, but it's a hack).
Not sure why you think Make is unapproachable. That's like saying shell scripting is unapproachable. No developer worth a damn is going to avoid shell scripting, and as long as they understand "this file turns into this file using this command" then Make makes sense.
As with all tools, Make and Autotools and CMake and probably Plz will be misused by developers that think they're being clever. In actuality, they make things worse at best and unusable/unmaintainable at worst.
As a build systems designer, I've evaluated Plz personally and found it to be neat but not suitable for most of the problems I think build systems should solve.
all of which are even worse than make itself.
Also curious to know, is there no work done in the realm of developer productivity inside the FPGA world?
I've only looked at modern tools briefly but everytime I come away realising we'd have to re-write everything. There is no out-of-the-box support for our tools/language and it would be a lot of work to learn someone else's DSL, re-implement everything and then potentially discover a load of ways it's broken for our workflow. Personally I think it'd be worth it in the long run but it's a really hard sell.
These tools operate at a higher level of abstraction. The BUILD file abstraction is something like "I am declaring a C library with these sources and headers." The build system then turns that into a set of commands to run. This higher-level abstraction provides a bunch of compelling features:
- Reliability: it's easy to write Makefiles that are fragile, for example one that works when run on a single CPU but fails when run with -j, because of undeclared dependencies. When the tool is in charge, it has more control and can sandbox each build step so that undeclared dependencies simply won't be available in the sandbox.
- Reproducibility: similar to the previous point, it's easy to write a Makefile that produces incorrect results with an incremental rebuild. For example, unless you implement your own header scanning (or manually list your header dependencies) "make" won't know to rebuild a source file when a header changes. These tools can ensure that builds are reproducible and incremental rebuilds are always correct.
Besides that, working at a higher level generally means writing less code to accomplish the same task, and the code you do write is closer to expressing your actual intent compared with a Makefile.
> The BUILD file abstraction is something like "I am declaring a C library with these sources and headers."
This is wrong. Even ninja has generic rules. Here's an example of a minimal makefile:
OBJ = src/a.o src/b.o src/c.o
libfoo.so: $(OBJ)
$(CC) -shared -o libfoo.so $(OBJ)
There's no reason this shouldn't be possible with make; it just hasn't been implemented so. Do bazel/buck/please actually do this? As far as I know tup is the only tool that actually verifies inputs/outputs of rules, and it needs FUSE to do so.
> For example, unless you implement your own header scanning (or manually list your header dependencies) "make" won't know to rebuild a source file when a header changes.
At least with GNU make, it's very easy:
CFLAGS += -MMD
-include $(patsubst %.o,%.d,$(OBJ))
Make has problems, but the ones you listed aren't they.
Your example does not contradict what I wrote. You manually specified the tool to be run ($CC) and all of the arguments to that tool.
It's true that there is a level of indirection through the $CC variable, but you're still operating at the level of specifying a tool's command-line.
> There's no reason this shouldn't be possible with make; it just hasn't been implemented so.
Make is 44 years old. If it were an easy extension to the existing paradigm, there has been ample time to implement such an extension.
> Do bazel/buck/please actually do this? As far as I know tup is the only tool that actually verifies inputs/outputs of rules, and it needs FUSE to do so.
Bazel certainly has some amount of sandboxing, though I don't think it's quite as complete as what is available internally at Google with Blaze. I haven't used Buck or Please, so I can't speak for them.
> True, it's a bit of a footgun, but by no means difficult.
Well footguns aren't great. :) As just one example, any header that is conditionally included (behind an #ifdef) could cause this cache to be invalidated when CFLAGS change, but Make has no idea of this.
Notably, I didn't specify the arguments to be used to compile the source files. But if you insist:
linklibrary = $(CC) -shared -o $1 $2
libfoo.so: $(OBJ)
$(call linklibrary,$@,$(OBJ))
>> There's no reason [sandboxing] shouldn't be possible with make; it just hasn't been implemented so.
> Make is 44 years old. If it were an easy extension to the existing paradigm, there has been ample time to implement such an extension.
‘Nobody's done it yet, ergo it's not possible or easy’ is not a valid argument.
> just one example, any header that is conditionally included (behind an #ifdef) could cause this cache to be invalidated when CFLAGS change, but Make has no idea of this.
Then you specify that the source files depend on the makefile.
Then you gratuitously rebuild everything whenever the Makefile changes, even if you only changed a comment.
Also this scheme is incorrect if the Makefile was written to allow command-line overrides of variables like CFLAGS, as many Makefiles do.
But these are just details. The larger point is this. The language of Bazel is defined such that builds are automatically correct and reproducible.
While it's true that Make has some facilities like "call" that support some amount of abstraction, it is up to you to ensure the correctness. If you get it wrong, your builds are back to being non-reproducible.
It's like the difference between programming in a memory safe vs a memory unsafe language. Sure, every thing you can do in the memory safe language can be done in the unsafe language. But the unsafe language has far more footguns and requires more diligence from the programmer to get reasonable results.
Ultimately, make and co are text oriented, while bazel and co are object oriented. Hacking object oriented capabilities into a text-oriented language isn't particularly fruitful or ergonomic.
Bazel and make are both text-based languages for describing symbolic, abstract structures. Just like pretty much every other programming language.
Bazel and make both use the same abstract structure, just like pretty much every other build system: a directed graph of build directives and dependencies.
But let's look at how a rule is implemented in bazel[0]. Here's a rule "implementation" for a simple executable rule[1]:
def _impl(ctx):
# The list of arguments we pass to the script.
args = [ctx.outputs.out.path] + [f.path for f in ctx.files.chunks]
# Action to call the script.
# actions.run will call "executable" with
# "arguments", saving the result to "output"
# access to files not listed in "inputs" will
# cause errors.
ctx.actions.run(
inputs = ctx.files.chunks,
outputs = [ctx.outputs.out],
arguments = args,
progress_message = "Merging into %s" % ctx.outputs.out.short_path,
executable = ctx.executable.merge_tool,
)
concat = rule(
implementation = _impl,
attrs = {
"chunks": attr.label_list(allow_files = True),
"out": attr.output(mandatory = True),
"merge_tool": attr.label(
executable = True,
cfg = "exec",
allow_files = True,
default = Label("//actions_run:merge"),
),
},
)
Targets of this form have metadata, they have input and output files that are known and can be queried as part of the dag. Other types of rules can be tagged as test or executable, so that `blaze test` and `blaze run` can autodiscover test and executable targets. This metadata can also be used by other rules[2], so that a lot of static analysis can be done as a part of the dag creation, without even building the binary. To give an example, a rule like
does_not_depend_on(
name = "check_deps",
src = ":opensource_thing",
forbidden_deps = [
"//super/secret:sauce",
]
)
Potentially, one could create shadow rules that do all of these things, but you'd need to do very, very silly things like, off the top of my head, creating a shadow filesystem that keeps a file-per-make-target that can be used to query for dependency information (make suggests something similar for per-file dependencies[3], but bazel allows for much more complex querying). That's what I mean by "object-oriented". Targets in bazel and similar are more than just an executable statement with file dependencies. They're complex, user-defined structs.
This object-oriented nature is also what allows querying (blaze query/cquery/aquery), which are often quite useful for various sort of things like dead or unusued code detection or refactoring (you can reverse dependency query a library that defines an API, see all direct users and then be sure that they have all migrated to a new version). My personal favorite from some work I did over the past year or so was is `query --output=build record_rule_instantiation_callstack`, which provides a stacktrace of any intermediate startlark macros. Very useful when tracking down macros that conditionally set flags, but you don't know why, and a level of introspection, transparency, and debugability that just isn't feasible in make.
That's what I mean by object-oriented vs. text oriented. Bazel has structs with metadata and abstractions and functions that can be composed along and provide shared, well known interfaces. Make has text substitution and files. While a sufficiently motivated individual could probably come up with something in make that approximates many of the features bazel natively provides, I'm confident they couldn't provide all of them, and I'm confident it wouldn't be pretty or ergonomic.
[0]: https://docs.bazel.build/versions/master/skylark/rules.html
[1]: https://github.com/bazelbuild/examples/blob/master/rules/act...
[2]: https://docs.bazel.build/versions/master/skylark/aspects.htm...
[3]: https://www.gnu.org/software/make/manual/html_node/Automatic...
Everything else that is needed for a modern build system is lacking:
- expressive scripting language
- platform-related configuration
- feature-related configuration
- handling hierarchical projects
- properly handling rebuilds when the Makefile itself has been changed
- dealing with dynamically generated parts of the dependency graph
- handling third-party dependencies
- build reproducibility
- etc... the list is very long
At a very minor cost after all: your sanity.
- Cannot handle multiple outputs for a single rule
- Does not rebuild when flags change
- Make rules may contain implicit dependencies
- Slow for large codebases
- Does not understand how to build for multiple platforms, sucks for cross-compiling
- Recursive make sucks (job control does not work across recursive invocation boundaries)
- You must create output directories yourself
- You must create your own rule to clean
This adds up to a fragile & slow build system, where you have to do a full rebuild to have a reasonable level of assurance that your build is correct—incremental builds aren’t safe.
There have been a few attempts to make a “better make” over the years like Jam, SCons, Ninja, Tup, etc. each with their own pros and cons.
The new generation of build tools—Bazel, Buck, Pants, and Please are all an attempt to redesign build systems so that the design is resistant to the flaws that plague Make build systems. You can use a shared cache (shared between different users) fairly easily, and you have reasonable assurance than incremental builds are identical to full builds (so your CI pipeline can move a lot faster, developers almost never have to "make clean" when they change things, etc).
Personally I’m working on a project right now that uses Bazel (which is similar to Please) and is for an embedded system. It’s been a great experience, and I can pass a flag to Bazel to tell it to build for the embedded target using the cross compiler or for the native host--that makes it easy to share code between tools and the target system, and I can do things like write tests that run on both the target & host. Anyone who does any cross-compiling is missing out if they are using Make—but, do note that setting up a cross-compiling toolchain in Bazel isn’t exactly a cakewalk.
https://docs.bazel.build/versions/master/tutorial/cc-toolcha...
No, it doesn’t. There’s NO syntax for multiple outputs. If you can show me what the syntax is and prove me wrong, I’d love to see it. At best, there are workarounds for the lack of multiple output support, with various tradeoffs.
> Make is a programming language with a built-in dependency mechanism, and as such you can use it to build whatever you want...
Make is NOT a programming language. End of story. You can… kind of… build things with it, given that variables in Make can be used like functions, but it’s goddamn horrible and you shouldn’t do it because nobody will want to use it and nobody will maintain it.
At most, you can build something on top of Make, but you’re still facing the limitations of Make and working around them. If you are interested in building something on top of a low-level build system, you want Ninja instead of Make, because Ninja is good at that. Make isn’t.
Make is, at best, something you would want to use for a small C program where all the files are in one directory. Once you grow past that use case, my rule is that you shouldn't be using Make any more, because there are just too many things that Make does wrong.
Make is successful because eh, it’s good enough for small things, you can suffer through the pain if you need to use it for big things, and it was the first tool to solve this problem. We have better tools now. We had better have better tools now! Make is in its fifth decade… if we didn’t improve on Make in that many years, that would be a damning indictment of the entire field.
(One point you don't mention in your list of reasons why Make is successful is that it's reliably available everywhere. For projects that ship as source for others to build, that matters, and it delays uptake of fancy new build systems in that segment.)
So do C, and Bourne shell. And SQL.
And yet, here we are. Are the benefits enough to overcome incumbency?
I would strongly recommend watching this talk if you want to understand why systems like Bazel and Please were created:
Build Systems with Andrey Mokhov (Jane Street)
So at the end of the day you end up with an executable artifact, but hopefully you did a lot less work to get there.
1. Reproducible/Hermetic: your builds cannot, say, access the network during a step or produce changing binaries for the same input. This makes sure that once you've downloaded your deps locally everything builds correctly from then on.
2. Caching: Because everything is reproducible in bazel-likes everything is cachable. Dependencies, builds, and even tests! If you've worked with Makefiles you've likely run `make clean; make tests` or something similar. I've never needed to do `bazel clean` to make sure things were working right.
3. Visibility: you don't only control who can see what in your source code. Different dependencies can be marked as private/protected/public to control who can import them. This is a huge boon to large monorepos.
4. Everything is uniform: code generation, compilation, etc is all described in "rules" and "toolchains" and can "easily" expanded to other languages. The community manages a Rust and Golang rules set for bazel itself and they're better then the Google rule sets for Java (only "officially" supported rule set right now) in some areas.
So if you have a lot of code/internal libraries/code generation, what to write a LOT of unit tests and cache their results, and write code in multiple languages bazel is probably for you.
You can also use tools like BuildBuddy [0] to share a cache with your team and extract build performance metrics to identify areas you can improve performance in.
[0] - https://www.buildbuddy.io/ or https://github.com/buildbuddy-io/buildbuddy
Without this tooling large organizations tend to silo which creates high engineering costs (every project starts from scratch, no shared frameworks/libraries, no standardizing around best practices, etc).
The advantage of a running, pre-existing system (and the theoretical advantage of an off-the-shelf system like this that you can supposedly turn into a running system) is that it's been built to satisfy most of the needs of a wide variety of developers working on all sorts of things, and so each engineer on each of those teams that says "All right, I'll spend some time thinking about how we build and release code" is incentivized to start from the common starting point, even if they have to customize it. That means that if two or three teams end up working on similar-enough projects down the line, the shared tooling can actually support communication between the teams. Without the shared tooling, they can communicate all they want, but it won't be rational for any team to abandon its existing tooling. (And of course you need the communication too - both are required.)
At the micro-level, tools help, no doubt about it. At an org level, communication helps (documentation is a form...). Even if using different tooling, concepts and strategies are the same. This isn't to take away from the argument that this tool the OP posted is pretty awesome.
If you want change, be the change. Evangelize the tool through your org, not just your team. If your org is silo'd, break those barriers down with a friendly lunch-n-learn or lightning talks session. We tout empathy for our customers, how about empathy for our coworkers too? Others may have the same pain...
There are already book-length arguments that I like to refer to for specific topics, such as trunk-based development.
Unfortunately, microsoft had to move away from VFS for Git to some a more limited-in-scope approach: https://github.com/microsoft/scalar
(Looking at https://github.com/jart/cosmopolitan it's an interesting hack. Might be useful in some cases, for example command-line tools like git.)
Have you made something that looks like a better make? If so, it would be nice to know more. Thanks.
The value proposition is also a bit more clearly defined there [2]
Another big difference is Please is written in Go so there's no dependency on VMs or runtimes.
In practice, I spend under two minutes to click around and see if can 'get' it. What does it do especially well? What's the syntax look like? If you are brave, what doesn't work well?
No idea why this exists or why I would use it. Bad website. No cookie.
What am I missing by doing that? This looks really slick, but I'm not sure how, why, or where to use it.
In your case, it seems you only have one non-trivial step: python dep install. So this system will be quite overkill for you.
Fast forward 5 years, now we're using Please to build all our code, generate hashes for docker images, template those hashes into our k8s .yamls, generate the documentation website from the docstrings in the proto files etc. etc.
With a language specific build system, you would have a lot of trouble handling things like this.
Yes, so why would I use Please over any of them? I've spent close to 10min reading and have no idea why this exists or why anyone would use it. It looks like Bazel with a different config format, in which case why wouldn't one just use Bazel?
However, I still have somewhere over 100 GB free on my seven-year-old laptop, and the download will take me ~40 seconds.
But even if the 600MB were a problem, Bazel doesn’t need an external version of the JVM. It bundles its own, and fits the whole thing in under 50MB. Some people are under the mistaken impression that you have to install the JVM first and then install Bazel, which is simply not true.
In my experience there is a strong correlation between executable size and how hard it is to unravel any issue that arises in it.
Bazel bundles the JRE, that’s why it’s 45 MB.
The correlation is kinda beside the point, though, because my experience with Bazel is that it’s easier to unravel issues with Bazel than unravel issues with Make, and Make is much smaller.
[1] https://en.wikipedia.org/wiki/List_of_countries_by_Internet_...
Most people don’t have the kind of bandwidth it takes to download a 50 MB binary? Are we still talking about Bazel here, because that’s the size of the download. What about tools like compilers? You have to download those, too.
> Running a background JVM daemon process is a NO for me and Bazel wastes system resources.
What system resources does it actually use up? A half-gig of RAM? This kind of optimization is penny-wise and pound-foolish. You are spending your precious time and energy worrying about a resource whose marginal cost to you is about the same as a cup of coffee.
I run Bazel on a terribly obsolete, seven-year-old laptop which is my daily driver. Sometimes Chrome will choke on a website, sometimes I'm waiting ages for Homebrew to update or for NPM to download some packages, but Bazel is not a problem.
I don't install tons of random garbage because I want to know and understand what is running on the hosts that I maintain.
I need to be able to debug the stuff I run, and complexity makes it difficult.
Tenths of MB of stuff is complexity. Tenths of build time or run time dependencies is complexity. A compiled tool when a script would have been sufficient is also complexity.
Bazel is way easier to debug than Make. Debugging a decent-sized build system made with Make is just an exercise in suffering.
I am… honestly… no longer interested in understanding the entire software stack. I understand my time on this earth is limited and want to spend it doing other things. With Bazel, I am spending less time fucking around with build systems and more time doing the stuff I care about.
> Tenths of MB of stuff is complexity. Tenths of build time or run time dependencies is complexity. A compiled tool when a script would have been sufficient is also complexity.
As someone who still writes C, I can understand the joy that people feel when you make some cool program and it’s measured in KB, not MB. However, what you’re describing strikes me as fetishistic.
"slimmed down"
lol.
I'm not suggesting Bazel is perfect: I think e.g. Starlark's insistence on being a separate language does more harm than good. (FWIW I also think the JVM objection is a little silly.) But I am saying preferring the Google tooling would ostensibly mean you like Bazel a lot already :)
I think the term DSL is overloaded here? Consider all the Lisp and Ruby stuff that's definitely DSL but doesn't need most of a language implementation.
Bazel has a lot of magic and crazy abstractions. Please is far easier to get your head around and as a result, far easier to bend to your iron will.
We wanted to switch away from Bazel because of the mismatch; it looked like Pants and Buck had the same issues. At the time Please didn't have sensible documentation so it was skipped. From what they have now it seemed like it's more reasonable in this respect, but I don't actually know yet; perhaps somebody who knows can chime in?
By custom rules do you mean macros, or full on custom rules?
I mean full on custom rules, exactly because of the inability for genrule to span more than one command (so we end up having to generate tiny shell scripts and running them).
I've also had issues debugging things any time I've tried to use genrule; the error messages tend to be unhelpful in my experience (though that's been a while and I don't recall the details). Things that look like they'd do what I want would just not work in unhelpful ways, and interrogating the system usually didn't work until there were no errors (at which point I wouldn't need to interrogate it anymore…)
What do you mean by a "command"? I'm very confused by the nomenclature here.
>All four of these systems are quite closely related in the scheme of things, being inspired by (or in Bazel's case, a direct open sourcing of) Google's Blaze.
>Several of us had worked at Google and used Blaze extensively there; we were excited about it being open sourced as Bazel but by then we were already using Please internally. It's a great system but we have slightly different goals, specifically we're aiming Please at being lighter weight and pushing the boundaries of what can be done within the BUILD language. Since Please is written in Go there's no runtime dependency on the JVM.
>We actually used Buck for some months internally before deciding to write Please and before it was capable of building our repo. We preferred it to other options available, but again we're focused on different goals; Please is easier to extend to new languages, has a bunch of features that we specifically wanted (e.g. test sandboxing) and has a stronger focus on BUILD language correctness. Conversely we have much less support for mobile development.
>We're least familiar with Pants; one of our developers briefly tried it and while we liked many aspects we didn't think it was the ideal fit for us at the time.
https://github.com/thought-machine/please/issues?q=is%3Aissu... https://github.com/thought-machine/please/issues?q=is%3Aissu...
What is Please?
Please is a multi-language build system designed for huge mono-repos. It was created by a couple frustrated ex-googlers who were familiar with Blaze (which was later open sourced as Bazel). We found the "real world" alternatives to be somewhat lacking and so Please was born!
Please draws inspiration on the Blaze paradigm. If you're familiar with Bazel, the biggest difference is Please aims to be simpler and have far less magic in the binary. We push the implementation of the build rules into the build language, dog feeding them to ensure Please is flexible enough for any task. Also Please is written in Go so doesn't require a JVM ;)
If you're not familiar with Blaze/Bazel, here's what all the fuss is about:
1) Hermetic builds: builds are run in their own tightly controlled environment. Each step of the build runs in their own temp directory isolating them from other steps and only having access to the files and environment variables they've declared as their inputs. Please also has sandboxing built in taking advantage of the linux kernel to further isolate tests.
2) Scalability through incrementallity: if you've used Make, you're probably familiar with caching problems. Make uses last modified timestamps on files to determine if they need to rebuild each step, which turns out is fallible. Please uses a hash based approach which is far more robust. Most of our developers don't even know how to clean the cache. As a result, we can incrementality build our entire repository locally and on our CI workers no matter how big our repo gets.
3) Flexibility: the build language is a dialect of python. This can be used to write "build definitions" which define a unit of work i.e. compiling a Go package. There's nothing special about the built in definitions; it's totally possible to write your own to automate nearly any part of your development process. You could generate code, template kubernetes .yamls and beyond!
4) Unified developer experience: The please command line provides a unified experience across your codebase. Want to test all the tests under a branch of your repo? `plz test //some/part/of/the/repo/...`. It doesn't matter what language you're using, what those tests depend on etc. etc. Please can always run them for you.
PS: Apologies for the website. We're a small team of build system engineers, not front end types. If you want to offer your skills, I'd be happy to point you in the right direction: https://github.com/thought-machine/please.
Please is a lot more lightweight than Bazel, so it’s easier to get deployed and work with different projects. The new maintainer has been shaving off a lot of warts lately and it’s getting a lot better.
If the above doesn’t describe your situation, then a tool like please, pants, buck, or bazel is probably not for you. And that’s ok too.
The idea of these build systems like Bazel is that the rules are correct so that you don't have to worry about writing correct rules, and you have a high probability of an incremental build producing a binary that's bit-for-bit identical to one from a full build. The result is that you don't do full builds anymore, and save the latency of waiting for things to build. (That latency shows up in the edit-build-test cycle, how long it takes to deploy software to production to fix an emergency bug, etc. So it's important!)
On that 30 minute note though: so, how big does the project need to be in order for Make not to be enough? And at that size, why wouldn’t the project invest the extra week it takes to get the Makefile correct?
In my opinion, this is a bit similar to languages with static vs dynamic typing.
If your current setup works well for you, it makes sense to keep it, though.
For me, I've found that the results are best when you use a build system and you use it exactly the way the author intends. For example, Go's built-in build system is so good that you don't even notice it's there. It automatically updates its configuration as you write code. It does a perfect incremental build every time, including tests. The developer experience is basically perfect, because it has a complete understanding of the mapping between source files, dependencies, and outputs. But, it is not extendable, so you're screwed when your Go project also needs to webpack some frontend code, or you need to generate protocol buffers (which involves a C++ binary that runs a Go binary), etc. So, people bolt those features on, and the build system becomes more general, but not quite as good. (Then there's make, which is as good or as bad as you want it to be.)
I think super small projects often do get their Makefiles right. But you can manually build small projects with something like "gcc foo.c bar.c -o foobar -lbaz", and so you don't really benefit from any improvement over the bare essentials. (Nothing wrong with keeping your projects tiny in scope, of course!)
But, sometimes you don't have the luxury of a super small project, and the Makefiles become quickly unfixable. Like I said, I am most scarred by a buildroot project I worked on (that's embedded Linux, basically). It never built what I expected it to build, and to test anything reliably I either had to yolo my own incremental build or wait a while for a full build. My productivity was minimal. I could switch between client-side and server-side tasks on that project, and so I really only touched the client if it was absolutely necessary. I would never be productive enough to undertake a major project that truly added value with that kind of build system, so I let others that didn't have the server-side experience write the client-side stuff. In that case, the poor build system silently cost our team productivity in terms of artificially splitting the team between people who could tolerate a shitty developer experience and those who couldn't.
I don't think anyone has fixed the buildroot problem, either. If you want to build a Linux image today, you are stuck with these problems. Nothing else is general enough to build Linux and the associated random C binaries that you're going to want to run.
It kind of feels like the best trajectory is to start small with a general build system, and upgrade as needed? And then if you are confident the project will grow, starting with the specific build system fine too.
I wasted a lot of time getting Bazel to use a modern JDK.
It does have some features to facilitate CI and testing at scale, e.g. `plz query changes` can be used to find a minimal set of tests to run for a PR.
You love to see it.
'build' kind of indicates that
If someone had the same idea, please don’t hesitate sharing.
> 'build' kind of indicates that
Not the GP, but as a developer reasonably conversant with build systems and their pain points, I still assumed this was going to be some form of lazyweb[0] related blog post or portal rather than a build system named 'Please'.
When I see someone giving such reasons for not using a more performant language, I immediately associate such project with janky, fast written codebase and disregard for proper, complete documentation. "Because you can just read and understand the code", yeah I can do the same in well maintained code of a faster language.
Arguably, making tools feel better to use is more important than raw performance. When you're doing a no-op build, the difference between 100 milliseconds and 500 is large, and the difference between 10ms and 100 is enormous.
Optimizing I/O performance certainly makes building software feel better though. Running make on a tmpfs directory feels absolutely amazing. It feels even better on subsequent runs since Linux uses free memory to cache files.
If you are executing 100,000 novel parallel C++ actions, then compilation for that might be I/O bound? If these actions are all cached though you would likely be CPU bound instead, as the build system works hard to discover that all the work is already done.
I would assume that is has the same advantage as most other Go programs: It's easily distributed as a single static binary, and doesn't need a heavy runtime to be installed.
And I want the binaries properly packaged with metadata, documentation, manpages to keep the host secure and tidy.
Which one you want to use in a given situation is contextual.
Murphy's law seems to dictate that the version of the build tool lovingly cared for by the OS's package manager is never the version you actually need.
Build tools in particular have a slightly different lifecycle to many things running on a host, which are "owned", cared for and configured by a system administrator.
Usually the project you're building "owns" the version of the build tool it needs and there's no guarantee that it will build with a different version. As a consequence, genuine build tool security issues must first be handled by the team making the tool and then by the projects depending on it when they verify compatibility.
Have shared libraries historically improved security much? Debian helpfully "updated" OpenSSL, and thanks to the magic of shared libraries, every program on a Debian system generated weak keys.
Personally, I don't think there's enough software on the average filesystem image to justify sharing libraries. Most stuff "in production" these days is in a container, which shares nothing with the other containers running on the same VM, which share nothing with the other VMs running on the same bare metal server. In the case of containers, a full rebuild from source is just as easy as updating a shared library -- maybe even easier. So the "quick security updates" doesn't seem to justify the high maintainability cost of shared libraries. (The high cost is having to maintain binary compatibility with applications you've never even seen before.)
The whole reason things like "Linux distributions" exist is for some central authority to coordinate and handle all the breakages this model causes. It's nice that they do it for us, but it's kind of makework to save a few bytes of network transfer now and again. Doesn't seem worth it to me.
> I want the binaries properly packaged with metadata, documentation, manpages to keep the host secure and tidy.
You've just invented containers. Many other people agree with you, which is why they became so popular.
I'm going to guess you never checked how many vulnerabilities have been fixed in Linux distributions before a release, or backported to existing releases: hundreds of thousands.
Cherry-picking a single incident (involving a library with a history of vulnerabilities) is hardly meaningful.
> Most stuff "in production" these days is in a container
It's provably not, outside of the SV bubble. The large majority of software in the world is still deployed traditionally.
> So the "quick security updates" doesn't seem to justify the high maintainability cost of shared libraries
A good number of large companies think otherwise. Also, people working in security.
>> I want the binaries properly packaged with metadata, documentation, manpages to keep the host secure and tidy. > You've just invented containers.
No, these are OS packages and predate containers by many decades. This is getting silly.
Of course it does. The runtime environment just happens to be installed alongside every Go executable by virtue of being statically linked to them.
Python is another thing. Most CLI utilities written in python are unusably slow.
https://news.ycombinator.com/newsguidelines.html
We detached this subthread from https://news.ycombinator.com/item?id=25240535.