Repobuild – Declarative-style build system, like Google Build | Hacker News Reader

Repobuild – Declarative-style build system, like Google Build

github.com

75 pointsalbertzeyer30 comments

Well, 1) I wish him all the best and would love another good build tool alternative, but 2) I don't see his utopia of demonstrably making open-source building easier happening because the OS world is a bazaar, not a Google-esque cathedral.

Google's build system is awesome because it can force everyone in the company to use it, going so far as to put every single project in one huge repository.

And, then, crucially, the model assumes that any dev workstation can compile (or pull down via cache) every single upstream project.

In the non-catherdal/bazaar world, take any non-trivial Java project, and all of it's upstream dependencies, and some are built with Ant, some with Maven, some with Buildr, some with Gradle, some require thrift installed, some require who knows what installed.

And you can't just go back and add some meta-data of "oh sure, every Java project compiles src/main/java -> target"...their build files are going to have all sorts of crazy one-off steps (Maven plugins, whatever).

There is no enforcement of "you must make your build use this BUILD file, and, btw, only program in Java, Python, or C."

With all of Maven's warts, the jar-based repository with transitive dependencies is, IMO, it's major innovation. I just don't see a transitive upstream build-from-source model (like Google's) ever working in the bazaar that is Maven central.

I don't at all mean to say Maven/Maven repository are perfect--for one, it's ridiculous how many broken dependencies are in Maven central. Naively, I had thought that was automatically enforced/not allowed, but, sigh, no, we all have to deal with, say, the ASM developers completely breaking their API from v3 and v4 and forever cursing the rest of us with jar hell.

Agreed.

Transitively-resolved, platform-independent binaries are the big win coming from maven. Only once have I worked with a project where I required new system-level dependencies to compile & run a maven project; and I have never had a cross-platform issue when carrying projects across Mac & Linux.

To be fair, the import-compiled-transitive-dependencies-as-binaries pattern that Maven likes to employ is probably less easily adapted to natively compiled languages like C/C++ than it is to managed runtime languages (Java/Erlang) or scripting languages (Ruby/Python).

As much as it's a pain to work with "maven plugin spaghetti", it's at least deterministic and keeps things easy to figure out.

There are those dependencies that you learn to remain wary of (asm/cglib/antlr/jline/etc), but at the end of the day I'm happy to work in an ecosystem where I don't have to compile every single line of third-party code myself.

> To be fair, the import-compiled-transitive-dependencies-as-binaries pattern that Maven likes to employ is probably less easily adapted to natively compiled languages like C/C++ than it is to managed runtime languages (Java/Erlang) or scripting languages (Ruby/Python).

The "scripting languages" don't even have this as easy as it might seem. Half the dependencies in my Python projects these days have important bits written in C (e.g. psycopg2, gevent).

Some significant work is being done in this area by the Open Source Robotics Foundation, which develops and maintains ROS. The ROS build and packaging systems (http://wiki.ros.org/catkin and http://wiki.ros.org/bloom, respectively) are built upon CMake, but a major design objective is enabling the integration of substantial third-party components which do not necessarily comply with ROS packaging rules.

It seems to work quite well; ROS C++ software can happily depend on boost, bullet physics, PCL, KDL, OpenCV, collada, qt, etc. There are over 800 packages available as debs in the current ROS distribution: http://www.ros.org/debbuild/hydro.html

You're right. I think this goal is difficult if not impossible given the way things are.

Missing, I think, is the requirement that open source projects are written with something like Repobuild in mind - consciously choosing to be part of an Open Package Build Network.

It does seem to run counter to open source to standardize around a build system. However, the power unlocked by such a build system (as alluded to in the OP) might be reason to make the move. Who wants to tinker around with the 20 different build systems each of your dependencies uses?

In other words, if the open source world starts moving in the direction of a standard build system, gaining momentum, it seems to follow the open source world can reach further in its endeavors, tackling more and more complex problems.

Hi,

Yes, you are completely correct, the current reality is a total mess. Repobuild (as it works now) assumes a lot of things which may not be workable. Mostly I just wanted to see what would happen :).

One of the main goals here is to allow folks other than the primary author to write a BUILD file, e.g. to back-port old systems. As a consequence, it has to work with other build systems and wrap them.

And yes, Maven does this pretty well with the centralized repository. I would like Repobuild to wrap Maven, though I haven't yet written that piece.

Interesting, if Chris can pull this off it will change the world in a huge way. Having used the build system inside of Google it was somewhat mesmerizing, you really could put together things pretty easily. I suspect there is something similar inside Amazon. But I completely agree that Github has done wonders for making many smaller projects accessible it has not significantly increased our ability to tie them together. That seems to be part of the motivation for the CCAN effort as well, although on a more language specific vector. But as a proof of concept the CPAN system (the one for perl) demonstrates just how powerful this function can be in enabling large scale rapid development.

The flip side though, the monkey in the wrench as they say, is that the projects that participate have to conform to the rules of participation. In CPAN's case its a standardized dependency, build, and install models with required unit tests. So perhaps the first step here is to provide an incentive for clean build processes on a project.

Providing incentives seems key. I wonder why CPAN came to work so well.

Possible incentives:

  - automated testing (TravisCI does this in isolation already)

  - automated distribution (build .deb, .msi, and .pkg files)

  - usage statistics?

I'll have to think about this.

If you come to any conclusions, let me know :).

Happily =)

>Having used the build system inside of Google it was somewhat mesmerizing, you really could put together things pretty easily.

I found using Leiningen for the first time to be a similar experience. I like it so much that I use it on non-Clojure projects too.

To expand on that, I was thinking about some standartization of sources (BUILD file in this case), standartize the way build tools could be plugged in, and also standartize the build outputs, so that they can be correctly included AND linked even without the sources.

Then you can build a lot of things on top of that, not only source sharing and automatic dependency resolver, but also remote build servers, sharing already built libraries to save time, and easier integration of various editors, IDEs and other tools.

This is really exciting - as a Java/Python guy who recently made forays into C++, I was disappointed at the state of dependency management and the lack of standardization across build tools. So many projects use submodules instead of cleanly declared dependencies.

I hope that a project like this can motivate the C++ community to provide some more centralized support for creating shared libraries, managing third-party dependencies, and other areas where the language falls short today.

You might find the blog post http://skalrant.blogspot.com/2013/05/build-systems.html interesting. It argues that a declarative style is not suitable for many software development projects. Coming from the author who is also the author of an ML like language it makes for a good read. Declarative is surely elegant, but not always easy to work with under specific and not uncommon situations.

Note, however, it touches on an orthogonal aspect of build systems. I think Repobuild's main thrust is in standardizing and assisting modularization whereas the post linked in this comment has a different focus, it argues for a non-declarative, fixed-point based style.

Those are the same link. Can you please update one of them? I'd like to read both that you mentioned :)

Judging by his description of BUILD, it sounds like was a big influence on Go's build system.

Well, no surprises there!

More likely Go's build system is influenced by Google's build system, which came first.

That's what I said. :-) BUILD had an influence on Go. Go is influenced by BUILD. One sentence is in the active voice, the other in the passive.

There is something I don't understand. You can add other project code to your own project, but how do you use that other code? In the .NET world where I live you can reference an Interface and let runtime fill that interface in (with a proper dependency injection method).

For example you know you want an Interface of MailSender that has the method .SendEmail() and it gets filled with SMTPMailer which has SMTPMailer.SendEmail()

Is this thing similar to that concept but in build-time? How do you know the interface/usage beforehand?

Most open source libraries are written in C/C++. The interface is usually described in header files, you need them to compile your program. The implementation itself is contained in a library, which you link into your program. There is also the possibility to resolve the implementation at runtime via shared libraries.

"reference the Interface" requires that the interface files exist, and that they (plus the implementation) gets built into your project. That is what Repobuild covers.

The actual mechanics of the .net runtime would stay the same.

There's a building collection of small pieces of Javascript code automatically fetchable from the browser in http://www.corslit.com/. Does it sound like it? I think it should.

Explanation here: http://williamcotton.com/another-way-to-publish-code

This looks great! May be a reason for me to ditch CMake, but I kind of need Windows support :/. Is it much work to add?

I wish someone made a similar way to share libraries like we can do in PHP (https://packagist.org) or Ruby (http://bundler.io/), but for C++.

I'd suggest to look at the freebsd ports build system. It is essentially declarative being built on top of Make language, and allows easy integration of various open source components and dependencies.

If targeting multiple platforms is of an essence, NetBSD pkgsrc provides a great model also built on top of Make.

I'd be curious about how this compares to the Buck project, which is another Google-like build system:

http://facebook.github.io/buck/

It's very similar. Here are a few other similar ones:

https://github.com/chrisvana/repobuild/wiki/Similar-Build-Sy...

A few differences:

* Buck requires installation (meaning clients need to have it installed), whereas Repobuild generates a Makefile.

* Buck has a spiffy build daemon to make things faster.

* Repobuild auto-install files from git.

* Buck covers java better, but not other languages.

* Buck went with a turning complete language, Repobuild intentionally went with only a config langauge (hotly debated religious difference)

Author here (a bit late to the party).

I'm happy to answer questions if anyone has any.

How hard it would be to add Windows support?

EDIT: Just a question, not a request.

(I have not investigated it fully, so this may be inaccurate)

Cmake:

Repobuild basically generates a Makefile. You could change it to generate a CMakefile instead. That would not be a lot of work, probably a few (full) days to change and test it.

Visual Studio:

Generating a visual studio project file may be trickier. A lot of the work overlaps that of the Cmake stuff above (changing the output format). Realistically, a 2 week project (I would do xcode projects at the same time).

Ok, good info, I might try it.

What I know for sure, CMake is not very adequate for separating work into different small, independant parts (somewhat "declarative" attept in CMake here: https://github.com/key-tools/key-machine-logs/blob/master/ke...), and then including build targets from other CMake parts into one.

Also, generating CMakeLists, then generating a project seems like an overkill :)

Reply on news.ycombinator.com