Pants: Build system by Twitter, Foursquare, and Square | Hacker News Reader

Pants: Build system by Twitter, Foursquare, and Square

pantsbuild.github.io

82 pointskolev39 comments

Here is the actual blog post from today announcing a bunch of Pants milestones: http://pantsbuild.github.io/announce_201409.html

This is actually a collaborative project between Twitter, Foursquare & Square. All three companies contributed a ton of work.

Here's the announcement on the Foursquare Eng Blog: http://engineering.foursquare.com/2014/09/16/introducing-pan...

And here's the one on the Square Eng blog: http://corner.squareup.com/2014/09/trying-on-pants.html

It's a fully open source project so we'd welcome contributions (or jokes) from any and all interested folks! Pants is a great build tool that we hope to grow and use for years to come.

> This is actually a collaborative project between Twitter, Foursquare & Square. All three companies contributed a ton of work.

Thanks for pointing that out. We updated the title.

It's not just "by Twitter" anymore. Since about 2012, we've had some great contributions from other organizations. http://pantsbuild.github.io/announce_201409.html has some history.

Finally, a modern-day build tool! I thought Facebook would open source theirs first.

(this looks a LOT like Google's and Facebook's and Amazon's internal build config as well, btw)

> I thought Facebook would open source theirs first.

Well there is Buck, Facebook's Java and Android build tool: https://facebook.github.io/buck/

It's not really a generic tool like Pants is, but it's something. Thanks for sharing!

They're explicit about this:

"Pants models code modules (known as “targets”) and their dependencies in BUILD files—in a manner similar to Google’s internal build system" [1]

But I have to wonder why they choose to spell out "dependencies" :-)

[1] http://pantsbuild.github.io/announce_201409.html

I had a query regarding C/C++ support in Pants in the near future. Is that in the road map at all ? What does Twitter use internally for building such applications ?

It's on the "we'd like to get to that sometime" or "pull requests accepted" list. It's not on the immediate roadmap unless we get a contribution from someone that needs it. Internally we don't have so much c/c++ that people can't just use traditional tools.

Here's the original post: https://blog.twitter.com/2014/hello-pants-build

Basically it's similar to other build tools, except over half of the functions are specific to building Python and Java. Here are two lists of functions: https://pantsbuild.github.io/goals_reference.html https://pantsbuild.github.io/build_dictionary.html Hopefully they can figure out a way to organize those better so it doesn't evolve into a giant wall of functions you have to dig through to find something you want.

Their BUILD file format is not easily human readable. I know we have to compromise between whether the machine or the human has an easier time managing a config, but the build system is one of those i'd prefer be easier for humans.

Hopefully they clean up the user interface to make it easier to run, too. For example, you almost always have to type './pants goals <arguments ..>', where you'd expect an alias like 'pgoals'. Some kind of web interface would probably be useful for a lot of the included functions. This seems like a tool that belongs integrated with Jenkins or something similar.

The BUILD file format looks a lot like Python code calling some functions. http://pantsbuild.github.io/first_tutorial.html#build-files . It'd be great to have something more readable. For what it's worth, Google's been using a similar format for several years, and folks seem to find their way OK: http://google-engtools.blogspot.com/2011/08/build-in-cloud-h...

The word "goal" should disappear from the command line at some point. Alas, we're not quite there yet.

Source code: https://github.com/pantsbuild/pants

Claiming it supports "a variety of languages" is stretching it a bit. Currently it builds the 2 biggest JVM languages and Python. I like where they're going, and if they truly manage to support a wide variety of projects in the end, it'll be fantastic, but for now the claims are too bold.

Can't wait for someone to add C++ (llvm and g++) support!

As Americans continue to release more and more tools with names like 'Pants' and 'Git', it really does start to look like they're just trolling British developers.

Linus is Finnish, not American. And I suspect he's aware of what it means, given the quote "I name all my projects after me: First Linux, now git."

I don't want to learn another build system. why would I?

What a strange attitude.

why?

What problem does this solve that hasn't been solved many times already?

The blog post goes into details, but the main advantage pants has over other build systems we could have used is that it is designed from the ground up for a large codebase that is broken up sensibly into modules, libraries, or projects. It is smart about what it builds (or more importantly: what it doesn't build) and operates from the conception that most of the code you work with is available to be changed in any given commit. This is very much not the style that other build systems support, where trying to work in this way either results in a huge, undifferentiated pile of code or forces you to completely break things up entirely (ie work on stable dependencies). Once you have a significant number of engineers, this becomes inefficient whichever way you go -- either by giving you long build times or by making it unnatural to work outside your project.

We are a team of 25 engineers working on the same sbt based project. I don't see why we would move to pants. Do you have any non-marketing reason for why Pants is better than sbt?

Maybe, maybe not. As your codebase grows, it's likely to turn into separate parts with some common code. Pants can get you faster incremental builds: quickly figuring out what needs building, building just that. If sbt is quick for you, you might not have hit the point where Pants helps.

If you reach a point where sbt gets slow and you're looking at other build tools, that's a good time to look at Pants.

What concerns me is that it doesn't seem possible (based on a quick overview) to modify the default lifecycle, eg, to avoid running integration tests by default. How would you approach this problem?

When you invoke Pants, you pass in "goals" (roughly, the verbs/actions you want to take) and "targets" (roughly, the things you want to act upon, often piles of source code).

If you usually invoke './pants goal test bundle src/my/awesome/project tests/my/awesome/project' (to run tests and bundle up an app), to skip testing instead invoke './pants goal bundle my/awesome/project'.

But in this case I don't want to skip all tests, just the slow ones. Or say I want to add a goal to generate compressed CSS. Can I do that?

To skip slow tests (except when you want them), you put them into separate targets and then don't invoke them on the command line (except when you want them). At the risk of tossing you in the deep end with a link to a big build config file, https://github.com/pantsbuild/pants/blob/master/tests/python... shows some Python tests that have been grouped into test suites. One of those test suites is named "integration". When folks want to run the fast tests, they invoke on tests/python/pants_test/targets target; when they want the slow tests, they invoke on tests/python/pants_test/targets:integration.

Yes, you can add a goal to generate compressed CSS. The docs are, alas, still skimpy, but they're getting better; meanwhile, they give an inkling: http://pantsbuild.github.io/howto_customize.html

That's kind of what I was looking for. Thanks.

Fine grained dependency management for large projects that large teams work on.

I only skimmed through the documentation, so I must have missed something, but I fail to see how it can be better than either redo[0] or tup[1] in either speed or dependency tracking.

[0] http://apenwarr.ca/log/?m=201012#14 - designed by djb, implemented by apenwarr.

[1] http://gittup.org/tup/ - the fastest build system you'll ever meet, with an essentially perfect automatic dependency tracker

shrug Haven't messed with redo or tup. Maybe they do great dependency tracking as well. It's not a new idea. Pants is basically a copy of Google's build system by a bunch of ex-Googlers.

Pants also does nifty things like artifact caching between members of a team which is nice.

It will continue to do more nice things in the future as it's under active development by a decent number of people at this point.

tup does perfect instantaneous dependency tracking: It monitors file system access to see which files were accessed during build (you don't have to tell it). It does not depend on tool co-operation, and it's the only build system that will figure out that you've upgraded your compiler (e.g. if javac or gcc was replaced) and thus a rebuild is in order.

redo is explicit; it doesn't automatically do dependency tracking, you have to tell it what went it. But it's fast and incredibly elegant - it's extremely easy to make it track any dependencies you know about, and it's built for easily rebuilding just a part of your system.

I'm disappointed to see that like many build systems, they're inventing their own backend instead of just using Ninja (http://martine.github.io/ninja/)

Well, maybe they weren't aware of it. I wasn't. Ninja looks nice! Thanks for sharing!

Another perspective http://learningtosketch.tumblr.com/post/84319786305/ity-at-m...

Shame about the name

Python Ant, seems apt.

Well, at least it's not underpants. :D

Twitter is going to make the internet a better place.

Reply on news.ycombinator.com