emelski

Hey, I created ElectricInsight! It's nice to hear somebody besides me remembers it fondly.

Information is Beautiful did a visualization a couple years ago that attempted to answer this exact question. It hasn't been updated since end of 2020 but perhaps this will still be useful to some readers here:

https://www.informationisbeautiful.net/visualizations/snake-...

But unless you're already using Bazel, it's impractical to suggest "just use Bazel to distribute your builds" as doing so requires that you first migrate your build to Bazel. distcc, as noted, can be applied to an existing build regardless of the build tool used.

I found that the LPS lighting in San Jose was very close in color to yellow traffic signals, which made it sometimes confusing (and therefore more dangerous) to drive there after dark.

Sorry for the belated response. Electric Make is part of CloudBees Accelerator: https://www.cloudbees.com/cloudbees-accelerator.

Electric Make has had sandboxing since 2002, no conversion from your familiar make-based builds to a new shiny build tool required, and it can make on-the-fly corrections to execution order if that sandboxing reveals that incomplete dependency specifications caused something to run in the wrong order (relative to a strictly serial build).

"Sandboxing" in a build tool cannot be claimed as Bazel's innovation.

Correct, both the technique described in that article and the feature that eventually wound up in GNU make only disentangle output from concurrently executing build processes. With significantly more work GNU make could probably be made to enforce a deterministic order.

However, Electric Make does emit the output in a deterministic order, in fact in exactly the same order as the build would have produced had it run serially. In practice the delayed output is not such a big deal -- people just don't really seem to care that much, when the build overall finishes 20x - 30x faster than it used to. Electric Make also generates an annotated build log, essentially an XML-marked-up version of the log, which contains a tremendous amount of additional information about the build and vastly simplifies debugging, actually.

If you're looking to serialize parallel build logs without changing to an entirely new build tool:

1. Electric Make, a high-performance reimplementation of GNU make and ninja, has properly serialized build output logs since it was introduced in 2002 (https://electric-cloud.com/products/electricaccelerator) (disclaimer: I'm the chief architect for ElectricAccelerator, of which Electric Make is a component).

2. I published a technique on CM Crossroads you could use with GNU make 3.81 to descramble parallel build logs in 2009. The article has moved around since then but these days it seems to be found at https://www.cmcrossroads.com/article/descrambling-parallel-b....

3. The maintainers of GNU make took the concept described in that article and baked it into GNU make itself in 2013 for version 4.0 (http://git.savannah.gnu.org/cgit/make.git/tree/NEWS?h=4.0)

Nothing in the linked 14-page window replacement guide has anything at all to do with safety in general or windows falling onto people. It is entirely to do with preserving the "historic character" of buildings. I don't think you can reasonably equate that to the regulations governing the design of caps on baby food.

Unfortunately very many people casually lump a huge range of maladies under the umbrella of "RSI" and treat them as if they all have the same root cause and the same remedy. Wrist/arm pain could be carpal-tunnel syndrome, or it could be tendonitis, or it could be any of a bunch of other things all of which are technically "RSI". For some, strength training can certainly help; for others, as in your case, it may do more harm than help.

That's true if you limit yourself to, say, GNU make -- but there are GNU-make-compatible alternatives like Electric Make, part of ElectricAccelerator (http://electric-cloud.com/products/electricaccelerator), that add features like ledger, to trigger rebuilds when compiler flags change; filesystem monitoring for truly accurate dependency detection; and parse avoidance, to avoid reparsing the makefile on every run; as well as a long list of other enhancements.

Disclaimer: I'm the author of TFA and Chief Architect for Electric Make.

I think when you say "make Android builds faster", you mean "make Android application builds faster" -- as opposed to making Android operating system builds faster. Those are two very different things, and for the uninitiated the casual use of language here is confusing. The Android operating system has never been built with ant, but historically was built with make until around Android N when that team started migrating to ninja-based builds instead.

This is the fourth post in the series, not the start.

Previous posts:

http://themacro.com/articles/2016/09/introducing-ask-a-femal...

http://themacro.com/articles/2016/09/ask-a-female-engineer-2...

http://themacro.com/articles/2016/10/ask-a-female-engineer-3...

Hard links as well, which are nastier because they introduce aliases for the things you want to track.

memoize is certainly not the first use of this technique. Electric Make (http://electric-cloud.com/products/electricaccelerator) uses a custom filesystem to track file accesses for purposes of augmenting dependency information, and predates memoize by five years. It even goes beyond simply tracking accesses to using the information to correct for missing dependencies on the fly.

The main problem with strace is not the printing (although that doesn't help) but with the ptrace technology underneath, which basically hits the traced process with a SIGSTOP/SIGCONT pair on every system call, as well as a context switch to the tracing process and another back to the traced process. Even a no-op ptrace-based monitor that does nothing will make individual system calls ~10x slower. In my benchmarks, the best case overall performance impact was about 5%, but some processes were as much as 560% slower.

LD_PRELOAD is faster but has other deficiencies, like it's trivially easy to circumvent the tracing by wiping LD_PRELOAD from the environment before starting a new process. It can also be tricky (though not impossible) to manage implicit state, such as following an application as it first chdir's to a new location, then accesses paths like "../../include". LD_PRELOAD is also tough to get right in the face of multi-threaded applications.

FUSE is interesting but so far I've found the performance to be disappointing. I think that's mostly because it bounces everything through userspace and effectively doubles the filesystem activity for (nearly) everything. For example, with a normal filesystem a read from a user process basically works like this:

user process -> read system call -> filesystem read operation -> return result to user process

With a FUSE filesystem it's something more like this:

user process -> read system call -> FUSE filesystem read operation -> FUSE userspace driver -> read system call on real filesystem -> filesystem read operation -> return result to FUSE userspace driver -> relay result to FUSE filesystem -> return result to user process

There are various caches in place to make this less disastrous than it seems on the surface, but fundamentally this is the architecture. For some applications that's fine; for high-performance build tools I think it's probably a deal-breaker.

This is why we wrote a custom filesystem for Electric Make, and why that's still the approach we take today, nearly 15 years later: nothing else is as robust, and nothing else comes close to the performance.

_Disclaimer: I'm the architect of Electric Make_

I'll take any excuse for a little code spelunking -- according to Perforce it was Scott, in October 2002.

I shouldn't be surprised that you published that before I did -- to be fair, it's quite a challenge to find _any_ topic related to make that you _haven't_ written about. :)

You can actually articulate that relationship in GNU make, as long as your outputs can be coerced into a pattern rule (like "%.c %.h: %.y"). Otherwise there's no way to do it that's reliable in the face of both incremental and full builds. There's been discussion of adding syntax to GNU make for this purpose on-and-off over the past several years, but nobody can seem to agree on what the syntax should be.

With Electric Make you can mark the rule as producing multiple outputs by adding "#pragma multi" above the declaration. See http://blog.melski.net/2013/01/01/pragma-multi-and-rules-wit....

Disclaimer: I'm the Chief Architect of Electric Make

_make_ was first released in 1977, but that was the PWB/UNIX version. The GNU variant of make didn't come along until sometime in the 1980's. It's hard to pin down the exact date of the first release because the developers didn't keep good records prior to the switch from RCS to CVS. However, the earliest ChangeLog entry now is dated July 15, 1988, and 1985 is the earliest date mentioned in any copyright statement in any of the GNU make source files.

ElectricAccelerator is predominantly written in C/C++, with some Java for the cluster manager server component. As of the 7.1.0 release, it's about 375KLOC in C/C++ (excludes comments and blank lines) and 100KLOC in Java (excludes comments and blank lines). Compiling and linking from scratch takes about 29 minutes serially, but only about 2 minutes when I use ElectricAccelerator to build it (with a cluster of 16 worker agents).

We use ElectricCommander to orchestrate our CI process:

* checkout from Perforce

* build (which uses ElectricAccelerator)

* run unit tests

* build installer

* install on test cluster

* run integration tests

I'm the architect of ElectricAccelerator (http://www.electric-cloud.com/products/electricaccelerator.p...), a high-performance replacement for GNU make. I use it every day for all my builds, including builds of ElectricAccelerator itself.

Depends on which version of "make" you're using. GNU make does not, but automatic dependency detection is just one of many enhancements in Electric Make, a drop-in replacement for GNU make.

Of course, you can both master a niche _and_ build awesome products and have a good time -- the two are not mutually exclusive as your comment implies.

The point is that your backups would be replicated to blobs on other people's hard drives, so you'd still be able to recover your data.

It's probably apocraphyl: http://www.snopes.com/weddings/newlywed/secret.asp.

Twitter's patent application was filed in 2008, so no, this is not related to the recent legislative change.

Does anybody know if the plan is to build atop the Shoreline Preserve in Mountain View? That seems to be the only large parcel of land that is both closer to the bay than Google already is and not currently occupied by other structures. Seems a shame to wipe out a hefty chunk of a nature preserve simply to satisfy a corporation's desire for self-idolatry.

No, it's not wrong. The whole idea that it might be reeks of elitism. Use whatever theme you want, and let the "members only" Svbtle clique be judged based on the quality of their content, rather than their association with some self-professed blogging luminary.

What build tool are you using?

To me, those are not even remotely comparable, so you can't really use the same criteria for evaluation. On one hand you're talking about allocation of funds -- it's just money. On the other, you're talking about depriving people of fundamental rights.

When it comes to the latter, the litmus test I use is, "Is there a clear and urgent need for this law?" In this case, as horrible as human trafficking is, I do not believe that prop 35 passes muster on those grounds. Bear in mind that we're not talking about going from "no penalties for human trafficking" to "some penalties for human trafficking". It's going from significant penalties, to even more severe penalties. There is no urgent need for this law, and therefore no justification for skewering civil liberties.