Hacker News Reader

Comment by phasmantistes | Hacker News Reader

Chromium continuous build infrastructure engineer here.

As he says, this is slightly out of date: we forked WebKit to make Blink in 2013, and actually merged Blink into Chromium proper just this week! It is no longer managed by gclient/DEPS, but instead developed alongside Chromium in the same repository. But many other things (pdfium, v8, webrtc, and tons of smaller libraries) are still pulled in via DEPS in exactly the way described.

I'd just like to say this about our development, testing, and release process: it really really works. We keep moving fast, everyone knows what's going on throughout the codebase because everyone is working on master, and there are never any big ugly merge conflicts to resolve when new features land.

There are, of course, problems with the system. We don't have enough resources to run every test for every commit, so they end up batched and we have to rely on humans and heuristics to determine which CL in the batch to revert. The set of configurations on the try-bots doesn't exactly match up with the set of configurations that are run against tip-of-tree. And flaky tests are, of course, a perennial problem for everyone. But we're working on making all of these better, and huge progress has been made in the last couple years.

Ask me anything if you have any questions about our build and test infrastructure!

View on HN

This is only tangentially related to the topics covered in the article and your comment, but I'm curious about the Chromium project's decision to mostly use static linking.

I've always found it interesting that Chrome is shipped mostly as one big binary (or on Windows, two: chrome.dll and chrome_child.dll), whereas IE and Safari use several (one for the rendering engine, one for the JS engine, one or two for networking, etc.). My guess is that Chrome uses one big binary to minimize startup time, since the OS can just map that binary into memory and go. Am I correct about the rationale? Another way to ask the same thing is this: what would be the downsides of using a component build for releases and not just for development? If you used a component build for releases, you'd have greater parity between dev and production, and that's generally a good thing.

I also notice that there are a few modules that get their own DLLs, at least on Windows. For example, in Chrome 45, we have chrome_elf.dll, libegl.dll, libexif.dll, and libglesv2.dll; I'd think these libraries could be statically linked just like the few hundred others. Are there good reasons for these exceptions? Or are these just little details that tend to get overlooked given the rapid pace of development?

Since all I've worked on for the last three years is the infrastructure, and not the binary that lands on your desktop/phone, I'm not really qualified to answer that. (Which is just a fancy way of saying I've got no earthly idea :))

But since I've seen @bengoodger in this thread, maybe he can answer your questions.

I can't speak to the current situation, but originally Chrome was one monolithic dll for exactly the reason you describe: we wanted renderer launches to be as fast as possible and having the code mostly already paged in helped a lot. Over time it became impossible to link and so we had to switch to having a separate renderer dll.

I'm not sure why there are now a handful of separate dlls.

Like Aaron mentioned, the renderer was originally split into a separate DLL because we hit the limits of Microsoft's linker at the time. Later, we ended up relying on that split for disabling win32k inside the sandbox (otherwise we would have had to deal with a bunch of broken USER and GDI initializations).

As for the remaining DLLs, it mostly comes down to licensing concerns. We don't include proprietary code in our main binaries, and in order to simplify IP concerns it's often best to split such code into other DLLs. For example, you asked about libegl.dll and libglesv2.dll, which I believe include a licensed copy of the SwiftShader library.

It's also worth pointing out that the performance impact of having a single, static binary in Windows is much less significant than it used to be. So, were we starting Chrome today I'm not so sure we'd make the same decision.

Being Google, what exactly keeps you from running every test for every commit? Maybe I am being a bit naive here, but as the owner of multiple data centers across the globe, shouldn't you have enough resources to do that, even for a project as big as Chromium?

Many but not all things can be usefully parallelized. The build, and most unit tests and integration tests can, though we struggle with the latter where real hardware is required. Some performance tests are also run on real hardware. This doesn't scale as well.

Chromium is open source, and will run try jobs for anyone from anywhere around the world (we won't accept your commit without lots of checks, but we'll run your tests). That means that we don't get to rely on a lot of Google-internal technology, including our production data centers.

Also, we are compiling and testing on Mac, Windows, Linux, Android, iOS, ChromeOS, and historical versions of all of the above. That's something that a mostly linux-server-based company doesn't do a lot of. So we maintain our own datacenters with our own hardware, and don't get to take advantage of all of Google's scale.

I'm curious whether you've got any interesting projects underway to work on the flaky test issue? I've been looking at FBs test lifecycle approach with some interest (http://goo.gl/QtUNxY), but would love to hear about any other approaches to solving the same problem.

And FWIW on building MongoDB with Evergreen (https://evergreen.mongodb.com) we've found that a stepback approach seems to give us the finest granularity with minimal cost in identifying the introduction of an error - we batch commits for execution, but when a specific test fails we start running just that test on each previous commit until we hit a passing execution. It obviously doesn't work perfectly in the face of flaky tests (see above question) but it seems to do pretty well.

We are working on a few things.

The first is automatic retries of fast tests. If a test runs quickly and fails, it costs us little to try again just to make sure. Most of our unit tests are configured to run up to 3 times.

Another thing is keeping track of a database of individual test case passes/failures across all time. This will let us automatically mark tests as flaky if they fail often, and ignore their results programatically rather than requiring a human to manually mark the test as ignorable.

A third thing is, obviously, automatically filing bugs against the owners/authors of tests which have been marked flaky. This is controversial -- often a test is just fine until one of its underlying libraries has a race condition introduced, and the real person to fix it should be the author of that change, not the author of the test. But it is still a step in the right direction much of the time.

Many people subscribe to the philosophy that "a flaky test is worse than no test", because you think it is giving you information when in fact it is giving you none. I subscribe to a slightly different philosophy: "A test with a known flaky rate is hugely valuable". If you know how often a test flakes (statistically), then you can measure variances from that rate to detect changes. Of course, a flaky test with an unknown rate of flaky is still useless. Hence the second initiative above: measuring the rate of flake of everything.

Mozilla has a big problem with flaky tests (aka "intermittents") too. The SpiderMonkey VM can run tests in a deterministic mode to make tests less flaky by making things like GC more predictable.

Firefox has an experimental "chaos mode" that takes the opposite approach. It purposely randomizes behavior by adjusting thread priorities, changing hash table iteration order, and randomize timer durations. Unfortunately, many flaky tests fail in chaos mode, so it is not enabled by default.

http://robert.ocallahan.org/2014/03/introducing-chaos-mode.h...

But we (Mozilla) are also doing many of the same things as the Chromium team here. In particular there is work in progress to automatically "ignore" the results of known-flaky tests until we detect that there has been a change in the rate of flakiness, at which point we will — assuming all goes to plan — trigger new test runs until we can determine the point at which the regression was introduced.

I think one of the lessons we've learnt is that with a browser-type project it's very hard to make test runs fully deterministic, for both technical and human reasons.

The technical reasons are touched on in the original article: these are complex codebases with lots of moving parts and lots of environmental dependencies. Of course there are various tactics to try and combat this; for example there is a wiki page dedicated to innocuous-looking code that leads to intermittent tests [1].

The human reasons centre around the difficulty of getting people to care about spending time fixing a test that fails one time in 1,000 (which is still very noticeable when you are running it hundreds of times a day). Unless the issue is something that fits a known pattern it's hard work, difficult to tell if your fix even worked, and not likely to be considered a top priority due to the diffuse, hard to quantify, nature of the benefits.

I think the fact that both Google and Mozilla still have significant problems with intermittents despite talented engineering staff and it having been a known problem for years implies that some of the standard thinking about making tests fully deterministic simply doesn't apply; for this kind of work you have to embrace — or at least accept — the randomness, and look for ways to get the data you need despite the noise.

[1] https://developer.mozilla.org/en-US/docs/Mozilla/QA/Avoiding...

That's a good point that making tests fully deterministic is not actually possible. But users aren't deterministic either, so we must accept noisy test environments because that's what users see. Tracking changes in the rate of flakiness is an interesting idea.

Could you run "all" the identify flaky tests by running all the tests 100 times on the same stable build (like the latest ESR)? Is it even possible to write a test that could pass 100 times in a row? :)

Running a test N times will certainly detect some fraction of all the flaky tests. It's something we occasionally do manually to work out if e.g. a certain intermittent is (likely) fixed and it's something that we'd like to do more to quarantine new tests.

Unfortunately there are various confounding factors that mean many intermittent tests would look clean in such a run might nevertheless be problematic. For example if you only run tests that you think are intermittent problems that are triggered by state left from a previous test won't be found. This is one reason that we've been trying to run particularly problematic test types (e.g. firefox browser-chrome tests) in smaller groups restarting the browser with a clean profile between groups to clear the state. A group size of 1 would obviously be ideal here, but when you have thousands of tests and limited resources it's not practical.

The other problem is tests that have unexpected sensitivity to the environment. For example the other day DNS was being slow on the test infrastructure. This isn't a problem for most tests since they use something like /etc/hosts. But some tests were intentionally trying to use a non-resolving domain and those tests sudden started to randomly time out.

I have to dealt with debugging flaky tests before on embedded Linux app with multithreads, GUI, audio, video streaming before.

I debugged those bugs with custom gdb scripts. The test system will run the gdb scripts to start the app. The gdb script setup breakpoints, log stack trace and local variables automatically for a particular bugs.

The debugging environment is scripted and automated. Test failure mark/track the log file with all the gdb info.

I schedule the tests to run continuously overnight and in the morning I analyze the log file with gdb info.

After that I can add more debug command in gdb script or try fixed the bug if I have enough info. The cycle continue until the bugs were all fixed.

I personally think it was very good for try to analyze / fixing the "flaky" tests issue.

Even for the the tests that fails 1 in 1000 times, the bugs can still be debug and fixable.

It was actually more relaxing for me to debug this kind of problem. 99% of time, the computer do all the works.

You guys might want to try it if not done in Google already.

At Mozilla we're actually currently working to run some of our tests continuously under RR (http://rr-project.org/), our record-replay debugger. When a flaky test fails the replay can carefully studied to understand why test only fails say 1 out of 10,000 runs.

You say you have to rely on humans and heuristics to determine which CL to revert. Couldn't you automatically re-run just the failing tests on the batch?

More importantly, you seem to be running all the tests, as often as you can. A commit changing a text label shouldn't require running all the tests, that just doesn't scale. Have you consider splitting Chromium into different projects, where commits for one project would result in the tests of that project, and only that project?

We actually are working on systems which re-run the failing tests on all CLs within a failing batch. Unfortunately, we haven't been able to do this previously due to resource constraints. In particular, if a batch had 10 commits in it, you can either run 10 additional compiles and tests (didn't have the resources), or you can binary-search/bisect to find the failure (takes too long). It is often faster to use a heuristic to say "Test_Foo failed, and Foo_Src was modified in change X, so revert change X". We don't like relying on heuristics, but until recently it was our only choice.

We actually don't run all the tests all the time. We have a good dependency system (called Gyp/GN, similar to Bazel) which knows which tests rely on which source files. When the infrastructure compiles a new change, it only runs the tests which depend on source files touched by that change.

Do you think a version of this system could work in a team that has mostly "b-players?" I just wonder how much of this stays afloat because the caliber of the engineers is higher than that of your typical company.

I don't think any of our decisions are driven by or dependent upon the caliber of our engineers.

For example:

a) No merges; and the only branches are release branches. This makes sense in almost any context. Whether your team is big or small, high-caliber or college sophomores, this works. The advantages are numerous. No one works on a feature branch in isolation, only to have tons of merge conflicts and surprises when they try to bring it back in. Bug fixes and features are never landed on a release branch and then not merged into master for the next release. Development doesn't slow to a crawl while you try to stabilize master for a release.

b) Keeping dependencies near tip-of-tree. Chromium has two kinds of dependencies: third-party libraries (e.g. yasm, lighttpd) and things we develop on our own but in separate repositories (e.g. pdfium, v8, boringssl). The former tend to have stable releases produced by their own maintainers; the latter is being developed in parallel and you want to be as up-to-date as possible. Most projects are similar: you have third-party dependencies (e.g. python packages) and your own libraries (like a common set of api interfaces). You want to keep the ones you control as close to tip-of-tree as possible so that you can move fast and integrate your own changes quickly.

The caveat to all of this "move quickly" stuff is "test Test TEST". If you are constantly rolling dependencies forward, stuff will break. Run all the tests before landing a dependency change. Revert dependency changes with impunity: since they are effectively merges of many commits made to the other repo, they are more likely to have breaking changes than any single commit to your main repo. So pretest, test, and revert.

None of this requires amazing engineers. It just requires a dedication to quality and good testing hygiene.

I wonder though, there /have/ to be merge conflicts? I suppose you continually rebase on the HEAD of master to keep up to date? So instead of taking care of conflicts at the end of a 'dev cycle' you take care of it in the middle.

Did I understand that correctly?

The difference is that, in our flow, the merge conflicts are on the scale of a a single developer making a single self-contained commit. In a flow with feature branches, the merge conflicts are on a scale of the entire branch conflicting with any other changes made on master.

But yes we also provide tools (https://commondatastorage.googleapis.com/chrome-infra-docs/f...) to make things like rebasing (see git rebase-update) your local branches on top of master easier.

I'm sorry if this is obvious, but I would love to understand exactly what you mean by "the only branches are release branches". What I think you mean is that the only branches are release branches and the development branches on each dev's machine.

At my startup, we use feature branches, and we expect devs to continually merge master into their branch while it's under development (ideally daily), and to fix any resulting conflicts or test failures. We have them push these branches to the repo (which triggers our continuous integration server), but they're treated as private works in progress. Then, when they finish and pass code review, we're confident that they will merge seamlessly, since they've been merging master all along. (We also use git merge --squash for the final merge to avoid seeing all the intermediate commits on the master commit log.)

I'm trying to understand if your practices are meaningfully different. Do developers have private branches on their machines for weeks on end? Are they encouraged to regularly merge master into those private branches?

Could I ask you a dumb question? What does CL stand for?

changelist

Thought so. I've seen CL used in odd ways on the v8-users list so I got confused. Thanks!

Now that Chromium has moved over to Git, any plans or timeline to migrate Chromium over to Gerrit, rather than Reitveld?

Heh, that's a contentious topic. Personally, I would like to do so. I think I speak correctly when I say that most of the Infrastructure team would like to do so, to increase security/auditability and ease the maintenance burden on ourselves.

However, large portions of the team are opposed to Gerrit because a) it doesn't integrate as well with our other systems (such as trybots and cq) and b) the UI has some issues. We are working with people to try to fix those, but there's no particular timeline for switching, or even a guarantee that we will.

> a) it doesn't integrate as well with our other systems (such as trybots and cq) and b) the UI has some issues.

Is there a tracking bug for all the integration and UI issues? How can folks working on Chromium but without an @google.com help?

I think this is the best tracking bug: https://code.google.com/p/chromium/issues/detail?id=465929

There are three things going on:

1) Rietveld exists

2) A new Polymer-based UI for Rietveld exists

3) Gerrit exists

The bug linked above is about evaluating the ability to make the Polymer-based UI (which is pretty and functional) work on top of Gerrit instead of on top of Rietveld. This is one of the possible paths forward for getting people to use Gerrit, and the one for which external contributors could be the most help.

Other paths forward exist: making Rietveld and the polymer Rietveld UI so good that no one wants to switch to Gerrit anymore; working directly on the Gerrit UI; or convincing devs that the Gerrit UI is good enough. But I think the bug and description above is the path on which external contributors could do the most good.

You should ask your friends at Opera about code review systems ;)

Does Google currently use or plan to use static analysis tools that prove absence of key bug classes?

https://en.wikipedia.org/wiki/List_of_tools_for_static_code_...

I know in safety-critical the Astree and SPARK tools have been particularly good at verifying code. Something like Polyspace might be more appropriate for Chrome, though.

SPARK is based on Ada. Astree also seems to be limited only to C. Chrome, to the opposite, is written in millions of lines of C++.

The reason I mentioned SPARK and Astree is because they're excellent at what they do. People Googling them will understand how these approaches work to prove absence of bugs rather than test for their presence. Then, they might experiment with some of the others on the list.

I mentioned PolySpace as an example of something more appropriate for Chrome because it supports C++ and is used in high integrity applications.

We've invested quite a bit of energy in static analysis tools over the years, but so far the return hasn't been there. There are still ongoing efforts, and we'll see if those pan out, but up to this point we've just run into a very low hit rate and a false positive rate that drowns everything else out from a security perspective. Formal methods are a different question, but I can't imagine the size of the undertaking needed to convert even a subset of Chromium over to formal methods.

Instead, we've focused pretty heavily on more conventional anti-exploit mitigations applied to existing C++ code. Our sandboxing is the most obvious one, but we have a variety of other protections on top of typical stuff like ASLR and NX (e.g. heap partitioning, a safe integer handling toolkit, etc). We're also leading efficient control-flow integrity work in clang, and busily working it up to a stable release. And then we have fuzzing as part of our continuous integration, running across thousands of cores of instrumented builds.

Sorry to hear that. Yeah, they're not as strong for C++ yet as it's more complex and has less R&D in that sector. My idea for C++ would be to try to find or even encourage development of those that catch the interface and concurrency errors at the least. So, that's Design-by-Contract [1] [2] along with something like ConSeq [3] if it's available or usable for your build. D-by-C, done right, gets a lot of mileage w/ ConSeq showing promise.

Do keep the exploit mitigation tech, though. The NaCl paper was one of best reads I had that year. Most interesting one right now is Code Pointer Integrity [4]. Curious, is there a configuration that swaps out the sandbox for something with the same interface but no real sandbox? Point is if we can swap out the existing sandbox (esp CFI) for a different technology (eg CPI, Softbound + CETS).

[1] http://www.drdobbs.com/an-exception-or-a-bug/184401686

[2] http://www.codeproject.com/Articles/8293/Design-by-Contract-...

[3] https://webcache.googleusercontent.com/search?q=cache:WZAQPu...

[4] https://www.usenix.org/system/files/conference/osdi14/osdi14...

Do Chrome for PC and Chrome for iOS come from the same code base, and if so, how do you automate testing for closed systems like iOS? (I ask because from my totally anecdotal perspective, Chrome for PC is a rock-solid dream of an application, but Chrome on my iPad crashes if it sees an animated GIF from across the room.)

It is unfortunate that Apple does not allow us to bring all of Chromium--our full rendering stack (Blink, etc.)--to iOS. If they did, we could build a much better browser for iOS. Instead, we are forced to use their provided WebView, and that means living with its limitations and bugs.

What are the policies behind the commit queues? Is it mandatory for a change to be queued?

The commit queue can only be bypassed by people who are 'committers' and therefore have direct push access to the repo. Becoming a committer requires landing multiple changes, demonstrating competence and desire to continue to contribute, and being nominated and seconded by your peers. This is true for all contributors, even Google employees.

Even committers are heavily encouraged to use the commit queue, and everyone does. Something like 96% of commits land via the CQ. It's simply a bad idea not to. The main use-case for bypassing the queue is quickly reverting a change which somehow slipped through, and even those usually go through the queue anyway with a special flag that bypasses all of the tests.

So, engineers work on the master branch, test locally and push to master? And the buildbot detects commits that break the build after the fact?

Engineers base their local development branches on top of origin/master. They can make as many local commits on that branch as they like. Then they upload the diff of that entire branch vs. its upstream (origin/master, unless they're doing chained changes) to the code review system. As it goes through code review, it is tested on the trybots. Once all the trybots pass and it has received approval from the reviewers, it is landed on master and fully tested again.

Maybe I'm still not understanding fully, but that really sounds just like having branches, except they aren't treated by the system as such. There's still the possibility that while developer A's change is going through code review and testing, developer B's changes (which conflict with developer A's) make it through and get landed.

I'd agree that having long-lived branches is often bad, but that's more a matter of discipline than anything else. I could certainly start working on a feature locally using Chromium's model, take my time, and get far enough behind the tip of the tree that by the time I want to submit for code review and testing, my stuff doesn't apply to the new tip.

Branches vs. no branches doesn't seem to be a factor here.

There are no branches in the golden-source-of-truth repository (except for release branches, but those are never merged back into master and no development is done directly on them). The only branches that exist are on the developer's local machine, and only exist for that developer's personal preference and bookkeeping.

Yes, it is totally possible for two developers to be working in the same place and try to land their changes at the same time, resulting in a conflict. This happens in one of three ways: a) dev B's change fails tests, because the try bots which try to test dev B's change have already sync'd dev As change (this is the most common) b) dev B's change passes all the tests, but fails to commit because dev A's change landed seconds earlier. In this case the system sends dev B an email asking them to rebase their change and try again. Simple. c) dev B's change passes all the tests, then lands and causes tip-of-tree tests to fail because dev A's change landed seconds earlier and they disagree on the name of an API method or something. This happens rarely, and is easily resolved by reverting one of the two and asking the author to try again.

Anyway, the point is that the dev branches are incredibly short lived: they exist only on the developer's machine and end up as a single, coherent, self-contained commit on master. If you get far enough behind that your stuff doesn't apply on master, it is your responsibility to clean it up, and no one else even knows that you had a conflict.

How long does a single dev typically go between applying their changes to master? Minutes, hours, days, weeks? If you have a big change to work on do you put it in master behind a feature flag, do the whole thing and then rebase, or what?

Do developers ever want to collaborate on an in-progress feature before it's ready for master? As far as I can see the only real difference between your workflow and a github-flow-like workflow is that they can't do that because their feature branches are local.

Features can take months and involve many people. In that case it will be worked on behind a feature flag on master.

There is no such thing as "not ready for master" in the chromium workflow, only "not ready to be enabled by default".

> We don't have enough resources to run every test for every commit, so they end up batched and we have to rely on humans and heuristics to determine which CL in the batch to revert.

You could automatically bisect with just the failing test to find which commit to revert. At least if it's not a flapping test.

Reply on news.ycombinator.com