phasmantistes

The commit queue can only be bypassed by people who are 'committers' and therefore have direct push access to the repo. Becoming a committer requires landing multiple changes, demonstrating competence and desire to continue to contribute, and being nominated and seconded by your peers. This is true for all contributors, even Google employees.

Even committers are heavily encouraged to use the commit queue, and everyone does. Something like 96% of commits land via the CQ. It's simply a bad idea not to. The main use-case for bypassing the queue is quickly reverting a change which somehow slipped through, and even those usually go through the queue anyway with a special flag that bypasses all of the tests.

As the article states, Chromium uses run-time checks (feature toggles) extensively. This lets half-baked features land and start undergoing testing before the feature is ready to be shown to the world.

But we don't say "oh no, this feature isn't ready, let's wrap it in checks". The feature is wrapped in checks from the very first commit, so that everything stays stable on master and on any release branches which are cut from master.

Then, when the feature is ready for prime-time, the developers working on it go through and remove the checks. So they don't accumulate in the codebase.

Heh, that's a contentious topic. Personally, I would like to do so. I think I speak correctly when I say that most of the Infrastructure team would like to do so, to increase security/auditability and ease the maintenance burden on ourselves.

However, large portions of the team are opposed to Gerrit because a) it doesn't integrate as well with our other systems (such as trybots and cq) and b) the UI has some issues. We are working with people to try to fix those, but there's no particular timeline for switching, or even a guarantee that we will.

A nit to pick here: Google releases Chrome, not Chromium. We only provide Chromium as a source repository, not binaries. Other independent entities (such as Linux distros) can provide releases of Chromium, but Google does not.

That said, Google doesn't support previous releases for security reasons. We are constantly improving the security profile of Chrome, and hackers (both black hat and white hat) are constantly finding exploits in older versions. (The same is true for all the other major browsers on the market.)

If you want to use Chrome on a previous/older release, please don't. If you want to use Chromium on a previous/older release, also please don't, but you can compile it yourself if you're so inclined.

And for the most part, no, customers don't demand stable old releases of Chrome. They've learned how bad of a security and usability mistake that is from IE6, and they don't want to repeat past mistakes any more than we do.

Well, we don't use perforce (we use Git), so I assume you're asking for your own benefit.

The buildbot masters watch the source repository for new commits. When they see a new commit appear, they tell all of their builders (each one representing a different build configuration) to do a build+test sync'ed to the new commit hash. The builders check out the repository at that revision, compile, and upload the binaries for the testers. The testers then download those binaries, test, and report results. Then we have a display (build.chromium.org) which shows the results of the compiles and tests on each configuration for all of the recent commits.

It's actually pretty rare for a patch to become unlandable while the tests are running. But if it does, it just sends a message to the owner of the change that says "Hey, your patch failed to land due to merge conflicts. Please rebase and fix them, then try again."

The difference is that, in our flow, the merge conflicts are on the scale of a a single developer making a single self-contained commit. In a flow with feature branches, the merge conflicts are on a scale of the entire branch conflicting with any other changes made on master.

But yes we also provide tools (https://commondatastorage.googleapis.com/chrome-infra-docs/f...) to make things like rebasing (see git rebase-update) your local branches on top of master easier.

We actually are working on systems which re-run the failing tests on all CLs within a failing batch. Unfortunately, we haven't been able to do this previously due to resource constraints. In particular, if a batch had 10 commits in it, you can either run 10 additional compiles and tests (didn't have the resources), or you can binary-search/bisect to find the failure (takes too long). It is often faster to use a heuristic to say "Test_Foo failed, and Foo_Src was modified in change X, so revert change X". We don't like relying on heuristics, but until recently it was our only choice.

We actually don't run all the tests all the time. We have a good dependency system (called Gyp/GN, similar to Bazel) which knows which tests rely on which source files. When the infrastructure compiles a new change, it only runs the tests which depend on source files touched by that change.

Engineers base their local development branches on top of origin/master. They can make as many local commits on that branch as they like. Then they upload the diff of that entire branch vs. its upstream (origin/master, unless they're doing chained changes) to the code review system. As it goes through code review, it is tested on the trybots. Once all the trybots pass and it has received approval from the reviewers, it is landed on master and fully tested again.

I don't think any of our decisions are driven by or dependent upon the caliber of our engineers.

For example:

a) No merges; and the only branches are release branches. This makes sense in almost any context. Whether your team is big or small, high-caliber or college sophomores, this works. The advantages are numerous. No one works on a feature branch in isolation, only to have tons of merge conflicts and surprises when they try to bring it back in. Bug fixes and features are never landed on a release branch and then not merged into master for the next release. Development doesn't slow to a crawl while you try to stabilize master for a release.

b) Keeping dependencies near tip-of-tree. Chromium has two kinds of dependencies: third-party libraries (e.g. yasm, lighttpd) and things we develop on our own but in separate repositories (e.g. pdfium, v8, boringssl). The former tend to have stable releases produced by their own maintainers; the latter is being developed in parallel and you want to be as up-to-date as possible. Most projects are similar: you have third-party dependencies (e.g. python packages) and your own libraries (like a common set of api interfaces). You want to keep the ones you control as close to tip-of-tree as possible so that you can move fast and integrate your own changes quickly.

The caveat to all of this "move quickly" stuff is "test Test TEST". If you are constantly rolling dependencies forward, stuff will break. Run all the tests before landing a dependency change. Revert dependency changes with impunity: since they are effectively merges of many commits made to the other repo, they are more likely to have breaking changes than any single commit to your main repo. So pretest, test, and revert.

None of this requires amazing engineers. It just requires a dedication to quality and good testing hygiene.

Since all I've worked on for the last three years is the infrastructure, and not the binary that lands on your desktop/phone, I'm not really qualified to answer that. (Which is just a fancy way of saying I've got no earthly idea :))

But since I've seen @bengoodger in this thread, maybe he can answer your questions.

We are working on a few things.

The first is automatic retries of fast tests. If a test runs quickly and fails, it costs us little to try again just to make sure. Most of our unit tests are configured to run up to 3 times.

Another thing is keeping track of a database of individual test case passes/failures across all time. This will let us automatically mark tests as flaky if they fail often, and ignore their results programatically rather than requiring a human to manually mark the test as ignorable.

A third thing is, obviously, automatically filing bugs against the owners/authors of tests which have been marked flaky. This is controversial -- often a test is just fine until one of its underlying libraries has a race condition introduced, and the real person to fix it should be the author of that change, not the author of the test. But it is still a step in the right direction much of the time.

Many people subscribe to the philosophy that "a flaky test is worse than no test", because you think it is giving you information when in fact it is giving you none. I subscribe to a slightly different philosophy: "A test with a known flaky rate is hugely valuable". If you know how often a test flakes (statistically), then you can measure variances from that rate to detect changes. Of course, a flaky test with an unknown rate of flaky is still useless. Hence the second initiative above: measuring the rate of flake of everything.

Chromium is open source, and will run try jobs for anyone from anywhere around the world (we won't accept your commit without lots of checks, but we'll run your tests). That means that we don't get to rely on a lot of Google-internal technology, including our production data centers.

Also, we are compiling and testing on Mac, Windows, Linux, Android, iOS, ChromeOS, and historical versions of all of the above. That's something that a mostly linux-server-based company doesn't do a lot of. So we maintain our own datacenters with our own hardware, and don't get to take advantage of all of Google's scale.

Chromium continuous build infrastructure engineer here.

As he says, this is slightly out of date: we forked WebKit to make Blink in 2013, and actually merged Blink into Chromium proper just this week! It is no longer managed by gclient/DEPS, but instead developed alongside Chromium in the same repository. But many other things (pdfium, v8, webrtc, and tons of smaller libraries) are still pulled in via DEPS in exactly the way described.

I'd just like to say this about our development, testing, and release process: it really really works. We keep moving fast, everyone knows what's going on throughout the codebase because everyone is working on master, and there are never any big ugly merge conflicts to resolve when new features land.

There are, of course, problems with the system. We don't have enough resources to run every test for every commit, so they end up batched and we have to rely on humans and heuristics to determine which CL in the batch to revert. The set of configurations on the try-bots doesn't exactly match up with the set of configurations that are run against tip-of-tree. And flaky tests are, of course, a perennial problem for everyone. But we're working on making all of these better, and huge progress has been made in the last couple years.

Ask me anything if you have any questions about our build and test infrastructure!

Wow, that's an uncharitable opinion of design and designers. They're highly-trained professionals too, and redesigns like this aren't done just for the heck of it -- according to the blog post, this measurably increases performance on small screens and in low-bandwidth contexts.