How Chromium Works
medium.com
medium.com
As he says, this is slightly out of date: we forked WebKit to make Blink in 2013, and actually merged Blink into Chromium proper just this week! It is no longer managed by gclient/DEPS, but instead developed alongside Chromium in the same repository. But many other things (pdfium, v8, webrtc, and tons of smaller libraries) are still pulled in via DEPS in exactly the way described.
I'd just like to say this about our development, testing, and release process: it really really works. We keep moving fast, everyone knows what's going on throughout the codebase because everyone is working on master, and there are never any big ugly merge conflicts to resolve when new features land.
There are, of course, problems with the system. We don't have enough resources to run every test for every commit, so they end up batched and we have to rely on humans and heuristics to determine which CL in the batch to revert. The set of configurations on the try-bots doesn't exactly match up with the set of configurations that are run against tip-of-tree. And flaky tests are, of course, a perennial problem for everyone. But we're working on making all of these better, and huge progress has been made in the last couple years.
Ask me anything if you have any questions about our build and test infrastructure!
For example:
a) No merges; and the only branches are release branches. This makes sense in almost any context. Whether your team is big or small, high-caliber or college sophomores, this works. The advantages are numerous. No one works on a feature branch in isolation, only to have tons of merge conflicts and surprises when they try to bring it back in. Bug fixes and features are never landed on a release branch and then not merged into master for the next release. Development doesn't slow to a crawl while you try to stabilize master for a release.
b) Keeping dependencies near tip-of-tree. Chromium has two kinds of dependencies: third-party libraries (e.g. yasm, lighttpd) and things we develop on our own but in separate repositories (e.g. pdfium, v8, boringssl). The former tend to have stable releases produced by their own maintainers; the latter is being developed in parallel and you want to be as up-to-date as possible. Most projects are similar: you have third-party dependencies (e.g. python packages) and your own libraries (like a common set of api interfaces). You want to keep the ones you control as close to tip-of-tree as possible so that you can move fast and integrate your own changes quickly.
The caveat to all of this "move quickly" stuff is "test Test TEST". If you are constantly rolling dependencies forward, stuff will break. Run all the tests before landing a dependency change. Revert dependency changes with impunity: since they are effectively merges of many commits made to the other repo, they are more likely to have breaking changes than any single commit to your main repo. So pretest, test, and revert.
None of this requires amazing engineers. It just requires a dedication to quality and good testing hygiene.
Did I understand that correctly?
But yes we also provide tools (https://commondatastorage.googleapis.com/chrome-infra-docs/f...) to make things like rebasing (see git rebase-update) your local branches on top of master easier.
At my startup, we use feature branches, and we expect devs to continually merge master into their branch while it's under development (ideally daily), and to fix any resulting conflicts or test failures. We have them push these branches to the repo (which triggers our continuous integration server), but they're treated as private works in progress. Then, when they finish and pass code review, we're confident that they will merge seamlessly, since they've been merging master all along. (We also use git merge --squash for the final merge to avoid seeing all the intermediate commits on the master commit log.)
I'm trying to understand if your practices are meaningfully different. Do developers have private branches on their machines for weeks on end? Are they encouraged to regularly merge master into those private branches?
Also, we are compiling and testing on Mac, Windows, Linux, Android, iOS, ChromeOS, and historical versions of all of the above. That's something that a mostly linux-server-based company doesn't do a lot of. So we maintain our own datacenters with our own hardware, and don't get to take advantage of all of Google's scale.
And FWIW on building MongoDB with Evergreen (https://evergreen.mongodb.com) we've found that a stepback approach seems to give us the finest granularity with minimal cost in identifying the introduction of an error - we batch commits for execution, but when a specific test fails we start running just that test on each previous commit until we hit a passing execution. It obviously doesn't work perfectly in the face of flaky tests (see above question) but it seems to do pretty well.
The first is automatic retries of fast tests. If a test runs quickly and fails, it costs us little to try again just to make sure. Most of our unit tests are configured to run up to 3 times.
Another thing is keeping track of a database of individual test case passes/failures across all time. This will let us automatically mark tests as flaky if they fail often, and ignore their results programatically rather than requiring a human to manually mark the test as ignorable.
A third thing is, obviously, automatically filing bugs against the owners/authors of tests which have been marked flaky. This is controversial -- often a test is just fine until one of its underlying libraries has a race condition introduced, and the real person to fix it should be the author of that change, not the author of the test. But it is still a step in the right direction much of the time.
Many people subscribe to the philosophy that "a flaky test is worse than no test", because you think it is giving you information when in fact it is giving you none. I subscribe to a slightly different philosophy: "A test with a known flaky rate is hugely valuable". If you know how often a test flakes (statistically), then you can measure variances from that rate to detect changes. Of course, a flaky test with an unknown rate of flaky is still useless. Hence the second initiative above: measuring the rate of flake of everything.
Firefox has an experimental "chaos mode" that takes the opposite approach. It purposely randomizes behavior by adjusting thread priorities, changing hash table iteration order, and randomize timer durations. Unfortunately, many flaky tests fail in chaos mode, so it is not enabled by default.
http://robert.ocallahan.org/2014/03/introducing-chaos-mode.h...
I think one of the lessons we've learnt is that with a browser-type project it's very hard to make test runs fully deterministic, for both technical and human reasons.
The technical reasons are touched on in the original article: these are complex codebases with lots of moving parts and lots of environmental dependencies. Of course there are various tactics to try and combat this; for example there is a wiki page dedicated to innocuous-looking code that leads to intermittent tests [1].
The human reasons centre around the difficulty of getting people to care about spending time fixing a test that fails one time in 1,000 (which is still very noticeable when you are running it hundreds of times a day). Unless the issue is something that fits a known pattern it's hard work, difficult to tell if your fix even worked, and not likely to be considered a top priority due to the diffuse, hard to quantify, nature of the benefits.
I think the fact that both Google and Mozilla still have significant problems with intermittents despite talented engineering staff and it having been a known problem for years implies that some of the standard thinking about making tests fully deterministic simply doesn't apply; for this kind of work you have to embrace — or at least accept — the randomness, and look for ways to get the data you need despite the noise.
[1] https://developer.mozilla.org/en-US/docs/Mozilla/QA/Avoiding...
Could you run "all" the identify flaky tests by running all the tests 100 times on the same stable build (like the latest ESR)? Is it even possible to write a test that could pass 100 times in a row? :)
Unfortunately there are various confounding factors that mean many intermittent tests would look clean in such a run might nevertheless be problematic. For example if you only run tests that you think are intermittent problems that are triggered by state left from a previous test won't be found. This is one reason that we've been trying to run particularly problematic test types (e.g. firefox browser-chrome tests) in smaller groups restarting the browser with a clean profile between groups to clear the state. A group size of 1 would obviously be ideal here, but when you have thousands of tests and limited resources it's not practical.
The other problem is tests that have unexpected sensitivity to the environment. For example the other day DNS was being slow on the test infrastructure. This isn't a problem for most tests since they use something like /etc/hosts. But some tests were intentionally trying to use a non-resolving domain and those tests sudden started to randomly time out.
I debugged those bugs with custom gdb scripts. The test system will run the gdb scripts to start the app. The gdb script setup breakpoints, log stack trace and local variables automatically for a particular bugs.
The debugging environment is scripted and automated. Test failure mark/track the log file with all the gdb info.
I schedule the tests to run continuously overnight and in the morning I analyze the log file with gdb info.
After that I can add more debug command in gdb script or try fixed the bug if I have enough info. The cycle continue until the bugs were all fixed.
I personally think it was very good for try to analyze / fixing the "flaky" tests issue.
Even for the the tests that fails 1 in 1000 times, the bugs can still be debug and fixable.
It was actually more relaxing for me to debug this kind of problem. 99% of time, the computer do all the works.
You guys might want to try it if not done in Google already.
I've always found it interesting that Chrome is shipped mostly as one big binary (or on Windows, two: chrome.dll and chrome_child.dll), whereas IE and Safari use several (one for the rendering engine, one for the JS engine, one or two for networking, etc.). My guess is that Chrome uses one big binary to minimize startup time, since the OS can just map that binary into memory and go. Am I correct about the rationale? Another way to ask the same thing is this: what would be the downsides of using a component build for releases and not just for development? If you used a component build for releases, you'd have greater parity between dev and production, and that's generally a good thing.
I also notice that there are a few modules that get their own DLLs, at least on Windows. For example, in Chrome 45, we have chrome_elf.dll, libegl.dll, libexif.dll, and libglesv2.dll; I'd think these libraries could be statically linked just like the few hundred others. Are there good reasons for these exceptions? Or are these just little details that tend to get overlooked given the rapid pace of development?
But since I've seen @bengoodger in this thread, maybe he can answer your questions.
I'm not sure why there are now a handful of separate dlls.
As for the remaining DLLs, it mostly comes down to licensing concerns. We don't include proprietary code in our main binaries, and in order to simplify IP concerns it's often best to split such code into other DLLs. For example, you asked about libegl.dll and libglesv2.dll, which I believe include a licensed copy of the SwiftShader library.
It's also worth pointing out that the performance impact of having a single, static binary in Windows is much less significant than it used to be. So, were we starting Chrome today I'm not so sure we'd make the same decision.
I'd agree that having long-lived branches is often bad, but that's more a matter of discipline than anything else. I could certainly start working on a feature locally using Chromium's model, take my time, and get far enough behind the tip of the tree that by the time I want to submit for code review and testing, my stuff doesn't apply to the new tip.
Branches vs. no branches doesn't seem to be a factor here.
Yes, it is totally possible for two developers to be working in the same place and try to land their changes at the same time, resulting in a conflict. This happens in one of three ways: a) dev B's change fails tests, because the try bots which try to test dev B's change have already sync'd dev As change (this is the most common) b) dev B's change passes all the tests, but fails to commit because dev A's change landed seconds earlier. In this case the system sends dev B an email asking them to rebase their change and try again. Simple. c) dev B's change passes all the tests, then lands and causes tip-of-tree tests to fail because dev A's change landed seconds earlier and they disagree on the name of an API method or something. This happens rarely, and is easily resolved by reverting one of the two and asking the author to try again.
Anyway, the point is that the dev branches are incredibly short lived: they exist only on the developer's machine and end up as a single, coherent, self-contained commit on master. If you get far enough behind that your stuff doesn't apply on master, it is your responsibility to clean it up, and no one else even knows that you had a conflict.
Do developers ever want to collaborate on an in-progress feature before it's ready for master? As far as I can see the only real difference between your workflow and a github-flow-like workflow is that they can't do that because their feature branches are local.
There is no such thing as "not ready for master" in the chromium workflow, only "not ready to be enabled by default".
More importantly, you seem to be running all the tests, as often as you can. A commit changing a text label shouldn't require running all the tests, that just doesn't scale. Have you consider splitting Chromium into different projects, where commits for one project would result in the tests of that project, and only that project?
We actually don't run all the tests all the time. We have a good dependency system (called Gyp/GN, similar to Bazel) which knows which tests rely on which source files. When the infrastructure compiles a new change, it only runs the tests which depend on source files touched by that change.
However, large portions of the team are opposed to Gerrit because a) it doesn't integrate as well with our other systems (such as trybots and cq) and b) the UI has some issues. We are working with people to try to fix those, but there's no particular timeline for switching, or even a guarantee that we will.
Is there a tracking bug for all the integration and UI issues? How can folks working on Chromium but without an @google.com help?
There are three things going on:
1) Rietveld exists
2) A new Polymer-based UI for Rietveld exists
3) Gerrit exists
The bug linked above is about evaluating the ability to make the Polymer-based UI (which is pretty and functional) work on top of Gerrit instead of on top of Rietveld. This is one of the possible paths forward for getting people to use Gerrit, and the one for which external contributors could be the most help.
Other paths forward exist: making Rietveld and the polymer Rietveld UI so good that no one wants to switch to Gerrit anymore; working directly on the Gerrit UI; or convincing devs that the Gerrit UI is good enough. But I think the bug and description above is the path on which external contributors could do the most good.
Even committers are heavily encouraged to use the commit queue, and everyone does. Something like 96% of commits land via the CQ. It's simply a bad idea not to. The main use-case for bypassing the queue is quickly reverting a change which somehow slipped through, and even those usually go through the queue anyway with a special flag that bypasses all of the tests.
https://en.wikipedia.org/wiki/List_of_tools_for_static_code_...
I know in safety-critical the Astree and SPARK tools have been particularly good at verifying code. Something like Polyspace might be more appropriate for Chrome, though.
I mentioned PolySpace as an example of something more appropriate for Chrome because it supports C++ and is used in high integrity applications.
Instead, we've focused pretty heavily on more conventional anti-exploit mitigations applied to existing C++ code. Our sandboxing is the most obvious one, but we have a variety of other protections on top of typical stuff like ASLR and NX (e.g. heap partitioning, a safe integer handling toolkit, etc). We're also leading efficient control-flow integrity work in clang, and busily working it up to a stable release. And then we have fuzzing as part of our continuous integration, running across thousands of cores of instrumented builds.
Do keep the exploit mitigation tech, though. The NaCl paper was one of best reads I had that year. Most interesting one right now is Code Pointer Integrity [4]. Curious, is there a configuration that swaps out the sandbox for something with the same interface but no real sandbox? Point is if we can swap out the existing sandbox (esp CFI) for a different technology (eg CPI, Softbound + CETS).
[1] http://www.drdobbs.com/an-exception-or-a-bug/184401686
[2] http://www.codeproject.com/Articles/8293/Design-by-Contract-...
[3] https://webcache.googleusercontent.com/search?q=cache:WZAQPu...
[4] https://www.usenix.org/system/files/conference/osdi14/osdi14...
You could automatically bisect with just the failing test to find which commit to revert. At least if it's not a flapping test.
I expected to run into a lot of problems with building, missing dependencies etc. but it just worked. This despite the fact that the fully checked out source is 37Gb!
I can highly encourage beginners to take a stab at contributing to Chromium, you can learn a lot.
Sometimes we forget that we are making software for humans, by humans, and not for compilers and computers.
That said, Google doesn't support previous releases for security reasons. We are constantly improving the security profile of Chrome, and hackers (both black hat and white hat) are constantly finding exploits in older versions. (The same is true for all the other major browsers on the market.)
If you want to use Chrome on a previous/older release, please don't. If you want to use Chromium on a previous/older release, also please don't, but you can compile it yourself if you're so inclined.
And for the most part, no, customers don't demand stable old releases of Chrome. They've learned how bad of a security and usability mistake that is from IE6, and they don't want to repeat past mistakes any more than we do.
I suspect that there are lots of enterprise customers that haven't learned that (encountered it, but not learned it), but they probably aren't Chrome customers, since they are still using IE.
If you instead view IT departments as the customer, then yes, they care and want to have control over the deployed version of the software on their networks. Chrome does have some controls for this. I'm not that familiar with them, though.
Kongregate currently tells you chrome broke Unity with its current version and you should use Firefox instead. If previous versions are not available, switching browsers is an alternative workaround.
Engineering is about tradeoffs. You can't implement everything someone might want. The desire to have the product continuously improve is in direct conflict with the desire to have the product never suddenly break.
In this case the team believed the positives dramatically outweigh the negatives and I agree. Continuous silent updates are one of the best features of Chrome. All consumer software should have it.
As a developer, I wouldn't recommend an old browser solely for security reasons, but also so I can stop support ancient browsers sooner…
my experience with projects that do trunk based development, i.e. a single branch and time-based releases, is that not all features can be ready when the release bell "rings", and something is necessarily half-baked. So you end up enclosing some code into "if (FEATURE_READY) {...}" so that it's there, but it isn't executed until a few releases later.
This carries some problems, like the explosion of the quantity of toggles over time, dead code that nobody cleans up, etc.
But we don't say "oh no, this feature isn't ready, let's wrap it in checks". The feature is wrapped in checks from the very first commit, so that everything stays stable on master and on any release branches which are cut from master.
Then, when the feature is ready for prime-time, the developers working on it go through and remove the checks. So they don't accumulate in the codebase.
With good software design practices we don't usually find bugs that only occur with some specific non-total combination of features. But of course it does happen.
The article isn't a way to make provably correct software; it's a way to keep the wheels on a massive, fast-moving project -- barely :).
(Of course, the Knight incident being fresh in our minds, our assessment of those risks might not be entirely objective)
In perforce terminology how is this done? You could use branches, but you could also shelve the change and have the build-bot unshelve it and test.
>We also have a commit queue, which is a way to try a change and have it landed automatically if the try succeeds
What happens if the change becomes incompatible due to other changes over the course of running the test? I assume the commit just fails in this case..
The buildbot masters watch the source repository for new commits. When they see a new commit appear, they tell all of their builders (each one representing a different build configuration) to do a build+test sync'ed to the new commit hash. The builders check out the repository at that revision, compile, and upload the binaries for the testers. The testers then download those binaries, test, and report results. Then we have a display (build.chromium.org) which shows the results of the compiles and tests on each configuration for all of the recent commits.
It's actually pretty rare for a patch to become unlandable while the tests are running. But if it does, it just sends a message to the owner of the change that says "Hey, your patch failed to land due to merge conflicts. Please rebase and fix them, then try again."
e.g. https://code.google.com/p/chromium/issues/detail?id=84177
(i also tried to find all the commits or issues where all the 3 ways to disable sending referrer headers were removed over time, but obviously, there are no issues and the commits are probably mixed up with other things)
(And as to your bug, I tried --no-referrers in Chrome 45 on Windows 8 and it worked fine.)