Clang Is Now Used to Build Chrome for Windows | Hacker News Reader

Clang Is Now Used to Build Chrome for Windows

blog.llvm.org

398 pointsjanvdberg100 comments

Have seen a couple interesting articles[0][1] highlighting the weirdness of building Chromium on Windows, so hopefully this LLVM improvement will be a step towards a smoother process.

[0] https://randomascii.wordpress.com/2018/02/25/compiler-bug-li...

[1] https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and...

On HN:

[0] https://news.ycombinator.com/item?id=16463572

[1] https://news.ycombinator.com/item?id=14733829

Hey, those are mine! Those two issues (and this one: https://randomascii.wordpress.com/2018/02/11/zombie-processe...) are equally problematic with clang. Two of them were problems with Windows, and one was a flaw in a Microsoft system management component.

Luckily they are all mitigated, fixed, or being fixed, so my Chrome builds run much more smoothly now.

Thank you for chiming in! Your articles are wonderful!

Did you guys ever reverse-spec the name mangling?

We first started fiddling with Clang for Chrome in late 2010: http://neugierig.org/software/chromium/notes/2011/01/clang.h...

The original work was just finding bugs in the codebase and compiler. It's gone so far!

> if your project is Windows-only, you can get a second compiler’s opinion on your code, and Clang’s warnings might find bugs

GCC (mingw.org and mingw-w64 triples) has existed and been able to do this, including and especially cross-compiling, for a long time. No need to manually download a Windows SDK either.

edited to add: to be fair, it's not ABI compatible with MSVC. But it deserves a mention and it's an easier option than MSVC-style build systems for many open source projects.

Apple hired developers of Clang and started supporting Clang around the time GPLv3 was being pushed by FSF, Apple at the time was using GCC in their development and feared that GPLv3 would taint their code.

other than technical merits, permissible license of Clang is attractive for some developers/businesses

I thought Apple didn’t just “hire” the developers of clang. They started the clang project in-house before eventually open sourcing it.

Yes; Clang was started internally to Apple. They did "just hire" the developer of LLVM.

> edited to add: to be fair, it's not ABI compatible with MSVC. But it deserves a mention and it's an easier option than MSVC-style build systems for many open source projects.

Pretty sure this is significant in Chrome's case.

Why, though? What does Chrome do that calls into C++ code compiled by anyone but Google?

on Windows the native ABI (or the default ABI I should say) explicitly reserves space for C++ stack unwinding on a per call basis.

A lot of the windows API can (ab)use this so it’s best to follow it, despite it being C (ish).

File dialogs (to save/load web pages, to upload files), print support, TCP sockets… all of these involve calling into system code.

do any of those require ABI compatibility at the C++ level? win32 API's are generally exposed as C

> win32 API's are generally exposed as C

A lot of newer Win32 APIs are IUnknown-based COM interfaces. APIs that are probably relevant for Chrome are shell interop, drag & drop, Direct2D, Direct Write, Media Foundation.

Does this option give you full debug information? My impression was that it does not. I require full debug information for debugging and profiling, and with clang-cl I get that (context: I work on Chromium)

My memory is that it supports DWARF debug info. So you get full debug info in DWARF and can use the mingw port of GDB on Windows.

Still, this is very different from the first-class support for compatible debug info that Clang provides.

One of mingw{,-w64}'s major contributions is a gcc-compatible replacement set of headers and import libraries, instead of using the Windows SDK or MSVC versions.

These work great most of the time, but if you start hooking deeply into windows, you realise they are a little bit outdated or incompatible. e.g. I get a lot of errors like

    #warning COM interfaces layout in this header has not been verified.
    #warning COM interfaces with incorrect layout may not work at all.
    #pragma message: Interface Ifoo has unverified layout.

in COM code, and some ATL things it is unable to build at all.

mingw-w64 llvm maintainer/developer here.

Over the past 2-3 years, with great help from a few other LLVM devs I have slowly pushed for native clang support for mingw-w64. With the current in tree HEAD you can now build and bootstrap mingw-w64 with a llvm-only toolchain (no binutils or gcc, thus no disregard of the PECOFF SPEC)

The stack is as follows. LLVM+CLANG+LLD+COMPILER-RT+LIBCXX+LIBCXXABI+LIBUNWIND+MINGW-W64.

The libraries and executables built this way can be dropped into visual studio quite easily, the two environments are now basically interchangeable. This also means we can borrow the PDB debugging work by Zachary Turner and/or any msvc features added to llvm for free that was done by the various googlers working on chrome. See a talk from Reid on the most recent LLVM developer meetup for more info on PDB and how it works. There has been no formal release of this environment yet but you should see something in the coming weeks/months. I keep build scripts here for those interested.

https://github.com/martell/mingw-w64-clang

I'm due another cleanup of the repo now that wine 3.0 is out so I can actually run x64 tests within a linux docker container. I'm going to be doing some blogging to make this more visible with various partners that want to support this.

The most interesting part of this for me was implementing a llvm-dlltool alternative to binutils dlltool that actually works within the PECOFF spec so msvc lib.exe and link.exe likes what it produces.

https://reviews.llvm.org/rL308379

This sounds great, it's been a long time coming and I'm sure many people will appreciate being able to debug their mingw generated executables in Visual Studio.

I did all of the PDB stuff in LLVM, happy to help if you need it (ping me on the mailing list or IRC)

Updated the comment to reflect this. :) Thanks for all the good work. Will ping you when I move onto that stage for support.

I've been watching the process of getting git to build with VS, but it's sounding like mingw-w64 will gain PDB output faster than the VS version will happen. Either way, if I can debug (and profile) git.exe then I'm happy.

Excellent work. I like to ignore visual studio completely and pretend it doesn't exist, but that's just me :) I've used cygwin's cross-compiler x86_64-w64-mingw32-clang package once in a while, though I find more projects' build systems choke on it and do the wrong thing (libgmp as a notable example) than with x86_64-w64-mingw32-gcc.

I tend to ignore both gcc and MSVC. Clang is really good at bootstrapping itself with this setup even when cross compiling without any real complexity. Without diverting into a cmake vs autotools battle I like that llvm and clang just work and bootstrap so easily. Gone are the days of messing around with gmp isl and co to bootstrap a toolchain.

Fair enough, but I actually need gmp more for runtime math, plus fortran deps like lapack and more obscure things. Maybe flang will be production-worthy some day - do you know if they're still outputting a textual representation of IR into string buffers?

Great work! We with Nixpkgs/NixOS have over the past year overhauled cross compilation, culminating recently with a bunch of parallel efforts recently including Windows cross compilation. People have built Nix with cygwin in the past, it would be nice to get it working in MSYS2/mingw-w64 too.

Are you targeting Linux, building on Windows?

No reason we couldn't expand into doing that!

Thanks for your work. mingw-w64 is amazing!

> On the other hand, if all platforms use the same compiler, if it builds on your machine then it’s probably going to build on all platforms.

This also works the other way: buggy code that "works" in Clang may now go unnoticed.

This means it will be harder to build Chromium using GCC or MSVC. I see that Arch Linux already has to depend on Clang to build Chromium [1], which isn't very reassuring.

[1] https://www.archlinux.org/packages/extra/x86_64/chromium/

Great work by the Clang team, well done!

I'm a bit surprised to see that build times in Clang are worse, and that resulting binary performance is comparable.

Maybe that will change when the Clang build starts to use LTCG, as the article mentions.

Yeah, just things that aren't optimized yet. Windows headers, etc have very different hot spots and need different paths to be fast than linux/mac ones.

For reference, in distributed builds, the build times are ridiculously faster using clang. (it shaves many hours off the time).

Is that due to the use of PCH in MSVC, so the hotspots only happen if the headers change?

What does clang offer in this regard? Is incremental build sufficient?

No, clang offers precompiled headers as well.

It's literally things like "parsing large numbers of pragmas is slower in clang because nobody has bothered to optimize it".

This is not uncommon for very different codebases.

Clang was, for a long time, many times slower at compiling the linux kernel, for example, because the linux kernel headers have a huge amount of inline assembly and nobody ever bothered to optimize the parsing paths for inline assembly. (now somewhat fixed).

This is just the very typical "hey, when you change the use cases dramatically, you may need to retune/rethink how certain things are done"

I think people just may not realize how wildly different these types of codebases are in what types of statements/things occur.

I didn't realize that clang is mostly a drop in replacement for cl, and matches the C++ abi. That in of itself is awesome.

clang isn't. clang-cl is (after years of work by the clang-cl team at Google). They had to do a lot of work, especially around PDB (symbol-file) generation.

More precisely:

clang-cl is a drop-in replacement for cl (even the command lines are the same).

But clang-cl is just a driver for clang, and it's still possible to get the ABI compatibility without clang-cl, but you have to know what you're doing and get the flags right.

For those interested in Clang history, here's Chris Lattner giving a tech talk at Google introducing LLVM and Clang.

https://www.youtube.com/watch?v=029YXzHtRy0

The Clang discussion begins at 18:53

Interesting. It seems Google is moving away from anything gpl/gnu lately (fuchsia, etc...)

Yep, GCC is no longer supported on Android.

I think it is mainly due to their concerns over the Tivoization clause which is added to GPLv3

That's true for any developer of embedded software. Any one who makes hardware of any kind will avoid trying to put any GPLv3 licensed code on their hardware.

Again, no, we didn't care.

First: We were already publishing all our GCC changes, as we did all our work upstream (we worked on a staging branch in the gcc repo, and changes were moved to trunk as fast as possible) Plenty of other companies used our branch and we were happy with it.

We do all our work upstream for clang/llvm as well, and pull compiler releases when upstream is green according to our internal testing. (Pretty much the only cherry picks are a small number of reverts sometimes)

Second, literally nobody has ever come and expressed a concern about using GCC to me.

I've had to explain the runtime and libstdc++ exceptions to people, but that was the other way around -- They weren't worrying, and I needed to make sure they worried and shipped source when they ship it as a .so.

I doubt this is very relevant in the case of Chrome compilers. It's more about the huge investment/expertise in Clang- or LLVM-based dynamic analysis and development tooling.

It's just the better platform for writing these things. Heck, this is what is was built for in the first place. GCC is a solid, but also a very traditional compiler.

> Heck, this is what is was built for in the first place.

LLVM was supposed to be a virtual machine environment (hence the name); I remember when it came out and I was in college doing program language research. It entirely dropped that history and got twisted into a compiler platform much later. And clang only came into existence because the developers of g c (in particular, Richard Stallman) did not want to work with Apple on maintaining LLVM as a gdc backend (which was a stupid stupid decision: Apple said they would be willing to give the copyright of LLVM to the FSF, which would have kept all compiler tech under GPL), which was all well after Apple had been switching to LLVM for their compiler stack (in order to obtain weird optimizations).

>(fuchsia, etc...)

Of all the reasons, I doubt they're working on fuchsia as a GPL license workaround.

To add (baseless) speculation to the mix, I think it's more NIH than GPLv3 concerns...

So, as the person responsible for deciding to make just about all of the compiler switches happen (and i used to run opensource licensing for Google), this is just coincidence[1].

In this case, we just felt clang/llvm was a better architecture to build on at this point. These are also many-year projects.

I was a GCC/gdb/etc maintainer for many years, so i've got nothing against GPL software.

[1] Well, part of it is that by numbers, most newer open source projects are not GPL. The percent of GPL'd new projects before, say, 2010 and after 2010 is very different. I haven't gone far back to see when that starts, but the percentages are quite significant. It used to be 40-50% of new projects, and now it's like 10-20, maybe.

Thank you for taking the time to shed some light on this. I understand your point about a multi-year project. I actually worked on a debugger before (to support a proprietary complier) so I know decisions in that area are far from random.

As a note for myself, I should check out llvm in greater detail.

With Microsoft producing MSVC/C2 compiler with Clang frontend, I am surprised they did not abandon the MSVC at all - they have a hard times even at supporting C99, not to mention modern C++ technologies. And all those tools and plugins built around Clang/LLVM. MSVC is a legacy of XX century and should be abandoned for good.

My impression was that problem with C99 support was that Microsoft was not willing, not that Microsoft was not capable.

IMHO, Microsoft has been doing a pretty good job. I don't think it's as bad as you make it out to be:

https://msdn.microsoft.com/en-us/library/hh567368.aspx

There's a link in the article here, but sadly it sounds like they've abandoned Clang/C2. I guess they got normal MSVC up to speed on the C++1y/z they wanted to support, and Clang/LLVM was always better than both at complex numbers, inline assembly, and other C features.

They have officially abandoned Clang/C2.

https://blogs.msdn.microsoft.com/vcblog/2017/05/10/c17-featu...

if a comment on a blog post counts as official

Do you know who Andrew Pardoe is?

It is as official as Google and Apple devs doing blog posts on Medium.

They don't have any hard time supporting C99, from Microsoft point of view C is done, time to move on to C++.

> We do not plan to support ISO C features that are not part of either C90 or ISO C++.

https://herbsutter.com/2012/05/03/reader-qa-what-about-vc-an...

All the updates regarding C99 and C11 in Visual C++ are related to ANSI/ISO C++ requirements regarding C compatibility, as Herb states.

Regarding legacy, Visual C++ is one of the top C++ compilers doing incremental compilation and linking, which still beats Rust compilation times on Windows.

Microsoft backpedaled not longer afterwards. See, e.g.,

https://blogs.msdn.microsoft.com/vcblog/2013/06/28/c1114-stl...

and

https://blogs.msdn.microsoft.com/vcblog/2013/07/19/c99-libra...

They even added C99 compound literals and designated initializers:

https://msdn.microsoft.com/en-us/library/hh409293(v=vs.120)....

Compound literals could never be added to C++ because the syntax and semantics conflict with existing C++ constructs.

C99 and C11 library support is required by ANSI/ISO C++.

As for the minor language changes, there was a blog post that I cannot locate now, stating it was to the extent required for porting well known projects to Windows, the remaining features would depend on key customers feedback.

> They don't have any hard time supporting C99, from Microsoft point of view C is done, time to move on to C++.

If that's the case then Microsoft should get its corporate ass out of C's standardization committee. It makes no sense to waste their time forcing requirements onto a standard that they refuse to comply with.

They don't have anyone on JTC1/SC22/WG14, where did you imagine that?

I’m sure you won’t like this comment, but doesn’t Microsoft have a vested interest in keeping C as unchanged as possible? If C were to run away in unorthodox directions, that would hurt compatibility with C++ and hence Visual Studio. So from their perspective it may make perfect sense to stay on the committee to keep it as conservative as possible. Why voluntarily give up soft power?

Who are the "we" they refer to in this blog? Are they Google's devs who build chrome, or are they clang's devs? If they are clang's dev, why are they building chrome? I am completely confused on this one.

Google employs a large number of clang devs. I would not be surprised if it was close to 50% of all clang (and LLVM) devs.

"We" as Google

And why not use mingw gcc? Do they find somthing wrong with it?

Edit Mingw-w64 its quite nice and includes gcc without all stuff cgwin does.

It's not ABI compatible with MSVC

Unfortunately, mingw profiling code and the toolchain is tainted with GPL.

Isn't it "GPL + GCC exception"?

Why use it? llvm is the future anyway and lots of players have a stake in it and contribute to it.

In fact, there are way too many c++ compilers if you ask me. I would gladly accept a world where clang is the one and only.

> In fact, there are way too many c++ compilers if you ask me. I would gladly accept a world where clang is the one and only.

Not me, IMO clang is where it is due to competition.

>I would gladly accept a world where clang is the one and only.

And luckily you don’t have any saying in such matters.

Sure. I just vote with my feet, like many others.

Lack of PDB support is one reason:

https://stackoverflow.com/questions/19269350/how-to-generate...

Lack of VC++ ABI support would be another reason.

Performance of the resulting binary, if I was to guess.

1) ABI compatibility is not present. With clang-cl you can compile half of your object files with MSVC and half with clang-cl, and link them together. That's very powerful. You might be thinking "but who is really gonna do that anyway?" However, there is more to the ABI than just this. For example, consider what happens if you need to use Structured Exception Handling. Well, with GCC, you can't.

2) Debug Information. GCC does not emit PDBs, it emits DWARF. This means you cannot use Visual Studio to debug, you cannot use vTune or WPA to profile, you cannot use symbol / source servers.

3) There are licensing implications, GPL != BSD and it's easier to use BSD.

These are the main reasons. There are other reasons though, such as if the entire goal is to get down to 1 toolchain and that toolchain is X, then why would you waste your time converting Y to Z?

> if you need to use Structured Exception Handling. Well, with GCC, you can't

You can on 64 bit.

I used to use mingw to cross-build a project.

At one point, after updating the toolchain from gcc 4.x to 7.x and from mingw32 to mingw-w64, it started to produce broken binaries.

The manifestation of that brokenness was that binaries refused to start properly on systems where the toolchain was not installed. It just silently exited during startup - no crash, or error message, or anything.

This applied both to binaries that were cross-built on linux, and binaries built using MSYS2 on Windows.

I decided to just switch to clang-cl + MSVC headers instead, which worked properly. The only gotcha there was the need to store the headers on a case insensitive file system (I chose vfat) when compiling on linux, because the case of the includes aren't consistent (e.g. Windows.h vs. windows.h), even in the MSVC headers themselves.

were you missing compiler runtime libraries? like any code compiled with gcc anywhere that links the runtimes as shared libraries, they'll need to be present for executables to run. depending which variant of mingw-w64 you used, there are more runtime library dependencies than there used to be, like winpthreads for c++11 thread support.

Fairly certain that wasn't the case.

As far as I could tell, the only DLLs the binaries referenced were msvcrt.dll and standard windows DLLs (KERNEL32, USER32, etc).

The application was also plain C.

Even if missing compiler runtime libraries were the issue, I would have expected the program crash or fail with an error message.

1. Not abi compatible

2. Not performant (Relative to other solutions)

3. Would be the odd man out. Literally all other development was on clang/llvm at this point.

Supporting mingw would have meant duplicating all performance/etc work for GCC.

4. Chrome also wanted other benefits that clang/LLVM would bring them (easy ability to build various forms of refactoring and analysis tooling, etc)

Last time I tried to build Chrome/Chromium I had to mess about with this build system called Ninja. (Which worked well it was just awkward to set up on Windows). Is that still in play?

Yes, ninja is still used.

Ninja is used because it's simple, multiplatform and very fast. The mainstream alternatives offer at most 2 out of 3.

"it's simple": yep, so much so that you inevitably need a meta-build system. In the context of Chrome, the build system only offers 2 out of 3.

It's by design that you need a meta-build system. Ninja build files were not designed to be written by hand, not even ninja itself has hand written ninja files.

Yes, I understand that it's designed that way.

My point was that, in the context of Chrome using ninja, the claim that "Ninja is used because it's simple, multiplatform and very fast. The mainstream alternatives offer at most 2 out of 3." is a bit odd considering that the decision doesn't seem to be ninja vs mainstream alternatives but rather ninja+gn(or gyp or whatever else) vs alternatives.

Can you elaborate on the difficulty on Windows? If you go to the home page there is a section "Getting Ninja" that links to a page with a zip file... ?

It was too long ago for me to remember specifics. Once ninja was up and running the build process was pleasant.

Did the LLVM folks fix the C++ structured exception handling incompatibility between MSVC compiled DLLs/programs and LLVM compiled ones? This was a problem for many years.

Clang 7 documentation has a section on MSVC compatibility, https://clang.llvm.org/docs/MSVCCompatibility.html

From that document:

Asynchronous Exceptions (SEH): Partial. Structured exceptions (__try / __except / __finally) mostly work on x86 and x64. LLVM does not model asynchronous exceptions, so it is currently impossible to catch an asynchronous exception generated in the same frame as the catching __try.

Yes.

(To the extent possible. There are some edge cases left, but they're very rare. Essentially all the real-world code we've seen works now.)

Tried to install Clang 5.0 Visual Studio integration, unfortunately it seems that I need to install VS 2015 (v140) compiler since it doesn't work with VS 2017 (v141).

I've tried clang for windows twice. Each time the problem hasn't been the compiler, which works well, it has been which library to use with it on windows. Clang 5.0 worked well up until I needed the filesystem library, which isn't there. Does anyone know if it still relies on the visual studio C++ library?

clang-cl is just a compiler. It relies on VS to provide the C and C++ standard libraries. VS 2017 has the filesystem library and it should work with clang-cl 6.

To elaborate on that, if you want to build Chromium for Windows you currently still need to install Visual Studio. We don't use its compiler anymore but we use its linker and other tools, its header files, and its libraries. Many Chromium developers also use it for editing code, and for debugging code.

Chrome is currently built using VS 2017 15.3 (that's update 3). Eventually we will probably upgrade to VS 2017 15.7 in order to get more STL conformance, linker improvements, etc.

You need the MSVC build tools and the Windows SDK. You don't need the entire Visual Studio.

Can you get the VS 2017 build tools separately any more?

Yep: https://www.visualstudio.com/downloads/#build-tools-for-visu...

Reply on news.ycombinator.com