Bringing Clang to Windows | Hacker News Reader

Bringing Clang to Windows

blogs.msdn.com

142 pointsintrepidsoldier66 comments

Open articleView on HN

I agree with folks who think this is a bit confusing.

This blog post is announcing:

1. Using clang as a frontend.

2. Using C2 (MS compiler) as a backend.

What the LLVM community (IE not MS so far, but participation is of course, welcome)) has been building:

1. Using clang as a frontend

2. Using llvm as a backend

and even more specifically

1. Using MS Headers + MS Libraries

2. Using clang as a frontend

3. Using llvm as a backend.

(see blog.llvm.org/2014/07/clangllvm-on-windows-update.html for a 9 month old view)

Some of the mingw32 folks have been working on replacing #1 with non-ms stuff, but let's put that aside for the sake of this description )

Using clang as a frontend is where almost all of the ABI compatibility happens anyway (and where various developers have been focusing), so you end up with VC++ ABI compatible binaries whether you use C2 or not, if you/they use the pre-existing clang stuff.

Whether they are doing that (and then lowering llvm ir to c2 ir) , or if they've built an alternative IR generation with clang (IE generating c2 ir directly from clang), that they haven't released, is not clear.

If they've built something that generates C2 IR from clang, they would have had to write their own ABI compat, which would mean it may be different than the binaries clang would build.

(exception handling is also codegen'd by llvm, so exception handling could end up different, though of course, compatibility is the goal)

Hi DannyBee, sorry if we didn't get the message out clearly. Jim Radigan gives a deeper technical explanation in the build talk he did yesterday. http://channel9.msdn.com/events/Build/2015/3-610 It's translating LLVM IR to C2 IR. We know that the Clang/LLVM community is hard at work on implementing the MS-ABI and we've reached out to a few developers working on ABI stuff as well as symbols support. We intend to engage with that community more openly now that we've announced but of course shipping the code will be our top priority. One important contribution of the Clang/C2 approach is that we will have full native VS debugger support. This is just the first step in a longer journey. Thanks, Steve, VC Dev Mgr

Hello! Is there any chance that this would work with other language front-ends that generate LLVM IR?

see Jim Radigan's build talk on compiling Objective-C for windows for more info on this :)

We went through this at my former employer (not MS) just before i left. In fact the politics surrounding it drove me to seek relative sanity of doing a startup. When I started working on it, there really were two camps: clang emitting house IR instead of LLVM IR, or translating LLVM IR either early (right out of clang) or late (as a target after all optimization). I was working on the prototypes for the latter pair of options before another constraint was added that invalidated all of those paths.

The problem was that clang is quite intertwined with LLVM IR. ABI is spread all over the place, both FE and BE. It was proving hard to get something the house optimizers/code generators would be happy with. So a more radical path was chosen — wiring clang's parser + ASTs to another C++ front-end's AST and then using the old IR generator to get all of the idiosyncrasies right for the target. It was wacky, but seemed to be half-way working when I left, such that major open source code was compiling.

The whole idea of taking just parts of clang/llvm reeks of a half-measure anyway. It's probably better to either keep the existing compiler stack, or cut it right out and put the secret sauce into LLVM as add-in modules.

if the other compiler doesn't happen to have a great AST story, then this approach looks difficult.

Can the clang+LLVM toolchain generate a PDB file that can be used by WinDbg? Can it generate .obj and .lib files that can be statically linked with those generated by Visual C++?

No, it can't generate a PDB file itself, at least not today. Maybe in the future the MSVC linker will be able to generate one even though you compiled with Clang/LLVM (it does that already today but the PDB file is almost empty).

You can definitely build obj and lib files and link them with files built by MSVC. That works well.

We also support putting DWARF debug info (the standard format) into the binaries, and the LLVM linker LLD will link them and you can debug the resulting program with the LLVM debugger LLDB (or GDB for that matter).

Hi Chandler! it's probably worth mentioning that we will be also opening up our lldb-mi based debug engine which will allow you to debug any lldb target from Visual Studio. And we'll continue to contribute back improvements to lldb-mi. Thanks, Steve Carroll, VC Dev Mgr

I'm a bit confused about how this will work exactly. If I have Clang and VC++ compiled code trying to interop with each other, what happens if Clang decided to use a different memory layout for std::string (or any other type really) than VC++? What compiler is doing which task in the pipeline here exactly?

I mean, the reason everyone does interop strictly in C is that there are zero standards for how a compiler implements the C++ features exactly down-low.

Clang has, for some time now, supported the MSVC ABI.

Here is their implementation of name mangling: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/AST/Micros...

Here is their implementation of record layout: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/AST/Record...

Here is their implementation of vtable generation: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/AST/VTable...

Here is their implementation of RTTI: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/CodeGen/Mi...

Here is their implementation of C++ throw metadata: http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/CodeGen/Mi...

But none of that was written by Microsoft employees, so I'm a bit confused about what exactly this blog post is announcing. It feels a bit like they're taking credit for the work done by the clang community.

We are announcing that we will ship a hybrid compiler that combines the clang frontend with the c2 existing backend. We will fully support that for building portable code within Visual Studio for Windows. We definitely don't want to take credit for the clang community's work. We have a ton of respect for those developers. We intend to make contributions back to the community and we've started to reach out to some of the relevant parties. Thanks, Steve, VC Dev Mgr

To be clearer, "Here is their implementation of ..." should read as "Here is clang's implementation of ...".

AFAIK, Microsoft hasn't contributed to any of these ends. I, for one, hope that they do.

That's nice, I wasn't aware that they were working on this specifically.

This was not done by microsoft at all :)

As I understand it, it's not bringing Clang/LLVM's libc++ to Windows. The memory layout of, for example, std::string isn't decided by the compiler - it's decided by the library/header files. Both Clang and MSVC would be using the same MS C++ header files, so the only thing up to the compiler would anything that isn't specified in the header files, behind the scenes.

We're already using Clang with msbuild on Windows for native C11 support (come on, Microsoft!), courtesy of pre-built Clang for Windows binaries [1] - using the Microsoft C/C++ headers and linking against the Microsoft C/C++ runtimes.

There's already a plugin for MSVC that makes the Clang toolset available. VS 2013 (or maybe it was 2012?) decoupled the MSVC build chain/toolset from the IDE, making it possible to swap out which toolchain you're using from the project properties. Actually, it may have even been VS 2010, because I was using it to compile with ICC, but then again, I remember having a hell of a hard time so perhaps it was VS2012/2013.

1: http://llvm.org/docs/GettingStartedVS.html

correct, this will use MS libraries. If you watch Jim Radigan's Build talk (he owns C2), he talks a little bit about how the ABI is really a part of the platform and it might clarify some of this. http://channel9.msdn.com/Events/Build/2015/3-714 about 23 minutes in. Thanks, Steve (vc dev mgr)

Clang was already planning on using MS libraries/headers.

IE this is literally not different from what was already being done, without MS :)

yes, we are well aware that clang and llvm is working on targeting windows. this is clang+c2 which we will fully support.

"The memory layout of, for example, std::string isn't decided by the compiler - it's decided by the library/header files."

Not quite true. Different compilers can decide to do different alignment/padding. It should be fairly simple to make clang and MSVC avoid that scenario, though.

Also, they must have taken care to use the same ABI and C++ name mangling rules in both compilers. I would guess Microsoft's version of clang follows MSVC here, so that one can link clang code to existing third party libraries.

It's not easy to mix two different versions of code compiled with different microsoft compilers, or even with the same compiler but different settings.

I'm starting to see a strategy here. I've been wondering how Microsoft can make money from all.this open source activity. Their platform is becoming obsolete, but is being replaced by OSX, iOS, and android. That makes developers unhappy because of the multiple OSes they need to support. But MS can step in and bring their own platform as an abstraction over these others and make developers happy. Build once, target everything. I'm betting key pieces will not become open source, they want to have control over the new platform. That is the midterm goal - regain control (even a little bit) of a dominant platform.

Windows is not becoming obsolete. Quite the opposite. In the enterprise Windows will stay for a long time. I'm working at company > 500000 employees and our IT will be very happy when they can replace the iOS devices with Windows equivalents. And certainly the Desktops aren't going to be replaced by OSX. Even more with Windows 10 around.

MS strategy is much more straight forward. Simply offer reasons and make it easier for customers to use their products. Mainly the cloud and mobile services. They're just smart enough to understand where Google and Facebook make their money with - not by selling OS.

I agree. We've been using OpenLDAP for 3 years now, and it has been a very scary, black magic experience, just trying to get basic stuff like DNS, NFS, mac, windows clients, working. Hoping to move to Windows AD as soon as possible.

I disagree because bringing Objective-C, Java/Android, and CLang compilation to Visual Studio doesn't provide any sort of abstraction.

The support for Objective-C, Clang, etc is, in my opinion, merely about bringing iOS and Android software to Windows (especially for phones and tablets). Microsoft knows they need current high quality apps on their platform to get users. Users which will buy their devices and operating systems. They don't really want developers to run iOS software unmodified on Windows but they do want to make it really easy for them to attempt a port.

Given the amount of application code that has been written for Win32 over the years and that is still in use, I do not see any viable alternative to Windows on corporate desktops.

Sure, you might have VDI and access a Windows VM remotely from your iPad or whatever, but Windows is not going away for a long time.

(And don't get me wrong, I would love replacing every single Windows desktop at our company with Debian or CentOS or even OSX, but unless a lot of companies decide to port their applications to one of these systems, it is not going to happen.)

(sigh)...

Hell yeah! Microsoft just became one of the viable platforms this week in the cheap segment with reasonable licensing.

Still not in the free as in speech and beer segment though.

"So, any minor divergence in features of Clang and Visual C++ will cause you as a developer to have to worry about these compiler differences or limit you to using the smallest subset of the two compiler features"

In other words, the set of vc++ features.

Yes. We know we are behind on conformance.

Wow! Microsoft! Wow! You should've had Satya Nadella as CEO ages ago! Congrats!

At approx. 26min he mentions four processors that their code generator has to target: ARM32, x86, x64, and a new one coming out in the fall. What would that be? ARM64? *edit - never mind. Last five seconds they hint at it as arm64. MS is definitely doing some interesting stuff these days.

If the Windows team at MS thinks of c2 and the MS linker as part of the Windows ABI, then I guess the MinGW toolchain (and a hypothetical clang toolchain where LLVM generates the native code) will always be second class, and therefore best avoided by developers who want to target Windows, even for classic Win32 desktop apps.

It all depends on how cutting edge you want to be in the windows platform. If you want access to the latest technology, MS tools will always be the necessary choice -- Borland learned this lesson after a lot of hard work. But if you want to create products using the stable version of the windows API, gcc is already a very good choice.

I kind of disagree here. Compilers reached commodity status a long time ago. Nobody makes money off them compared to how it was in the 90s and the list of ex-compiler makers is as long as your arm (Watcom, Metrowerks etc).

'clang' is probably approaching the point that its good enough so the question for Microsoft is do they still see any value in continuing the develop their own compiler or should they just switch to 'clang' ?

I think this is the big shift we are seeing in MS this year. Instead trying to owning everything, they seem to be shifting their focus to owning the parts that have value and commoditizing the rest. The strategy was successful for Apple so its unsurprising that Microsoft is looking to a similar strategy as part of their salvation.

I agree with you. What I am going to say might be foolish, but if you push this philosophy further, Windows might end up with a linux kernel. There is no value to make in the development of a kernel, they should stop it ! And since linux is better than wondows_NT, they should switch to linux.

I know, I know, that's foolish, but it would be so great if that could be true ^^

I think they want compatibility with other compilers, but unless they move alway from Windows, there is a lot of their APIs that require close compiler/library support: COM, DCOM+, .NET, etc. They will probably need at least to maintain VC++ for a long time.

Add WinRT to that list and you have a set that is somehow reason enough for Visual C++ compiler front-end to stay relevant.

Latest technology? What latest technology?

If you mean C and C++ features, Microsoft standard conformance has slowly improved over the years from not giving a fuck to reasonable, but the only bleeding edge tools they've ever created are experimental research projects. They are completely out of clang and GCC's race to implement C and C++ standards and future standard changes.

If you mean some new API, they are libraries and all compilers have equal access to them.

The broader stroke is MS to be relevant in the platforms side of the story. Sometime people will argue that windows is used extensively in enterprises, but obviously the today startup which will be tomorrows enterprises are using almost no MS eco system.

Where does C2 come in on the stack? Is it compiling/translating LLVM IR?

The latter.

What's the situation with SEH these days in Clang?

http://clang.llvm.org/docs/MSVCCompatibility.html says minimal. Note that this is about cross-plat conformant c++ code. SEH is windows specific.

That document is out of date. SEH works OK for 64-bit X86, 32-bit X86 is ongoing. Catching C++ exceptions has been implemented for 64-bit X86 but isn't ready for prime time.

> and is fast becoming the preferred C++ compiler for non-Windows platforms.

They wish. Microsoft still hates GNU.

It is, though. And Microsoft isn't alone in hating GNU - FreeBSD no longer ships with or uses GCC; the entire FreeBSD kernel and userland is built with Clang these days. And, of course, Apple's the one that started this whole stampede away from GCC in the first place. LLVM/Clang really and truly have hit GCC hard - and definitely for the best. Look at the improvements to GCC spurred by newer features in the Clang build toolchain.

EDIT:

In reply to davidgerard's comment below:

> I will not believe this falsifiable claim without numbers. FreeBSD has almost no users compared to Linux.

Non-Windows does not mean Linux.

And what falsifiable claim? They didn't say Clang is the most popular - they said it's becoming. But if that's really the game you want to play, then it's Linux that has almost no users compared to the number of iPhones/iPads/iWatches/MacBooks out there - Clang/LLVM-powered, each and every one of them.

There are several reasons why FreeBSD is ditching GCC, one of them is licensing. I am wondering what are the technical reasons though...

There was a Stack Exchange question answering exactly this question:

http://unix.stackexchange.com/questions/49906/why-is-freebsd...

Another contributing factor may be momentum. Clang is easier to work with than GCC, and doesn't have the baggage of GCC, so more and more people and companies are starting to use it, fund it, and commit code to it. It's been improving in leaps and bounds, and still rapidly improving. It's only a matter of time before it's all around better than GCC.

> I am wondering what are the technical reasons though

This is based on fairly limited experience, but I believe it's simply a better toolset. LLDB was easier to work with for me. The error messages make things quicker to diagnose. I prefer the added verbosity for things like flags.

The reason why I perfer clang is better -and colored- error message output.

GP actually used falsifiable properly - meaning that the claim is testable (evidence could swing both ways). A non-falsifiable claim would just have to be taken on faith alone or rejected altogether

I will not believe this falsifiable claim without numbers. FreeBSD has almost no users compared to Linux.

Define "user". You're on the internet. The bits you're downloading are probably passing through devices somewhere in the path that run FreeBSD.

~30% of North American traffic is from Netflix, whose servers are FreeBSD.

Everyone is a FreeBSD user and they don't need to know it to for it to count. Nobody actually gives a shit about numbers of users sitting on FreeBSD desktops.

Unless you think of iOS and OS X as flavors of FreeBSD.

Isn't Darwin forked from BSD proper, instead of FreeBSD etc?

Mac OS X is separately accountable, and that's <i>entirely</i> because Apple hates GPLv3.

That's not correct. Apple needed technical things from a compiler that gcc could not feasibly do, and started moving things to LLVM before GPLv3 was released. Even if gcc had been relicensed under a permissive license such as BSD, Apple would have moved to LLVM and Clang.

A couple examples. Core Image needs an optimizing JIT compiler that can take a chain of image filters and turn it into a single efficient filter for execution on the GPU or CPU (whichever will be faster on the particular system for the particular effects). XCode needs to parse C/C++/Objective-C code in the editor and in the debugger.

What Apple needed for these things was a modular compiler system, designed to work as a compiler toolkit from which you can pick and choose the parts that you need for your particular needs. A compiler toolkit that is designed to be easy to interface with outside tools.

Gcc was explicitly designed to not be that. It was only competition from LLVM/Clang that forced some liberalization onto gcc.

Not much of a hate but rather protecting their IPs by avoiding licenses like GNU. They are not alone in this, most of the big companies are avoiding it. Some companies use it and hoping for not getting caught (VMWare). I still don't see hate here, MS is just smart enough to avoid these sort of situations.

Microsoft is on record as hating the GPL.

If that's changed it's not showing in their careful use of the (very fine) Apache license rather than the GPL.

Microsoft haters are on record as attributing all manner of emotions to Microsoft decision-making, but from a detached perspective it all seems fairly dispassionate, if sometimes overly aggressive.

Services for Unix was full of GPL stuff. They're not protecting anything they have any sane threat model on. It's Microsoft being Microsoft.

Were they in violation of GPL with anything? I can't find credible reference.

I know that they identified the USB/DVD tool and re-released it under GPL.

http://www.zdnet.com/article/microsoft-admits-its-gpl-violat...

No, as far as I know they fulfilled all GPL obligations on the SFU stuff. And I know of no cases of Microsoft violating the GPL (and I would have heard if there were). Not wanting GPL is entirely "won't" rather than "can't".

Reply on news.ycombinator.com