Hacker News Reader

Comment by ryanmolden | Hacker News Reader

I am a bit confused. That thread seems to claim that Microsoft isn't producing a 64 bit linker, my install seems to disagree with them (vc\bin\amd64). Then you claim that the linker won't produce 32 bit code, but /machine:x86 seems to disagree with that. I tested this by building as 32 bit, then just repeating the link using the 64 bit linker and /machine:x86, worked fine.

So what are their problems exactly? That they are building with VS 2005 (from what I saw in that thread) and MS isn't backporting changes to the 2005 toolset?

View on HN

The problem is link time code generation. With link time optimization[0], which is a prerequisite for profile guided optimization, cl just compiles the source into some intermediate representation. Actual code generation is all done in link.exe at the final link. And /machine:x86 doesn't work with link.exe when using -GL. The x64 link.exe is not capable of generating x86 binaries with LTO.

[0] http://msdn.microsoft.com/en-us/library/0zza0de8%28v=vs.100%...

(Mozilla is building with 2010 now.)

> (Mozilla is building with 2010 now.)

Also note that the problem still exists in VS2012, and in fact is even worse because the linker memory usage has gotten higher in general. Fortunately we now have a fairly good workaround (developed with help from Microsoft -- maybe some of the people in this thread?) that limits which files participate in PGO:

https://bugzilla.mozilla.org/show_bug.cgi?id=871712

Ahh, one variable I hadn't factored in. The mix-n-match option mish-mash of C++ is so fun :)

The problem is cl.exe for x86 is only 32-bit, and the design of most build systems don't make it easy, especially as Mozilla was using PGO.

You mean it only produces 32 bit code? This disagrees (http://msdn.microsoft.com/en-US/library/x4d2c09s(v=vs.80).as...) going back to at least 2005.

The following list describes the various versions of cl.exe (the Visual C++ compiler):

...

x64 on x86 (x64 cross-compiler) Allows you to create output files for x64. This version of cl.exe runs as a 32-bit process, native on an x86 machine and under WOW64 on a 64-bit Widows operating system.

What they are currently doing is running a 32-bit linker to produce 32-bit output on 64-bit windows.

This 32-bit linker is apparently LARGE_ADDRESS_AWARE, so running it on 64-bit windows gets them 4GB to play with (which they are apparently rapidly burning through.) What they need is a 64-bit linker to run on 64-bit windows that can produce 32-bit output.

      OS  Linker  Output  Mem
  1   32  32      32      3GB
  2   32  32      64      3GB
  3   64  32      32      4GB
  4   64  64      32      8TB

They use to be doing 1, now they are doing 3. They need to do 4, but apparently cannot. My understanding of this may be wrong, or they may be wrong, I don't know.

You need to add cl.exe to the table.

I'm not sure where that fits in. Are they wanting to use a 64-bit cl.exe and a 64-bit linker to make 32-bit output? Does mixing and matching the cl.exe and the linker change how much memory the linker can use?

I am baaarely familiar with Microsoft-world development.

Yes, they can use a 32-bit cl.exe and 64-bit linker, but most build systems don't make it easy.

You can't use a 32-bit cl.exe and a 64-bit linker to produce 32-bit binaries if you're doing PGO (which is the whole point of this "using lots of memory" issue). In particular, the 64-bit linker can't produce 32-bit binaries when you're using link-time code generation, which PGO does.

I am talking about the cl.exe executable that output x86-32 code itself, not the code it produces. You can use a 32-bit cl.exe to cross compile 64-bit code, but not the other way round.

Ahh gotcha, I hadn't look at that angle. That is unfortunate.

Reply on news.ycombinator.com