Hacker News Reader

Comment by bonoetmalo | Hacker News Reader

Could somebody point me to a technical explanation of why it's sometimes non trivial to just compile your app against x86-64 and call it a day?

For example, something I encounter every day is Visual Studio and it's helper processes being 32 bit. Because Visual Studio regularly, even on the latest 15.7 preview shits the bed with OutOfMemoryExceptions on our large solution, I'm inclined to rage "why don't they just make it 64 bit? If it could just load more into memory it could get past this indexing hurdle and give me back the UI". But I also understand that if it was that simple they would have done it by now.

Something else, that I understand more, is the LabVIEW RT and FPGA modules only working on 32 bit LabVIEW. I would assume it's related to the compiling and deploying to the 32 bit ARM/x86 RT target.

View on HN

The main issue would be that 64-bit may cause an app to use more memory. Every pointer doubles in size, the alignment of a structure containing a pointer may grow, etc. Basically, if you needed a lot of structures allocated and they all grew, your memory use becomes noticeably bigger.

Sometimes legacy code can make assumptions about pointer size. These hacks were more common in the days of porting older systems to 32-bit but it could still happen moving to 64-bit.

If there’s code that tries to manually populate the bytes of data structures, sometimes bugs appear when the target field size changes (e.g. somebody ends up not initializing 4 of the 8 bytes in a now-wider field).

In the case of Apple, the huge pain will be their decision to not port all of Carbon to 32-bit (despite Forstall getting on stage years ago and stating that Carbon was going to be 64-bit “top to bottom”, it never happened). This can mean addressing a whole pile of nothing-to-do-with-64-bit-whatsoever problems before even starting to solve 64-bit problems.

> despite Forstall getting on stage years ago

Well we all know what happened to Forstall ... and the entire software development roadmap after he left ...

Apple's 64 bit strategy was in place by 2007 (It would hardly make sense to omit some 64 bit Cocoa APIs in MacOS X 10.5, only to bring them back later, would it?). Forstall left in 2012. Your alternate history timeline may need some work ;-)

I don’t think so. Was there a projected date for completion of the migration? We’ve only got pervasive 64-bit in the last 6 years so doubtful it would have been before then.

As far as I recall, userland APIs had all their 64-bit support in place in 10.5 (2007). Kernel 64-bit support came in 10.6 (2009). Which 64-bit features are you thinking of in the last 6 years?

To clarify, I'm referring to macOS releases. On iOS, 64-bit support obviously came within the last 6 years, but Carbon was never in the picture on iOS anyway.

Carbon, as per GP ...

> despite Forstall getting on stage years ago and stating that Carbon was going to be 64-bit “top to bottom”

This is the second time I've seen you make that claim. Can you, by any chance, recall on which occasion he said this (or dig up the exact quote)? AFAIK the plan was always to use the 64 bit transition as an opportunity to shed deprecated APIs, so to imply otherwise in public would have been rather irresponsible.

I don't know about that particular quote, but this has a photo of the WWDC side claiming 64-bit Carbon and Cocoa, so that was definitely the original plan

https://www.wired.com/2007/06/leopard-won-t-support-64-bit-c...

Hmm, that is plausible. I stand corrected.

The increase in pointer size is trivial in todays memory limits. An application with a million pointers will only use 4 megabytes more memory.

CPU cache will not magically double in size to accommodate, so performance will suffer

I'm pretty okay with Visual Studio using significantly more memory than it does now, if it meant it would stop thrashing the disk constantly and locking up the text editor. My dev machines have between 16 and 32 GB of RAM, and even the wimpiest laptop I've used in the past five years had 8GB. But anytime it gets close to 2GB of memory used, it starts turning into "Microsoft Visual Studio (is not responding)"

Usually it's old code that assumes sizeof(int) == sizeof(void *) - you could stuff a pointer into an int on 32 bit platforms, you can't when an integer is 32 bits long, and a pointer is 64 bits. In C (or Fortran, ...) it's pretty easy to make this mistake, especially in pre-K&R C - so there's some work to be done porting over.

If I recall correctly, Win32 let you stuff a 32-bit value into a window. If you wanted to write an object-oriented application, you'd stuff your this-pointer in that 32-bit value. Since every little thing (button, label, checkbox) is a window, you'd have a lot of that. Visual Studio might be old enough that they wrote the pieces that are still 32-bit in straight-up Win32, or maybe MFC. If so, I'd bet no one wants to touch it to port it over to modernity...

You're probably referring to SetWindowLong. A 64-bit compatible version (SetWindowLongPtr) of that function is available since Windows 2000.

I'm pretty sure there's a guide on how to properly port applications to 64-bit on MSDN somewhere.

I've always been confused by this: why aren't ints 64 bits on x86_64? They're 16 bits on 16-bit architectures, and 32 bits on 32-bit architectures, and that distinction was originally rather the point of using "int" as your type rather than short/long/etc., so... what happened?

"Word" only means 16 bits on x86; on 32-bit ARM, for example, it means 32 bits, and the architecture manual uses "doubleword" to refer to 64 bits. [edit: the parent's post was edited but originally asked about this]

Having int be 64-bits is known as ILP64 (int, long, and pointers 64-bit); some obscure systems handled the 64-bit transition that way (Cray), but Unix went with LP64 (long and pointers 64-bit), and Windows went with LLP64 (long long and pointers 64-bit). Here's an interesting document from 1997 comparing the approaches:

http://www.unix.org/version2/whatsnew/lp64_wp.html

Basically a matter of tradeoffs around compatibility, performance, and consistency.

Use "stdint.h" in all new code. I only use 'int' and 'long' for counters I know won't be that big, etc. I use stdint types for anything where I might care about the size.

Then you're instead going to break apps that assume ints are exactly 32 bits. There's no easy solution here.

When x86_64 was introduced, other 64-bits systems had made that choice, too. It was natural to follow that.

See http://www.unix.org/version2/whatsnew/lp64_wp.html and http://www.unix.org/version2/whatsnew/login_64bit.html.

A counter-example (the only one I’m aware of) is tru64 UNIX from Digital (later Compaq, later HP), now EOL. If you used “int” in C on that platform, it was 64-bit.

In C, you only have char, short, int, long and long long (signed and unsigned). Other integer types like uint32_t and size_t are typedefs to those. If int is 64 bits and char is 8 bits, you have to choose if short is 16 or 32 bits, and other one won't exist at all.

long really should be 64 bits of 64-bit systems, and it is on Linux. Windows kept it at 32 bits to make porting existing code easier.

First, it could be as simple as you use a legacy proprietary software that you don't have access to the sources to recompile it 64 bit, on macOS is less of a problem because 32bit was used for a short transition period, but if you think maybe you want to run 32bit Windows software with wine.

Another reason is that the program was not written with portability in mind, and so it works well on 32bit and on 64bit it has strange behaviors, this could be due to a infinite number of possibilities, and so if you want to use it on a 64bit you must not only recompile the program but debug and fix it.

And then it could have be done on purpose, yes Microsoft compiles Visual Studio 32bit on purpose, the reason is that the main advantage of 64bit is a bigger address space, and a couple more registers, otherwise on the Intel architecture the performance is the same, but with 64bit you consume significantly more memory, because every pointer inside you program is now twice as big: so if you program doesn't need an address space bigger than 32bit and you want to save some RAM, it's not a stupid idea as it would seem to still compile it 32bit. As it's not a stupid idea to use a 32bit OS on a PC with 2Gb of RAM or even less.

All the hacks that make assumption on some type sizes like pointers that will change on 64bit architectures will break your program.

Hacks are seen as "good" C programming, and encouraged, so this kind of thing happens.

Not even a case of hacks; before modern (C99) C, standard C didn't have fixed-width integer types at all, so behaviour can very easily change with architecture.

>>Hacks are seen as "good" C programming, and encouraged, so this kind of thing happens.

Perhaps among new developers. Us old-timers who have to maintain this stuff have learned to avoid "clever" code rather quickly and admonish those who write it. Elegant, to me, includes "easy to read and understand."

It could have been all of those things, but the real reason was to keep Visual Studio fast.

Here's a rather opinionated blog post on the subject, from one of the people originally responsible for making the call: https://blogs.msdn.microsoft.com/ricom/2015/12/29/revisiting...

In a nutshell, his position was basically, "If you can't do what Visual Studio needs to do in 4GB (four gigabytes!) of RAM, you really need to think a little bit harder about your data management."

I personally didn't have a terribly strong opinion either way, up until about a year ago when I switched to Java and started using IntelliJ on a daily basis. Now I have come to agree quite vehemently with Mariani's opinion on the subject.

And yet on large programs unless you are very careful about which symbols get loaded VS will crash deterministically while debugging. This gets worse and worse over time as windows itself pulls in more modules for the same code with every update.

4GB simply isn't enough address space to keep every symbol in memory. I could manually manage them but that should be the computers job not mine.

They've actually talked about why they haven't done this here[1]:

"So why not just move Visual Studio to be a 64-bit application? While we’ve seriously considered this porting effort, at this time we don’t believe the returns merit the investment and resultant complexity. We’d still need to ship a 32-bit version of the product for various use cases, so adding a 64-bit version of the product would double the size of our test matrix. In addition, there is an ecosystem of thousands of extensions for Visual Studio (https://visualstudiogallery.msdn.microsoft.com) which would need to also port to 64-bit. Lastly, moving to 64-bit isn’t a panacea – as others have noted (https://blogs.msdn.microsoft.com/ricom/2016/01/11/a-little-6...), unless the work doesn’t fit into a 32-bit address space, moving to 64-bit can actually degrade performance."

Also, a lot of people don't realize that there is a 64-bit version of the toolsets[2] (for C++ at least). I don't tend to have high memory use by Visual Studio itself but often run out of heap space using the compiler and linker so having access to those can be very helpful.

1. https://visualstudio.uservoice.com/forums/121579-visual-stud... 2. https://docs.microsoft.com/en-us/cpp/build/how-to-enable-a-6...

It's funny, Microsoft Office has the same problems to an even greater degree (vastly more users, more 3rd party extensions, an entire suite of apps each probably more complex than VS), and yet they've had a 64-bit edition available since 2013

I suspect that being one of Microsoft's primary cash cows, they'd also have vastly more developers to throw at the problem.

They've always recommended the 32-bit version to users who require compatibility with those 3rd party extensions:

https://technet.microsoft.com/en-us/library/ee681792.aspx

https://support.office.com/en-us/article/choose-between-the-...

Took them long enough, too. For the longest time they couldn’t switch because they were using Carbon instead of Cocoa.

The linker (especially when doing LTCG) can run out of space, even if it's the 64-bit one. There was some 2GB limit (or was it 4GB), but don't remember. Happened some time ago (years?) when tried LTCG on the WebRender (chrome/bink?) that Qt bundles.

I always scoff when I read this. “ I don’t run out of memory that means nobody else does”

My team regularly (on a daily basis) runs into high memory usage VS issues, which inevitably end up with VS hanging and being force killed and restarted. I d gotten to the point that I restart VS in the morning and at lunch every day to work around the issue.

> Could somebody point me to a technical explanation of why it's sometimes non trivial to just compile your app against x86-64 and call it a day?

Because the program uses a deprecated 32-bit API.

Once the deprecated 32-bit API is dropped, the program simply won't compile.

Also, the modern 64-bit API is different. And by different I mean it has a completely different design and set of interfaces. So to get the program to work again you have to port the program from using the old 32-bit API to using the non-deprecated 64-bit API. That's non-trivial work.

Edit: clarification

Are you talking about Carbon/Cocoa?

Because it is non trivial when using C or any derived language that has copy-paste semantics with C.

It starts by C not having fixed sizes for its datatypes.

Sure there were always macros/typedefs with such fixed sizes and C99 introduced stdint header.

However not everyone actually uses them.

Then there are the bit fiddling algorithms, unions and casts that assume a specific memory layout.

Followed by code that might actually become UB when switching to another memory model.

All of that scattered across hundreds of files, not written by a single person, with an history of decades of code changes.

But Visual Studio is not C. It's C#, and even it's C++ parts would be somewhat com-ish, dot-ish. I'm sure Microsoft is well prepared for 32-bit -> 64-bit transition. My guess is that there are probably massive amounts of plugins (not done by them) that are not prepared, especially ones that have to use native code. So Microsoft can't say No there. Just guessing, I dont' know for sure.

Visual Studio is more C++ than you're giving it credit for, it's largely old C++, and making assumptions about how forward-looking that C++ is is probably pretty questionable.

Also, yes, plugins are a large concern.

> making assumptions about how forward-looking that C++ is is probably pretty questionable.

Especially since Visual Studio didn't even compile C++ correctly until fairly recently. Variables declared with `for (int i = 0; ...)` would remain in scope after the loop, the same as if it had been declared outside the loop. I don't know when they fixed that, but I think I ran into this problem as recently as 2010. Meanwhile, GCC and everyone else had been doing it correctly for years.

The last version of the compiler with that problem was VC6 which came out in 1998. This version was superceded in 2002.

I wouldn’t say 16 years ago was recent. If you really ran into this in 2010 it would have either been using an old toolset or the compiler flag that lets you enable the old behaviour for compatibility if you need it.

Nope, C++ is the undisputed king of coding and I have no love for C++. Not only Windows is mostly written in C++ (and I would not be surprised if they use some C and Assembly as well) but even external projects like Unity (a popular game engine) that suppose to be used only as C# libraries they are just C++ projects. As long as Windows is by far the most popular OS on desktop and Android on mobile devices C++ remains the most essential, well supported, mature although infinitely ugly Programming Language.Because the OS remains still in the center of software development.

I doubt that there's a lot of C++ code in Windows, at least in the lower level system libraries. It should be mostly C (and that's a good thing). The same for Linux/Android and the BSD parts of OSX/iOS.

There is plenty of it, specially since Windows 8, which was when C++ code was officially supported on the DDK.

Since they decided they are done with C, Microsoft has been cleaning up Windows code to make their code compliant with the C subset of C++.

Since Windows Longhorn failure, with Vista COM got the main API role for the components that were originally designed to be written in .NET.

Which lead to the design of UWP as improved COM, using the ideas they originally had for Ext-VOS, but decided to create .NET instead.

https://www.reddit.com/r/cpp/comments/4oruo1/windows_10_code...

Ah alright, thanks for the clarification. Nothing in those direction changes convinces me that Windows will improve over WinXP and Win7 though ;)

I really wish the Visual Studio team would give C a bit more love. Getting at least full C99 support in is most likely less work than any random C++17 feature.

Their are done with C, there is no more love to give beyond what ANSI C++ requires.

ANSI C++14 requires C99 library, while ANSI C++17 upgraded it to C11.

Microsoft has contributed to improve clang on Windows for those that still want to keep on using C, but then good luck accessing COM and .NET APIs from C, it is possible but I wouldn't do it for fun.

Even the new C runtime library is actually written in C++.

It would be fine with me if I could simply compile any valid C99 or even C11 code with Visual Studio C++ compiler, but C99 support is only half-assed, and C11 support probably will never happen.

D3D11's C API works fine (that's the only Windows API I care about apart from the usual Win32 windowing stuff), but already D3D12 let that rot and is only usable from C with workarounds (doesn't matter though since D3D12 is fairly niche).

All in all it's a shame though because the newer C features are much more sane and useful than anything in C++14 or C++17.

The name says it all, Visual Studio C++ compiler.

There is no written rule that C++ compilers are obliged to be C compilers as well.

Again, as C++ compiler, they are only required to be as compatible with C code as ANSI C++ requires.

> Because Visual Studio regularly, even on the latest 15.7 preview shits the bed with OutOfMemoryExceptions on our large solution, I'm inclined to rage "why don't they just make it 64 bit?

There's technical reasons alone and there's decisions being made based on other factors.

Iirc the product manager for Visual Studio was against blindly moving VS to 64-bit just because it uses a lot of memory. He considered that fixing the symptome and not the cause. He wanted the team instead to spend their time trying to identify inefficiencies and memory leaks.

Now... If you can say that decision has paid off or not is not for me to call, but I do appreciate the reasoning behind the decision.

If they can make it work well with a 32-bit constraint, that's clearly better than yet another app which uses 8GB memory for no reason.

>Could somebody point me to a technical explanation of why it's sometimes non trivial to just compile your app against x86-64 and call it a day?

https://docs.microsoft.com/en-us/cpp/build/common-visual-cpp...

http://www.informit.com/articles/printerfriendly/2339636

BTW: The 32bit limit is per-process, not for the OS. You can easily have multiple 32 bit processes consuming more than 4GB memory in total. You don't need to port to 64bit to get that extra memory.

macOS makes the 64-bit transition a bit harder than that because a lot of the older user space APIs were purposely not ported to 64-bit. If your app was written against the Carbon APIs, you have to do a fair amount of work to change all the GUI code.

Yep. But developers have no excuse for being late to that show -- Carbon was always explicitly a transitional technology, for a transition which ended over 10 years ago. Warnings have been plastered all over the documentation for ages.

Truth.

There's something to be said for the MSFT approach of your code will run forever. I understand there's tradeoffs, but there's something to be said for it.

Pointing a bunch of whiny users at devs -- some of whom gave code away for free -- is not awesome for the ecosystem. IMO.

Not awesome in the short term, but better in the long term.

Backwards compatibility is a very tangible benefit in the long term.

Imagine if you have a dependency on a third party library, and they only offer 32-bit DLL, or the company is out of business and you are stuck with the 32-bit dynamic library. You can't just "compile against x86_64" in this case.

In many cases for small dlls you can thunk it and provide a compatibility standard library.

Essentially what wine does but cheaper. The ease of doing it is related to number of dependencies.

Assuming that you know that those 3rd party extensions are sticking to the published API and not stuffing around by scratching around in Visual Studio's internal APIs and data structures......

Because Windows programmers have such a great track record of sticking to the published API....

I think the long term plan is to keep adding stuff to VS code, keeping it modular so that it can be anything from a simple editor to the full IDE VS is, and then phase out Visual Studio. Achieves multiple goals in one go, and gives a smooth transition path without the second system syndrome pressure. Everybody wins.

Completely agree, and usually the answer I get is - well plugin developers should just move their stuff OutOfProcess, and deal with it. I went to GDC and talked to JetBrains about ReSharper C/C++ - which is a phenomenal product, yet limited by what else lives with it in the process space. So since last week I had to kill (uninstall/disable) anything else but it, because I depend on it. I briefly had to re-enable th Qt plugin, the P4 one, but had to disable a lot of Microsoft's (whatever I can), Intel's, and had to leave Sony's SN-DBS otherwise compile times are slow as hell... So it's manageable, and there was way to have different environments setup (haven't looked deeply into it)... but really moving to 64-bit would solve a lot of it... even if it would take more memory... (bigger pointers, that's usually the stupid excuse)

I gave up ReSharper in 2015. It's a great tool and I miss it, but our solution has nearly doubled in size over the past few years and even vanilla VS 2017 struggles with it. We have dev ops working on nugetizing some of the bloat and Microsoft premier support is helping us to understand why our solution is so stressful on Visual Studio.

The in process limitation is a real killer for ReSharper.

I recently didn't renew ReSharper for my team. It makes VS crawl, and I honestly don't understand why on earth it cannot utilize some of the other 15 cores available on our machines. It's a fantastic tool, but it doesn't scale with solution size at all.

> Could somebody point me to a technical explanation of why it's sometimes non trivial to just compile your app against x86-64 and call it a day?

"In 32-bit programs, pointers and data types such as integers generally have the same length. This is not necessarily true on 64-bit machines. Mixing data types in programming languages such as C and its descendants such as C++ and Objective-C may thus work on 32-bit implementations but not on 64-bit implementations." [1]

It's non-trivial to ensure every developer writes portable code multiplied by the varying degree of skills for developers.

In a language like C++, assuming the size of an integer is 4-bytes, or that the size of all pointer types are the same, can introduce hard to track down bugs, which may result in 64-bit versions of apps to be unstable.

[1]: https://en.wikipedia.org/wiki/64-bit_computing#64-bit_data_m...

If people have done architecture-specific things they may not compile (or worse, crash inexplicably at runtime).

I tried doing more or less that (compiling a C++ Windows app for 64-bit) a few jobs ago and eventually discovered that way down in a library somewhere a dev (who had long since departed) did some trickery in string processing routines that relied on arguments being lined up on the stack, which is no longer the case with the x86_64 calling convention. That remains the worst piece of code I've ever worked with - it's innocuous to look at, and any developer who understands why that works should also have known better than to do it.

More generally, integer size issues can arise - if `int` remains 32 bits it's no longer enough to capture the difference between two pointers (and obviously should never have been used for that, but often these things happen).

Two basic APIs are not available in 64 bits: Carbon and QuickTime. Steve Jobs promised to port Carbon to 64 bits when macOS X was introduced. So many of us developed for Carbon, and a few years later Carbon was deprecacted, and Apple announced they would never port it to 64 bits. Even software from Apple such as iTunes was built and Carbon, and it took years for Apple to remove the Carbon code out of iTunes...

You’re forgetting the part where this was long time ago. You’ve had at least a dozen years to migrate your code to Cocoa.

https://www.viva64.com/en/a/0004/

> Could somebody point me to a technical explanation of why it's sometimes non trivial to just compile your app against x86-64 and call it a day?

The reason for that is quite similar to the Y2K problem.

https://en.wikipedia.org/wiki/Year_2000_problem

Dependencies can be an issue also. If you're dependent on some 32-bit libraries this can cause you problems moving to 64 bit.

If you assume longs or pointers are 32 bits then it's a problem.

I learned that when pointers were 16 bits wide. You should never assume the size of your words or addresses. Ever.

You never should but people do, and that's when there's a problem.

Reply on news.ycombinator.com