Why is my x64 process getting heap address above 4GB on Windows 8?
blogs.msdn.com
blogs.msdn.com
I'm a Unix person, so the notion of long being less than word-length seems pretty silly to me, but I understand why they did it, sort of.
It's silly for Windows to do something a certain way just because that's not how Unix does it?
What's silly is assuming that long and unsigned long are more than 32 bits wide, since neither C nor C++ has ever guaranteed anything more.
What are the advantages of having data types that behave differently on different platforms, like C does? It seems to make code less portable and I'm having difficulty seeing what the benefits are.
I can appreciate that, on an 8-bit processor, you'd want access to an 8-bit data type for speed - but wouldn't that be better accomplished by explicitly choosing an 8 bit data type?
The practical decision to leave the sizes undefined was made so that nobody was burdened unfairly. It's choices like this - not biasing stakeholders toward forgoing a standard for clear business reasons - that makes the adoption of standards likely. Subsequent C standards have adopted the same undefined behavior for the same reasons [there are still 8 and 16 bit systems developed with C] and to facilitate backward compatibility of the new standards.
There are some other types (i.e int32_t, int64_t, etc) that you could use if you want to be sure that your variable would have the same size regardless of the architecture.
Than what? Than a freshly designed C-like language? Yes. But ISO C wasn't an effort to invent a new language. It was an effort to standardize existing practice, and provide a platform for improving C. One of the core goals of ISO C is to maintain every last bit of backward compatibility possible, which is why integer widths are not precisely specified: had they been, multiple vendors' implementations and the code that was written for them would have broken.
But to your main point, I agree: explicitly sized integer types are almost always what you should use. For example, don't store a pointer in an unsigned long (and certainly not a signed long); store it in a uintptr_t, or hell, just store it as a pointer. In fact, the fundamental types like int, long, etc. are effectively just shorthand for int_fast16_t, int_fast32_t, etc., but without the mnemonic value.
In your link they have examples where size_t and ptrdiff_t are 16 bits. Obviously this is absurd on today's machines. But you can imagine a machine where array subscript is most convenient at 16 bits and total addressable space is larger (16 bit x86 comes to mind). By the same token I don't think it would be too weird to have a 32 bit size_t on a 64 bit architecture. It would just make memcpy et al more annoying when your allocations are large.
Anyway the quotation from your link:
> The types used for size_t and ptrdiff_t should not have an integer conversion rank greater than that of signed long int unless the implementation supports objects large enough to make this necessary.
I guess if longs are 32bit and you can have a 4g allocation this pretty well follows the spirit here.
LuaJIT does it for example. Interestingly enough some programmers do this without even understanding how memory allocation works. For example the author of the newish wren language: https://github.com/munificent/wren
Somehow most programmers - including almost everyone who frequents reddit.com/r/programming - believe that "virtual address space" means that the addresses returned by malloc() will restart from (somewhere close to) 0x0 for every new process i.e. that it is safe to truncate pointers if you never intent to allocate more than ~2GB per process anyway.
Maybe to get more compact data structures? But in that case they shouldn't rely on that and try detecting that behavior at runtime.
E.g. the hotspot jvm tries to map memory in the low end of address space to compress pointers but falls back to full pointers when that fails.
The correct variable type to store a pointer would be intptr_t/uintptr_t for portable software (and DWORD_PTR for Windows-only software), but the "intptr" types didn't exist until the 1999 C standard (and, as far as I could find, the default Windows compiler didn't add the corresponding header file until around 2010). Software older than that would often assume that a "long" (or a DWORD, which IIRC was just another name for "long") was big enough to store a pointer, or perhaps tried to use the C99 types and used "long" if the header wasn't found.
Is it? I mean, if you're lucky it won't cause a compiler warning on x86, but the sane thing to do is to use the types which the API docs specify. Which would usually be things like LPVOID or HANDLE.
That's basically how I managed to port a service and several drivers to 64-bit Windows many years ago with almost no changes.
I know as I also worked on the huge codebase where the oldest pieces of the code were written around 1995.
Even if this is why, there is never a valid excuse for blindly truncating 64-bit pointer to 32-bits. It's just like saying "I'm gonna just go ahead and truncate all bank account balances to 16-bit integers to save memory in my data structures." It may be the reason, but that reason is still incredibly stupid.
Sometimes there are stupid reasons, but sometimes even reasons which are incredibly stupid from a technical standpoint make sense in a broader context.
For a short-term fix, for example, I can very well imagine to prefer this "trick" forcing sub-4G allocations if the other alternative would be to change 1000 places in undocumented legacy code doing crazy casts... And if it's for a product (or product component) on life-support only needed for a forseeable future, to me it makes perfect sense to ask the initial "crazy" question, even if it makes me cringe...
But that's not what this is doing - it's taking an address allocated at an arbitrary place and then pretends that it was allocated sub-4G, whether or not it actually was. If it happens that it wasn't, this will likely cause memory corruption or an access violation.
Even if your product is nearing end-of-life, I'd still consider such a move a big "f* you" to your customers as you'd basically be tolerating that your product may crash without warning at any time.
From what I understood, what makes this idea so bad isn't truncating to 32bit - it's truncating to 32 bit while setting the LARGEADDRESSAWARE build flag - which basically tells the OS "I'm fine with all kinds of 64bit addresses, bring them on!"
The correct fix is outlined at the end of the article - and it does not involve combing through undocumented legacy code:
> If there is some fundamental reason that they have to truncate pointers to 32-bit values, then they should build without /LARGEADDRESSAWARE so that the process will be given an address space of only 2GB, and then they can truncate their pointers all they want.
#if (ULONG_MAX < UINTPTR_MAX) && !defined (SUPPRESS_TRUNCATION_ERROR)
#error Truncating pointers to long is a bad idea here
#endif
static_assert(sizeof(DWORD) == sizeof(void*), "assume 32-bit pointers");