Hacker News Reader

Comment by gpderetta | Hacker News Reader

Some motorola 68000 machines did automatically mask out some of the most significant bits of any pointers. This means that two pointers to the same object can have different representation. That did happen in practice as programmers used the highest bits to stuff metadata and relied on the automatic masking. That made things interesting when the platform wanted to increase the address space. And preventing this sort of backward compatiblity issue is the reason why AMD64 will trap if the unused bits of the address space are not all 1 or 0.

Bottom line, the pointer to integer conversion should be specified by your platform ABI. Read it, then either refuse to support your program on platforms with insane ABIs or have a fallback path for them.

View on HN

You can do this on most microprocessors just by leaving the address lines uninterpreted. On the 68000 the top 8 lines were flapping in the breeze, so you could use those bits for anything and still dereference the same object.

The first MacOS used those bits for some things; notably desk accessories. It took them years to kill that practice so that later virtual memory versions of the Mac would work. IBM made a similar mistake in OS/360 (24 bit address space, 32-bit pointer values . . . that was very bad).

Wasn't this the case with 68000 based designs in general? The CPU only had 24 address lines. I think it was first the 68020 that had a full 32 bit address bus

And you're right, people absolutely did use it.

On the Amiga I believe that Amiga BASIC (developed by Microsoft) was perhaps the most prominent example of Amiga applications that allegedly did this (only source I've found for this is a comment by a former Commodore employee - Dr Peter Kittel), amongst a massive amount of other problems. People did get it running on 68020+ systems, with a combination of patches and disabling fast ram, but it was pretty much forgotten from AmigaOS 2.x onwards anyway.

"Wasn't this the case with 68000 based designs in general?"

Possibly, I wasn't sure so I hedged :).

And yes, I heard that the issue was mostly encountered on the Amiga.

In early Mac OSs, the memory management designs used the top byte for flags as those units didn't have a hardware MMU.

System 7 was the first OS to support 32-bit.

Yes, lots of people used this on Atari ST and Amiga (Tempus editor on Atari comes to mind, which used the highest byte of address registers as additional faux registers).

Same tricks are used even today. E.g., have a look at Facebook's DiscriminatedPtr from the folly library, which uses the top 16bit of a 64bit pointer (otherwise unused) for type tagging.

There is a long tradition of embedding tags in pointers, for all kind of reasons (type tags, pointer/int discriminants, ABA counters, etc.). That's perfectly fine and it is one of the reason intptr_t exists.

The problem is when the architecture lets you get away with explicitly masking out the tag bits (i.e. a tagged pointer is a valid pointer) which makes it hard to evolve the architecture.

If your program targets a 68000, and uses tags in the upper bits of pointer, and that program also hashes pointers, then of course you're going to reconcile these two facts: your hashing function will have to strip out the pointer tags.

Or perhaps not; it depends on whether the tags are immutable attributes. For instance if they indicate type, and type is immutable, then they can perhaps just be mixed into the hash. Under this design, the only way two pointers to the same thing can have a different hash is if you have a use-after-free bug: you're hashing a pointer to something that was de-allocated and re-allocated as a new kind of object. Two objects which occupy the same address at different times don't have to have the same hash, even if it is a pointer hash.

Looked at in another way, those tag bits, if they are immutable, effectively extend the address space. An object with tag 0101 and another one with 0110, all other bits equal, can be considered to be in different spaces (with the constraint that since they collide to the same physical address, they cannot exist simultaneously).

I'm not quite sure what you're replying to. I was not talking about hashing pointers at all. Bare, raw, pointers on the M68000 are 32 bits, but the CPU only has 24 address lines, and the top 8 bits are disregarded. But starting with the 68020, the CPU has 32 address lines and the top 8 bits are not disregarded.

The point is/was that relying on bits that are unused today is risky because it has historically tended to ensure your software breaks when the next CPU generation needs more of the possible address space.

Note specially that a lot of the apps that used the top 8 bits did not use it for type tags that might imply safe usage, but often used it to store unrelated data.

E.g. lets say you had a data structure with a number of pointers and a number of flags. You might very well decide to pack the structure so that the flags overlapped the top 8 bits of the pointers.

> pointer to integer conversion should be specified by your platform ABI

heh, I just published a post on this that I worked on over the weekend: https://nickdesaulniers.github.io/blog/2016/05/30/data-model...

> preventing this sort of backward compatiblity issue is the reason why AMD64 will trap if the unused bits of the address space are not all 1 or 0.

This helps debugging, but it doesn't help future extensions of AMD64. Programs exist that store data in bits 48..63 of a pointer on AMD64, and expect to retrieve a pointer by doing "x << 16 >> 16" on the integer data. Increasing the address space is a pain no matter what.

To make such program work it's sufficient to map it into the lowest quarter-petabye of address space. I can imagine a flag for that in the executable if it ever becomes a problem.

> That did happen in practice as programmers used the highest bits to stuff metadata and relied on the automatic masking

The old Mac OS did this: https://en.wikipedia.org/wiki/Mac_OS_memory_management#32-bi...

I read about an early computer (JOHNNIAC) with the same issue. It's instructions had 12-bit addresses, but the original machine had 10 bits of memory attached. Programmers often interleaved data through the upper two bits of the address, which wasn't a problem until they quadrupled the machine's memory capacity later in its life.

This ( using unused bits of addresses ) was a terrible idea at the time and it's a terrible idea still.

Reply on news.ycombinator.com