The trouble with 64-bit DMA in Linux | Hacker News Reader

The trouble with 64-bit DMA in Linux

lwn.net

47 pointschmaynard19 comments

See https://lwn.net/SubscriberLink/904210/46bbced99963b245/

Thanks.

@dang - Can you edit the top link to this subscriber link so folks with out an account at lwn don't have to search the comments please?

Also for everyone else - just a reminder, LWN is a great resource and is funded through subscriptions. Personally, I've found the information on this site to be extremely helpful to understanding how the kernel works - to the point that multiple products I've built were only really doable because of LWN articles explaining the subsystem I'm working with (namespaces, io_uring, etc are all are the topic of fantastic articles there). A personal subscription is extremely affordable and very much worth it. They have corporate site license subscriptions too.

The string “@dang” has no special effects at HN. If you need a response, you’ll need to use the Contact link in the footer to email the site mods with link change requests.

I don't think it's really appropriate to help folks reach subscriber only content without a subscription, especially on such a niche, subscriber supported site.

LWN Editor Jonathan Corbet regularly drops in on HN threads like this to reassure people that posting subscriber links to HN is okay.

It's an officially supported feature; if lwn doesn't mind, why should we?

This is a tangent, but I was reminded of early Macintosh days, where the 68000 processor in the original Mac hardware only paid attention to the lower 24 bits of addresses, and only the lower 22 were actually connected to memory. The heap manager used the upper bits of pointers as a convenient place to store flags, and didn’t bother zeroing them on dereference because they were ignored anyway.

The release of the 68020 and Macs with more RAM resulted in the need for a whole transition to “32-bit clean” system and apps that took a while. I also seem to recall something about early versions of Microsoft Mac apps being limited to the lower megabyte of RAM for a similar reason.

As we get to 64 bits and more the temptation to do “useful” things with the unused bits will grow …

There's even official hardware support for doing that.

ARM calls it Top Byte Ignore, it's supported on all 64-bit ARM CPUs, and it's been enabled on Linux for a long time (see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... and https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...).

Intel calls it Linear Address Masking and AMD calls it Upper Address Ignore, however these will be available only on very recent hardware, and the Linux patches for them are still under development (see https://lwn.net/Articles/902094/ and https://lwn.net/Articles/888914/).

People are already doing all sorts of useful things with it! Only the 48 bits of the address space are actually being used by virtual memory (no current 64-bit OS is anticipating people to use more than 256TB of address space), so you can use the remaining 16 bits for... lots of things. (Note that there's also the lower 3 bits that aren't being used since memory addresses are aligned to a multiple of 8.)

The primary example is Lua (before 5.4, and also LuaJIT): they use NaN-boxing to make their object take only 8 bytes. Although a double (Lua's only number type) already takes up 8 bytes of memory, there are some bits residing on the high 16-bits that are free to use when the number is NaN. With that you can store the type tag in there, which enables you to use the remaining bits for things other than double (like for example, storing memory addresses...) The Crafting Interpreters book has a much better explanation of this: https://craftinginterpreters.com/optimization.html#nan-boxin...

You can look up at https://en.wikipedia.org/wiki/Tagged_pointer to see some more examples. Apple seems to use this a lot for their Objective-C runtime on iOS where the leftover bits are used for things like refcounts or various boolean flags for their objects. Personally I had experience exploiting this in my game engine: for storing the generational index for a memory arena (to catch any use-after-free errors).

Nan-boxing is also used by Firefox, especially in JS implementation.

Even more extreme: https://en.wikipedia.org/wiki/A20_line

Andy Hertzfeld called this his "worst blunder" with the original Macintosh. He talks about this, and a few other things he wishes they did differently in retrospect, in his folkore.org post titled "Mea Cupla".[1]

[1] https://www.folklore.org/StoryView.py?project=Macintosh&stor...

"One of the core rules of kernel development is that good things rarely result from breaking Linus Torvalds's machine..." sums up this post beautifully.

I was personally surprised that maintainers would "be brave" at all with a codebase as widely used as the Linux kernel - just as much as to hear Linus suggest that this sort of thing happens time and time again.

The implication of “we can probably find out where it breaks and the distros will catch it” is kinda fly-by-the-pants. Honestly surprised by it.

It is kind of fly-by-the-pants... But when you know a change could break at runtime on some hardware, and there's no testing organization with a full or even representative sample of supported hardware, and there's no real way to try it and then fallback at runtime, the options are push it out through the (emergent) testing process and see what happens or push it out disabled where almost nobody will use it.

If only a few issues were noticed, those device dricers could likely be fixed with small code changes and most people would be happy. When Linus's machine breaks, there's a heuristic that many machines will break, so that's a recipie for a release that will cause lots of upset.

For super large systems, you do something like DPDK or XDP instead for large memory and super high throughput, the kernel's network stack is not designed for that anyways.

HPC has been moving huge data cross PCIe running Linux for a long while, there must be multiple ways to overcome IOVA-4GB-allocation limit.

Last, they can be dealt with at hardware level, let the (smart) hardware(e.g. an offload engine, or accelerator) to manage the memory on its own for DMAs.

Reply on news.ycombinator.com