Hacker News Reader

Comment by fulafel | Hacker News Reader

It's surprising that we have stuck with 4 kB pages on x86 since the 386, even though computers have ~10 000x as much memory now (4 MB -> 32GB).

View on HN

I wrote a thesis on this in 2008 (Transparent large-page support for Itanium Linux https://ts.data61.csiro.au/publications/theses_public/08/Wie...) and Matthew Wilcox was already involved in the area then. I admire his persistence, and have certainly have not kept up in the state of the art. Itanium had probably the greatest ability to select page size, probably more than any other architecture (?). On x86-64 you really only have 2mb or 4k to work with in a general purpose situation. It was difficult to show the benefits of managing all the different page sizes and, as this notes, re/writing everything to be aware of page sizes effectively. Those who had really big workloads that benefited from huge pinned mappings didn't really care that much either. It made the work hard to find traction at the time.

There is a series of page sizes that steps in increments of 5 bits per width. 8k/256k/8M therefore have regular join points in the address space, the first of which is at 8G.

Does superpage management get easier when each superpage is composed of only 32x of the next size down? When I first stumbled on this idea, it seemed like it would have many more opportunities for forming intermediate-sized superpages.

Linux still has a lot of assumptions baked into the page size. Power9 and some aarch64 systems have 16kB pages, but occasionally you run into some corner cases - for example, you can't mount a btrfs partition created on a x86 machine on a power9 one because the btrfs page size must be >= the mmu page size.

64K pages, actually. Also, POWER9 and aarch64 are perfectly capable of running with 4K pages, but not everything does. My desktop POWER9 with Fedora is a 64K page system, but Void PPC runs the same hardware with a 4K page.

That's the fault of btrfs (ext4 or xfs handle these cases just fine), and even btrfs is getting support for it in the latest releases

While you may be technically correct, I guess from a practical point of view, OP is correct in that page-size still matters if you trying to run Linux on "unconventional" platforms.

RHEL used 64k page size on aarch64 for a while. I believe we have switched back to 4k. It caused some problems, from memory:

* Blow-ups in various kernel data structures. There was some virtio code which was allocating N pages per driver queue.

* Problems with GPUs, either the driver or the firmware assumed 4k pages. (Edit: This actually affected Power, not ARM, but the issue is caused by page size: https://lists.fedoraproject.org/archives/list/devel@lists.fe...)

* Filesystems make assumptions about page size versus block size.

* Processes generally take more RAM, with RAM wasted because of internal fragmentation.

I bet this would also screw masses of user code making assumptions about optimal size, or simply designed/optimized for a smaller size. LMDB was the first thing that came to mind

> RHEL used 64k page size on aarch64 for a while. I believe we have switched back to 4k.

Last I checked, at least CentOS 8 (which should be the same as RHEL 8) is still using 64k pages (search at https://git.centos.org/rpms/kernel/blob/c8/f/SOURCES/kernel-... for CONFIG_ARM64_64K_PAGES=y).

And AFAIK, to access the maximum amount of physical memory in AARCH64 (52-bit physical addresses, instead of 48 bit physical addresses), you must use 64k pages. Since RHEL is normally used on servers, it makes sense to want to be able to access huge amounts of physical memory; that's probably the true reason (or even the sole reason) RHEL uses 64k pages on AARCH64.

AMD64 supports only 48-bit addresses, and is still the most popular server ISA.

  You have: 2^48 byte
  You want: tebibyte
          2^48 byte = 256 tebibyte
          2^48 byte = (1 / 0.00390625) tebibyte

how many servers do you have with more than 256 TiB of RAM?

56-bit support was specced and had software support already in ~2016. 48 bits is not a hard architecture limitation. see eg https://www.kernel.org/doc/html/latest/x86/x86_64/5level-pag...

It's not only ram that goes in that 256 TB, memory mapped i/o uses it too. You probably don't have TB of video card RAM, but maybe some flash devices offer the whole thing memory mapped?

Good points. Re the last one: you of course also save some by having fewer pages and their metadata. Wonder what the space-optimal page size would be taking these opposing factors into account.

Indeed, the M1/A14 on mobile has larger pages which lets them have more effective TLB coverage with a smaller cache. In some applications this can boost performance by double digit % (which you can simulate by enabling large pages on x86).

A lot of peripherals have 4kB address space. So it becomes complicated if you change to 16kB as you'll be mapping in more than you bargained for.

You can also do large pages, 2MB or 1GB or whatever the obscenely large page size is for 5-level paging on latest systems.

2MB vs 4kB isn't quite the same ratio as 4MB -> 32GB, but it's still a lot less pages to cache in the TLB, and it's not too big to manage when you need to copy on write or swap out (or compress with zram) and whatever else needs to be done at the page level.

Reply on news.ycombinator.com