ReBarUEFI: Resizable BAR for almost any UEFI system

(Incidentally this is also one of my gripes with the Intel Macs, wanting to put Linux on one but having zero configurability pre-OS is a huge pain. Meanwhile a standard PC BIOS with the equivalent hardware has plenty of options. Maybe I'll get around to (vibe)porting a regular BIOS to mine...)

In the old days, they were a fixed size. If your GPU has 16 GB of ram and you want to access all of it via memory addresses, you'd need a 16 GB BAR ... but lots of (older) systems wouldn't be able to map in a large BAR because of a lack of address lines (or wahtever), so GPUs had stayed with 256MB for VRAM access because it was compatible. With a smaller BAR than the VRAM, you have to use some sort of windowing / paging setup. Resizable BAR lets the BAR start small so older systems will work, but grow larger with capable systems.

Much better than having a jumper to set the BAR to big or small, and you can skip VRAM window management.

If a program runs slower when you give it more RAM, the problem isn't giving more RAM.

For historic reasons (e.g. 32 bit address spaces, plus the need to reserve the space for multiple pci peripherals) it has been a narrow, movable aperture.

Resizable BAR lets the size of the aperture be chosen (which is usually chosen to allow all of VRAM to fit in and be directly accessible).

MMIO (Memory Mapped I/O) is essentially memory (Whenever RAM or ROM) from OTHER devices that is directly visible on the CPU Address Space. My understanding is that from the CPU side, MMIO is mostly transparent (Except for the massive increase in latency) because it gets used like if it was interacting with its own workspace with regular instructions like MOV.

What PCIe ReBAR changes is that before, you could only see a 256 MiB window onto the GPU VRAM, so there was an added overhead since the GPU may need to relocate things from inside that window somewhere else on its total VRAM (So yes, it may be interpreted as if what you see from the CPU side is just some kind of exchange buffer). I believe the best way to describe how it operates is comparing it to EMS (Expanded Memory) from the DOS days since it also worked with a similar, if not the same idea. You could only see a portion of the total memory from what was installed on the EMS card (A 128 KiB window located on the upper part of the 1 MiB address space from the 8086 CPU), so you had to switch which Page (Region) of the memory was visible, adding a lot of overhead and most likely requiring an additional buffer in main RAM to move data from one Page to another. However, since I have no knowledge if the GPUs really work like that I can't confirm. I never knew whenever the 256 MiB is "fixed" (You always see the same Region) or if you can decide which section of the VRAM to make visible.

When you insert a GPU into a PCIe slot, the memory mapped regions in memory can't be put in a hard wired location because any arbitrary device can be inserted and it can provide an arbitrary amount of memory (yes the GPU provides its own memory to the CPU). A BAR reserves a memory mapped region in the CPU space that is backed by the PCIe device.

When the BAR is smaller than the memory of the inserted device, the CPU cannot communicate with all of the memory on the inserted device directly anmore. This means if you want to perform a write to a region in the GPU outside a BAR region you have to go through the BAR region anyway. It's not a RAM buffer for PCIe packets.

>I assume it's not like memory mapped peripherals on an AXI bus which is why you need the buffer?

The PCIe controller is an AXI peripheral...

And if my experience from embedded development is in any way transferable, they're probably fairly spread out and probably takes a fairly big chunk of address space too.