Comment by zir_blazer | Hacker News Reader

Comment by zir_blazer | Hacker News Reader

Parent Full thread

Since about 5 or 6 years ago, PCIe Video Cards advertise a capability known as Resizeable BAR. PCI Devices like GPUs require some memory to use for PCI MMIO, which is directly visible on the CPU Address Space. As a side note, this Address Space is shared with RAM, and anyone that was around when having 4 GiB RAM with a 32 Bit OS was common (Earlier than 2010 or so) knows that you only saw about 3.25 GiB RAM or so because of sharing the Address Space with PCI MMIO, and those fortunate enough to have used SLI usually saw even less than that, like 2.87 GiB RAM.

GPUs has been using a 256 MiB PCI MMIO window regardless of how much VRAM they actually have since... nearly forever? At least since PCIe is a thing, since I recall than AGP Aperture Size was seteable in era accurate BIOSes. PCIe 3.0 specification introduced a feature known as Resizeable BAR, where the PCI Device can tell a compatible Firmware how much MMIO it actually wants. GPUs uses that to tell a ReBAR capable UEFI Firmware that it wants more MMIO (Usually as big as the GPU VRAM), or uses legacy 256 MiB otherwise.

GPUs have been using a 256MB window since they started coming with 256MB of VRAM, as it would otherwise be a pointless waste of address space to have a window larger than the actual amount of VRAM. Previously, they would have a window exactly equal to the size of VRAM (likely rounded up to the next power of 2.)

There would also be a block of memory mapped hardware registers also needing address space that you have to poke to make the GPU actually do GPU things, instead of just being an expensive way to add extra memory to a system, no?

And if my experience from embedded development is in any way transferable, they're probably fairly spread out and probably takes a fairly big chunk of address space too.

Yes, typically the memory mapped memory is in one (64-bit) bar, memory mapped registers in another (64-bit) bar, plus for compatability with vga, probably a 32-bit memory bar and an i/o bar. 64-bit bars take up two bar slots, so that fills all six slots in the PCI config.

Just to make sure I understand: Is the resizable BAR/MMIO a RAM buffer for PCie packets? Or does it have some deeper integration with DMA or something? I assume it's not like memory mapped peripherals on an AXI bus which is why you need the buffer?

It's just memory mapping the peripheral.

For historic reasons (e.g. 32 bit address spaces, plus the need to reserve the space for multiple pci peripherals) it has been a narrow, movable aperture.

Resizable BAR lets the size of the aperture be chosen (which is usually chosen to allow all of VRAM to fit in and be directly accessible).

No idea on the AXI Bus you're talking about, so can't make comparisons.

MMIO (Memory Mapped I/O) is essentially memory (Whenever RAM or ROM) from OTHER devices that is directly visible on the CPU Address Space. My understanding is that from the CPU side, MMIO is mostly transparent (Except for the massive increase in latency) because it gets used like if it was interacting with its own workspace with regular instructions like MOV.

What PCIe ReBAR changes is that before, you could only see a 256 MiB window onto the GPU VRAM, so there was an added overhead since the GPU may need to relocate things from inside that window somewhere else on its total VRAM (So yes, it may be interpreted as if what you see from the CPU side is just some kind of exchange buffer). I believe the best way to describe how it operates is comparing it to EMS (Expanded Memory) from the DOS days since it also worked with a similar, if not the same idea. You could only see a portion of the total memory from what was installed on the EMS card (A 128 KiB window located on the upper part of the 1 MiB address space from the 8086 CPU), so you had to switch which Page (Region) of the memory was visible, adding a lot of overhead and most likely requiring an additional buffer in main RAM to move data from one Page to another. However, since I have no knowledge if the GPUs really work like that I can't confirm. I never knew whenever the 256 MiB is "fixed" (You always see the same Region) or if you can decide which section of the VRAM to make visible.

Just to close the loop, AXI bus[1] refers to a very common implementation, in the gateware/verilog, of a MIMO.

[1] https://en.wikipedia.org/wiki/Advanced_eXtensible_Interface

It’s also similar to how mmap() on 32-bit systems maps files larger than 2GB.

There is no buffer at all. BAR is "base address register" (hint, there is no "the BAR"). Each PCIe device / function can have up to 6 32-bit BARs or 3 64 bit BARs.

When you insert a GPU into a PCIe slot, the memory mapped regions in memory can't be put in a hard wired location because any arbitrary device can be inserted and it can provide an arbitrary amount of memory (yes the GPU provides its own memory to the CPU). A BAR reserves a memory mapped region in the CPU space that is backed by the PCIe device.

When the BAR is smaller than the memory of the inserted device, the CPU cannot communicate with all of the memory on the inserted device directly anmore. This means if you want to perform a write to a region in the GPU outside a BAR region you have to go through the BAR region anyway. It's not a RAM buffer for PCIe packets.

>I assume it's not like memory mapped peripherals on an AXI bus which is why you need the buffer?

The PCIe controller is an AXI peripheral...

Reply on news.ycombinator.com