What does this do? I perused the README but I'm still clueless on what the purpose and use-case is for ReBAR.
(Posting this under the assumption others will also appreciate a bit of quick context.)
(Posting this under the assumption others will also appreciate a bit of quick context.)
GPUs has been using a 256 MiB PCI MMIO window regardless of how much VRAM they actually have since... nearly forever? At least since PCIe is a thing, since I recall than AGP Aperture Size was seteable in era accurate BIOSes. PCIe 3.0 specification introduced a feature known as Resizeable BAR, where the PCI Device can tell a compatible Firmware how much MMIO it actually wants. GPUs uses that to tell a ReBAR capable UEFI Firmware that it wants more MMIO (Usually as big as the GPU VRAM), or uses legacy 256 MiB otherwise.
For historic reasons (e.g. 32 bit address spaces, plus the need to reserve the space for multiple pci peripherals) it has been a narrow, movable aperture.
Resizable BAR lets the size of the aperture be chosen (which is usually chosen to allow all of VRAM to fit in and be directly accessible).
MMIO (Memory Mapped I/O) is essentially memory (Whenever RAM or ROM) from OTHER devices that is directly visible on the CPU Address Space. My understanding is that from the CPU side, MMIO is mostly transparent (Except for the massive increase in latency) because it gets used like if it was interacting with its own workspace with regular instructions like MOV.
What PCIe ReBAR changes is that before, you could only see a 256 MiB window onto the GPU VRAM, so there was an added overhead since the GPU may need to relocate things from inside that window somewhere else on its total VRAM (So yes, it may be interpreted as if what you see from the CPU side is just some kind of exchange buffer). I believe the best way to describe how it operates is comparing it to EMS (Expanded Memory) from the DOS days since it also worked with a similar, if not the same idea. You could only see a portion of the total memory from what was installed on the EMS card (A 128 KiB window located on the upper part of the 1 MiB address space from the 8086 CPU), so you had to switch which Page (Region) of the memory was visible, adding a lot of overhead and most likely requiring an additional buffer in main RAM to move data from one Page to another. However, since I have no knowledge if the GPUs really work like that I can't confirm. I never knew whenever the 256 MiB is "fixed" (You always see the same Region) or if you can decide which section of the VRAM to make visible.
[1] https://en.wikipedia.org/wiki/Advanced_eXtensible_Interface
When you insert a GPU into a PCIe slot, the memory mapped regions in memory can't be put in a hard wired location because any arbitrary device can be inserted and it can provide an arbitrary amount of memory (yes the GPU provides its own memory to the CPU). A BAR reserves a memory mapped region in the CPU space that is backed by the PCIe device.
When the BAR is smaller than the memory of the inserted device, the CPU cannot communicate with all of the memory on the inserted device directly anmore. This means if you want to perform a write to a region in the GPU outside a BAR region you have to go through the BAR region anyway. It's not a RAM buffer for PCIe packets.
>I assume it's not like memory mapped peripherals on an AXI bus which is why you need the buffer?
The PCIe controller is an AXI peripheral...
And if my experience from embedded development is in any way transferable, they're probably fairly spread out and probably takes a fairly big chunk of address space too.
Most PCI devices expose some memory and/or I/O ports to the CPU. That memory (or I/O ports) is mapped to somewhere in the address space visible to the CPU. Besides the memory and I/O ports, all PCI devices also expose a separate set of configuration registers; among these registers, there are the Base Address Registers (BARs), which configure where the memory or I/O ports is mapped.
Here's an example output from "lspci -vv" for a GPU:
Region 0: Memory at 7c00000000 (64-bit, prefetchable) [size=8G]
Region 2: Memory at 7e00000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at f000 [size=256]
Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at fcb00000 [disabled] [size=128K]
Capabilities: [200 v1] Physical Resizable BAR
BAR 0: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
BAR 2: current size: 256MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB
Why is this useful? This particular GPU has 8GB of VRAM; it's quite obvious that region 0 is a direct view into that VRAM. When using the maximum BAR size, the CPU can directly read and write anywhere into the VRAM; when using a smaller BAR, the CPU can only see a small window into the VRAM, and has to use less direct methods to access it.
(As an aside: go right now and do a "sudo lspci -vv" on your computer, if you see a Resizable BAR capability which isn't using the maximum size, you can probably gain a bit more speed for free by going into the BIOS and enabling "Resizable BAR" and/or "Above 4G decoding". If you can't find these options, well, AFAIU that's what this project is all about..)
IIRC Linux doesn't need resizable bar enabled in the BIOS since the kernel will resize the bar if supported by the GPU, Windows however relies on the UEFI doing it which is where it being enabled in the BIOS is needed.
[1] This is nothing new in the tech industry. Intel rebrands DVFS as SpeedStep, IOMMU as VT-d, AMD rebrands the NX bit as Enhanced Virus Protection, etc.
It's not like ReBar is a single toggle "Make Things Faster", but a different option in how it can map gpu memory to the cpu. The driver still needs to use it - and decide where it's use vs the "staging buffer" approach would actually be be benefitial.
Isn't it just being able to shift the window of GPU memory visible to the CPU?
Wasn't helped by my experience reacting very strongly to story of early Arc drivers being problematic because the devs were too used to GPU being just a ring-bus away ;-)
If a program runs slower when you give it more RAM, the problem isn't giving more RAM.
Literally the second sentence in the repo:
> This provides performance benefits and is even required for Intel Arc GPUs to function optimally.
What is a BAR let alone a resizable one? Readme just jumps in, which is fine, but I’m not sure why this is on HN or why I should care.
Also, ReBAR can get complicated as you can see from other responses above. The articles on the web are also not really good as they tend to replicate marketing material and make vague statements about performance, but fail to explain how specifically performance is improved.
Same as his: to voice my frustration about something on the internet.
Wrong.
> I'm still clueless on what the purpose and use-case is for ReBAR
Are you seriously debating this?
Or to put it differently, I would take the sentence "He asked for a purpose and a use-case not an explanation." and replace the word "not" with a comma.
I agree with "He asked for a purpose and a use-case". (This is what your quote supports.)
I disagree with "not an explanation".
When asking for a purpose and use case, they were asking for an explanation.
In the old days, they were a fixed size. If your GPU has 16 GB of ram and you want to access all of it via memory addresses, you'd need a 16 GB BAR ... but lots of (older) systems wouldn't be able to map in a large BAR because of a lack of address lines (or wahtever), so GPUs had stayed with 256MB for VRAM access because it was compatible. With a smaller BAR than the VRAM, you have to use some sort of windowing / paging setup. Resizable BAR lets the BAR start small so older systems will work, but grow larger with capable systems.
Much better than having a jumper to set the BAR to big or small, and you can skip VRAM window management.