Just to make sure I understand: Is the resizable BAR/MMIO a RAM buffer for PCie packets? Or does it have some deeper integration with DMA or something? I assume it's not like memory mapped peripherals on an AXI bus which is why you need the buffer?
MMIO (Memory Mapped I/O) is essentially memory (Whenever RAM or ROM) from OTHER devices that is directly visible on the CPU Address Space. My understanding is that from the CPU side, MMIO is mostly transparent (Except for the massive increase in latency) because it gets used like if it was interacting with its own workspace with regular instructions like MOV.
What PCIe ReBAR changes is that before, you could only see a 256 MiB window onto the GPU VRAM, so there was an added overhead since the GPU may need to relocate things from inside that window somewhere else on its total VRAM (So yes, it may be interpreted as if what you see from the CPU side is just some kind of exchange buffer). I believe the best way to describe how it operates is comparing it to EMS (Expanded Memory) from the DOS days since it also worked with a similar, if not the same idea. You could only see a portion of the total memory from what was installed on the EMS card (A 128 KiB window located on the upper part of the 1 MiB address space from the 8086 CPU), so you had to switch which Page (Region) of the memory was visible, adding a lot of overhead and most likely requiring an additional buffer in main RAM to move data from one Page to another. However, since I have no knowledge if the GPUs really work like that I can't confirm. I never knew whenever the 256 MiB is "fixed" (You always see the same Region) or if you can decide which section of the VRAM to make visible.
[1] https://en.wikipedia.org/wiki/Advanced_eXtensible_Interface
For historic reasons (e.g. 32 bit address spaces, plus the need to reserve the space for multiple pci peripherals) it has been a narrow, movable aperture.
Resizable BAR lets the size of the aperture be chosen (which is usually chosen to allow all of VRAM to fit in and be directly accessible).
When you insert a GPU into a PCIe slot, the memory mapped regions in memory can't be put in a hard wired location because any arbitrary device can be inserted and it can provide an arbitrary amount of memory (yes the GPU provides its own memory to the CPU). A BAR reserves a memory mapped region in the CPU space that is backed by the PCIe device.
When the BAR is smaller than the memory of the inserted device, the CPU cannot communicate with all of the memory on the inserted device directly anmore. This means if you want to perform a write to a region in the GPU outside a BAR region you have to go through the BAR region anyway. It's not a RAM buffer for PCIe packets.
>I assume it's not like memory mapped peripherals on an AXI bus which is why you need the buffer?
The PCIe controller is an AXI peripheral...