I forgot that there is built in support for this model using the MemoryManager<T> class [0]. A memory manager is an abstract class that represents a block of other memory, including possibly unmanaged memory. It implements IDisposable already so you can just plug into this.
The Memory<T> struct can optionally internally point to a MemoryManager instance allowing you to plug your perfered style of allocation and freeing of memory into parts of the framework.
There is a little irony that a MemoryManager<T> is itself a class and therefore managed on the gc-heap, but you can defeat this by using ObjectPool<T> to recycle those instances to keep allocation count steady state and not trigger the GC.
I have used this before (in the toy database i mentioned earlier) to allocate aligned blocks of unmanaged memory.
[0] https://learn.microsoft.com/en-us/dotnet/api/system.buffers....