> Unfortunately, Shader programs are currently restricted to the Logical model, which disallows all of this.

That is not entirely true, you can use phyiscal pointers with the "Buffer device address" feature. (https://docs.vulkan.org/samples/latest/samples/extensions/bu...) It was an extension, but now part of Vulkan. It is widely available on most GPUS.

This only works in buffers though. Not for images or local arrays.