I would assume all linux GPU drivers would need to be adapted at least a little to support the larger page size (most linux AARCH64 kernel level code is writing assuming 4kb pages).

Yes, but NOT for GPUs. Apple Silicon does not support non-Device mappings over Thunderbolt, so eGPUs will never work.