LUKS supports detached headers, maybe this would be useful for your setup? https://wiki.archlinux.org/title/Dm-crypt/Specialties#Encryp...

Your approach sounds pretty cool, by the way. I've thought about such an approach in the past, and I've used it for some auxiliary computers under my control, but not for my daily driver. I have a Framework laptop and I could indeed use this approach in quite a stylish way, though: https://frame.work/de/en/products/storage-expansion-card?v=F...