Are you still using LUKS for these? How are you invoking decryption with the pass phrase protected key?
I do not know if LUKS could be used for this, I have not examined it. IIRC, LUKS stores the actual decryption key in the encrypted disk (protected by a passphrase) or in the TPM, like most commercial products for disk encryption, which are methods that I do not approve.
Your approach sounds pretty cool, by the way. I've thought about such an approach in the past, and I've used it for some auxiliary computers under my control, but not for my daily driver. I have a Framework laptop and I could indeed use this approach in quite a stylish way, though: https://frame.work/de/en/products/storage-expansion-card?v=F...
The last time when I have looked at LUKS was some years ago, when this feature did not exist yet.
In my opinion, this is the only right way to do SSD/HDD encryption. The detached header allows plausible deniability and it avoids downgrading the strength of the encryption key to the strength of the passphrase.
By using the detached header option, LUKS could be used exactly like in my custom setup.