Sure. That's why I wish I could do LUKS authentication via TPM, which can do hardware rate-limiting. Is this kind of setup possible these days?
You can also use a private gpg key and an gpg-encrypted keyfile. Yubikey can store gpg keys (with attestation if you want), and you can enable both yubikey access and gpg in initramfs. That's good for servers if you pair it with initramfs sshd.
TPM is kind of broken, as I understand it, so I didn't even bother looking into it. There's really no reason I can't carry my keys on my physical keychain, so I don't see a need for TPM. I can also put a USB stick on my keychain and use that to boot.
That way even if an attacker has physical access to the hardware, it would be require some really advanced firmware/microcode hackery to compromise the machine.
TPM dies if your motherboard dies, so you'd need a backup key anyway.
There's also no way for you to independently audit the TPM device, so you don't know how secure it really is. Personally, I don't trust Intel to keep the TPM secure, as they've shown they can't (see the above link).
The keys are in the chip if you use them for FDE, so you're expanding your trust that someone with access to the TPM can never dump the keys. This could be a problem if your machine is stolen.
By going the yubikey route, you can move the trust to a device that's physically on your person. That way, if the machine is stolen, there's absolutely no way to recover the keys, and there never will be even if more intel vulnerabilities are found.
You could use a yubikey for the FDE keys, and only use TPM for trusting the boot device. But if you're already carrying a yubikey to decrypt your device, you might as well boot off USB too.