TPM dies if your motherboard dies, so you'd need a backup key anyway.
There's also no way for you to independently audit the TPM device, so you don't know how secure it really is. Personally, I don't trust Intel to keep the TPM secure, as they've shown they can't (see the above link).
The keys are in the chip if you use them for FDE, so you're expanding your trust that someone with access to the TPM can never dump the keys. This could be a problem if your machine is stolen.
By going the yubikey route, you can move the trust to a device that's physically on your person. That way, if the machine is stolen, there's absolutely no way to recover the keys, and there never will be even if more intel vulnerabilities are found.
You could use a yubikey for the FDE keys, and only use TPM for trusting the boot device. But if you're already carrying a yubikey to decrypt your device, you might as well boot off USB too.