I've personally never seen it work that way - usually because RDP doesn't pass through direct USB devices, only their abstracted forms (e.g. smartcards don't get passed through, only the "Smart Card" device registered in the OS, and only if you enable that to be passed through in an mstsc session.

There are products like Silverfort (https://www.silverfort.com/) that can handle agentless auth, and might be able to do that kind of MFA inside an RDP session. But, products like this usually require some 3rd device (i.e. your phone) to perform the MFA action, which is kind of not really just a simple WebAuthn logon...

It's kind of a pain, but possible. You have to redirect it as a generic USB device. I wrote a guide here:

https://queensidecastle.com/guides/use-a-yubikey-remotely-ov...