Here's Yubico's documentation on Filevault integration: https://www.yubico.com/support/knowledge-base/categories/art...
It seems an attacker with physical access still requires your password to unlock the disk. At that point, they'd need the Yubikey to login (assuming they haven't already decrypted the disk and taken your data).
Someone on Reddit suggested saving a static password to the Yubikey and then entering that at boot time to get around this: https://www.reddit.com/r/AskNetsec/comments/3dpa2q/how_do_yo...?