I have to admit that in all of my use of my Yubikey Neo in Chrome I don't recall ever being asked for permission to access the device. Firefox hasn't asked either.
On the other hand, it's basically functioning as another keyboard device and not a special USB device so it shouldn't be that surprising, right? (serious question)