I've found that when I'm cloning submoduled private repos via YK-backed SSH key, I'll need to touch multiple times but there's not always text in the terminal notifying me to do so. Easy to miss the small flashing green light.
So you need to approve the usage of that secret by touching the yubikey.