Yubico themselve say that the 4096 bit key only adds 16% more security. So the Neo is practically as safe as the 4.

https://www.yubico.com/2015/02/big-debate-2048-4096-yubicos-...

> While it is true that a longer key provides better security, we have shown that by doubling the length of the key from 2048 to 4096, the increase in bits of security is only 18, a mere 16%. Moreover, besides requiring more storage, longer keys also translate into increased CPU usage and higher power consumption.