The downside is that there's now a single point of failure, albeit with more factors. If you get someone's CPR number, their NemID password, and their current NemID card with some indication on it of which the next unused code is (most people mark off the used codes), you can log into everything: all Danish banks, the tax authority, the municipal authorities, your library account, etc., etc.
They do try to minimize it by writing strongly worded warnings everywhere not to store your NemID password in your wallet. A typical wallet contains a Danish health card with CPR number, and the NemID code card, so it's fairly important that the NemID password not also be there.