Ahh, but I have two interfering security cards in my wallet, one in each flap! With your system I'd need to actually take the cards out of the "wallet" before I used them.

Also, coins. Also, the inside of your wallet is less abrasive than the inside of your pocket, that could contain keys or loose change.