Many HSMs also add advanced authentication capabilities, such as M-of-N access control and/or hardware authenticators (e.g. you need 3 of 5 smart cards to use the HSM). The other key feature usually found in HSMs but not smart cards is backup/cloning without exporting the key (in PKCS#11 terms). This means that the key can be moved between HSMs with all the protections in place. I've yet to see a smart card that does this.