Right next to the hashed passwords. The point of salting isn't to add an additional level of secrecy, it's just to prevent the reuse of hashing work for attacking other users.
If both hashes and salts were isolated, I suppose it would be much more secure, although maybe too slow.
With salt: Generate table of hashes vs. Plaintext by appending salt to each possible plaintext of given length. For first entry use first salt to generate this table. Table is now useless for second salt if it is different. Time to attack n hashes scales linearly.
What the article really points out is what people have been saying for a while now, that if your passwords are exposed, salting really doesn't add more than a roadbump to a cracker if you are computing your hashes with fast message-digest algorithms like MD5 or SHA.
If you've read the article, you have a pretty good idea of how it works without salting. You have a list of N password hashes, and you come up with candidate passwords. You run each candidate password through the hash function, and compare the resulting hash to your list. If it matches any of them, you've got the plaintext for those user accounts. The key thing here is that you only have to hash each candidate password once, and you can compare that hash to as many accounts as you'd like.
Now, let's add in a salt. It's stored next to the password, so after stealing the password db you've got a list of password hashes, each with its own salt. In order to check a candidate password against a particular user account, you have to append that user account's salt to the candidate, then hash it and compare against the stored hash. That part isn't any slower. The problem is when you try to take that hash and compare it to other accounts. Even if two users have the same plaintext password, their password hashes will be different because they have different salts. The end result is that you have to hash each candidate password fresh for every user.
If you give the attacker the salts, difficulty scales linearly. For two identical plaintexts, all that differs is the salt, but it is given to him.
If you store the hashes and salts separately (the technical details of which I know nothing about), then you augment the keyspace exponentially. For N byte salts, using P characters, you augment it by N^P. Furthermore, who forces you to append a salt? You could prepend. Or n-pend.
I'm just throwing ideas around.
Edit: Or one could move just the salts into the device and store an index into them in the password file. Hashing would be carried out by the device but an attacker would only gain access to the indices. Without stealing the device itself it'd be impossible to properly salt the passwords for hashing.
That one server/device is now a single point of failure and a bottleneck in processing logins.
You're also still relying on the security of a computer, just a special-purpose one.
And yes, it would not be possible to dump the contents of the devices.
It's also theoretically impure in any case, as you've done nothing but add an additional peripheral to the computer. You're seeking obfuscation, not real cryptographic integrity.
It's not obfuscation. The peripheral has a far smaller attack surface than a server. This is real security, even if it comes at a cost of reliability (though one can envision ways of fixing this, too).
Stop trying to patch a hole that isn't there. Salt is not secret data. If you want to protect the hash with secret data, take A1kmm's advice and use the smart card to encrypt it. But don't call that a salt, because it fundamentally is not one.
The only issue is cost of HSMs; they're about $20k/ea right now, since there are only two significant vendors, and they're not widely used.
If someone wanted to do "HSM for general purpose web login, to eliminate the DoS potential of scrypt, and the brute force hash db problem of anything else, and the idiocy of plaintext", the price could probably drop down to $500 or less.
If a cracker gets access to the web app, they'll be able to monitor the exchange, but they'll need actual privilege escalation to be able to read/dump the whole hash database.
What's making it hard for you to analyze this is that you're using the wrong terms. The secret stuff is password - the non-secret stuff is salt. If you want two secret bits, you want two passwords, or to break the password into two pieces for separate storage.
But after all that, you still want a salt, because you're not using it for secrecy but for ambiguation of identical plaintexts.
But splitting the password hash is a bad, or at best neutral, idea because passwords are likely to be semi-human readable and thus a guessable password that matches hash1 is likely the actual password, and will match hash2. This wouldn't be true if we used random passwords, but we don't. So splitting the hash is mostly totally ineffectual, as is having two separate hashes of the same string - the attacker usually doesn't need to examine both.
What you're thinking of is a secret-splitting scheme; where you split a secret among multiple parties, where you need the information from both parties to reconstruct the secret. That's a valid cryptographic technique, but that's not what's meant when you say "salted password".
The problem is, for passwords, that approach isn't really workable. Whatever machine is verifying the passwords will need access to the full hash to do so, so if it's compromised, you could get access to the full hash.
I suppose you could split the verification into two parts, on two separate machines. Each one would have it's own password database, with independent salts, and the machine trying to authenticate the user would request verification from each of the other machines before allowing you in. However, that would make your password system slower and more fragile; now if either of those machines goes offline, your login functionality breaks (and yes, you could do a "two out of three" system to avoid that, but now you're talking about tripling your hardware requirements just for your password verification system).
Really the best bang for your buck will come from (a) securing your system so that it doesn't reveal your password database in the first place and (b) using bcrypt, scrypt, or PBKDF2 to make brute forcing a database that is disclosed much more difficult. It's far more likely that you will build a secure system by following basic security practices like isolating services and giving each the minimum privilege that it needs, doing code review, keeping inessential services off of important machines, using prepared statements rather than building SQL queries from user-supplied strings, and so on, as well as using off-the-shelf security components that have been well tested like the above hash algorithms, than you will by designing your own password system using some novel technique to spread the secret out among multiple machines.