E.g. your first idea might be "ok, so the client just sends a SHA-256 hash of the password, simple." But... if you do that, then the SHA-256 hash _is_ the password. A rogue server can capture that SHA-256 hash to log in elsewhere.
So you might try and improve that scheme by having the server supply something to hash together with the password. But that means you can't use /etc/passwd anymore, since that doesn't store the password, it just stores a hash with one particular salt.
Now, to be fair this does mean that you have to give every client the salt for the user they're trying to authenticate as. However IMO a password storage system should still remain secure even after you get the salt for the passwords (you didn't get the hash, so the salt is not useful information to an attacker).
Would that be a viable method, or have I missed something obvious?
But your scheme has made the password hash be the password, which is unacceptable because it's equivalent to storing the plaintext password on the server.
Illustration. Here's /etc/shadow
cyphar:salt:hash
With your scheme, an attacker is allowed to log in by providing f(nonce + hash)
so that's a red flag: the attacker doesn't need to know the password, just the hash. Looking at it another way, you've made the hash be the password.(FWIW, I initially thought your scheme worked, but couldn't believe a solution had been overlooked and so I tried it with a concrete example and only then realised where the problem lay. )