If it's important enough to still need a password on top of that, the password can go on the key.
I'm not sure if you can put your ssh keys on a specific (non-login) keychain.
If you want those, you may want to go to Keychain Access > Preferences > First Aid > uncheck "Keep login keychain unlocked".
This makes it possible to keep valuable items in one or more auxiliary keychains set to always prompt (lock after 0 minutes). (This technique isn't a panacea but it contributes to defense in depth.)
However, it doesn't solve the agent caching problem. Once a key has been added the agent, Keychain never asks again, even after it's locked.
ssh-add has an option "-t <seconds>" to make added keys automatically expire. That will work, but it only works for newly added keys. As far as I can tell, the Keychain helper calls ssh-add to add keys, and I don't see a way to have it pass -t.
Edit: Looks like holmar's suggestion below to change the ssh-agent daemon to run with -t would fix this.