If I got you right about ssh-agent, there is a console frontend https://www.funtoo.org/Keychain.