Off-topic: Has anyone ever managed to get PAM/Google Authenticator working with RADIUS? I spent a while messing about with this last year, and never got it working exactly as I had hoped. I'm no longer working with RADIUS, but this post reminded me I never finished scratching that geeky itch.
http://it.isevil.org/blog/2011/11/13/authentication-service-...
Code: https://github.com/bithive/example-totp-vault
For FreeRADIUS we use rlm_perl to define our own authenticate() method; it just calls the web service to validate the codes.
Yes, if you lose either factor, you can't access the server. This is why with Google Authenticator you also get a one time pad with emergency codes. However, I don't know how well this would work with two-factor SSH... you'd need a separate one time pad for each server. And I'm not sure how the Google PAM module handles emergency codes.
http://code.google.com/p/google-authenticator/source/browse/...
If you lose your phone and your scratch codes, you've only lost access via SSH. So it's an inconvenience, but one you can overcome with the right setup.