You have to provide the secret key to each device on initial setup. After that, you just need your password.
There's a tourist experiencing this scenario probably every minute.
How you choose to safeguard it depends on your preferences and your "threat level".
For example, you can keep it in a bank vault or print multiple copies to store it under your pillow, taking a picture, or save it in your email, etc.