I've established impromptu SSH tunnels from other people's machines to my local network so that I could watch my media on their TV.
If you dropped me naked on the other side of the planet I could get a copy of my identity documentation and access my email, bank accounts, etc from any internet connected machine I find.
This whole thread is a little alarming.
Life is about trade offs, if someone really wants to spend the time to get access to my home dev box then I may have to spend a couple days on the phone with the bank and restoring from my offline backups. Big whoop.
Your home is likely insecure from my standards. Do you have a firearm at the ready? Do you know how to use it? Does your family have codewords to communicate without letting others on to your plan? How hard is it to kick in your doors? Not just the front door, but the bedrooms. Do you have a dog to wake you in the night? How stocked are you, can you last a month with no resupply? Do you even have a panic room?
I protect what matters beacuse you can't protect everything.
I know the risks well, I don't find them to be worth the hassle of avoiding them.
Edit: I looked it up to confirm. First time password is encrypted analogous to tls
I would contrast this with the weakness of keys being that if the devs keys are compromised so are all the other servers he has access too. (I can memorize my passwords, or write them on a note card. Say what you will about that, it's out of band.)
In light of that the question as to what's best is your threat model. Poor opsec per dev or an upstream network sniffer.
Thanks for all your input all over this thread. I'm revisiting my convictions.
Would you disagree with I've said?
Also - I suspect this Caddy server doesn't support it, but OpenSSH does - you can use FIDO and then the keys physically are objects in the real world, from say Yubico or a dozen other vendors so now "losing the keys" is like losing your office keys, except that when they give you a new one they can trivially make the old one stop working.