cubesnooper

I have another Raspberry Pi sitting next to my desk, with a keyboard and a tiny screen, dedicated to systems administration. My user on this machine has an SSH key that on every machine logs into an account with sudo access. To revoke a user key, I run a script from this machine that logs into each host and updates sshd’s RevokedKeys.

I have no mechanism at the moment for revoking host keys, which is a harder problem to solve as it would involve updating a number of laptops, phones, etc. that may not be powered on at a given time, but that’s less of a problem since if I knew a host key had been compromised I wouldn’t be logging into it anyway.

> Your personal (private) key lives on your workstation, and CA key lives on RPi.

Yep.

> There are some other machines that you want to SSH into. They trust CA public key.

Yep, with TrustedUserCAKeys.

> The remote servers trust the certificate issuer, verify that you own the certificate’s private key, and let you in.

Yep. Additionally, the servers themselves present CA‐signed certificates alongside their host keys. My GlobalKnownHostsFile contains the CA public key, so when I connect to a host for the first time, there’s no “unknown host” warning and my user’s known_hosts file is not updated.

> If so, how do you issue the certificates that live on RPi?

The CA has a directory containing the public keys of every user and host that I set up. A cronjob periodically runs ssh-keygen -s against these files, and copies them to htdocs. Each host and user has a cronjob that periodically fetches its certificate and copies it to /etc/ssh or ~/.ssh, respectively.

> How is this more secure than simply creating new certificates and replacing the old ones is the authorized_keys files?

It’s more convenient for me than updating authorized_keys. When I build a new machine, for example, I first generate a new SSH keypair on the machine. Then I copy the server and user public keys to the CA. Once I drop them in the right folder, certificates get generated automatically and served over HTTP. Then on the new machine I set up cronjobs to refresh the certificates every two weeks. I didn’t have to update authorized_keys on a dozen other machines, and I didn’t have to inspect any host fingerprints.

> If a host has been accessed by an attacker due to an exfiltrated key in the past it's tainted forever.

Yes, that’s obvious (although I did mention shell script in my previous comment—been a while since I thought this through). The scenario I imagined at the time was more like, did I ever copy my private key to an unencrypted flash drive, and then lose the flash drive? I never let private keys leave a machine anymore, but maybe I wasn’t so strict about that five years ago, and when did I generate my main SSH keypair? Was it four, five, six years ago? I no longer have to worry about such things. I could have just rotated my keys, but with short‐lived certificates I now get the benefits of key rotation without having to do any of the work.

Now I tie all my SSH keys to a WebAuthn key, which provides even more protection, because within the three‐week window that my certificates are valid, an attacker would have to also physically possess my Yubikey.

SSH provides native support for certificates; they’re a custom (non‐X.509) format. The signatures are generated with the ssh-keygen command; I set my infrastructure up purely by reading the manpage, not referring to any blog posts or anything, so I think the documentation is a good way to get started.

https://man.openbsd.org/ssh-keygen.1#CERTIFICATES

I’m glad Userify works well for you. For my purposes, whipping up a couple of cronjobs involving curl and OpenSSH is more appropriate than relying on an external cloud service.

> But I've been burned before when a central auth server was down and I couldn't log into my servers

There is no central auth server involved here. My servers check login credentials against the CA’s public key, which is installed alongside my sshd config file. I make my certificates valid for three weeks, but regenerate them every two, so if some failure happens with creating or fetching certificates I have a week to notice and fix the problem.

> So anyone on your LAN can visit the URL and download the CA private key?

No, the only files served to the LAN are the certificates, which contain the signatures by the CA of the public keys of other machines. Those are safe to distribute openly because they’re useless without the private key of the public key that was signed.

You’re thinking of known_hosts, not authorized_keys.

Rotation is actually a lot easier with certificates. I just generate a new key on the client, copy the public key to the CA, and I’m done, with no need to repopulate authorized_keys on all my other machines.

In another comment I went into more detail about how I keep the CA secure.

I have a Raspberry Pi dedicated to generating certificates. It serves the files to my LAN statically via a webserver, and is otherwise heavily firewalled. I don't run any other software on the Pi, so barring an exploit in the webserver, I’m not worried about the signing key getting compromised.

Compared to my desktop, where over the years I ran all kinds of stuff from the package manager, downloaded Python scripts and configure scripts from GitHub and Sourceforge… I tried to be careful but you can’t audit everything.

> On the other side, as a personal user of SSH with basically one person to worry about, the effort of setting up a certificate seems like just a waste versus the existing key-based infrastructure; I don't understand at all what attack it would prevent or what convenience it would provide for the cost of learning it.

The main benefit I get from using SSH certs at home is expiration. Before, I always had a niggling feeling at the back of my mind that the public key I’d been using for four years could have been surreptitiously exfiltrated by some shell script three years ago, and I’d never notice.

Now my certificates expire regularly, so I no longer have to worry about whether my machine remained secure throughout the entire continuous past; I only have to worry if my machine is secure in the present.

I’m reading the third book right now and so far the series has been a real struggle to get through. In the first two I found the setting mildly interesting, but mixed with uninteresting characters, dialogue, and plot, with writing that was entirely too verbose and with too many repetitive phrases (like “he raised an eyebrow,“ which happens about once a chapter and sometimes even multiple times per page—see also chuckling, rolling eyes, and snorting, which occur at about the same frequency). Clearly not everybody finds this irritating but I definitely did.

I don’t think I had any problems like that with the writing of other fantasy series like Lord of the Rings and A Game of Thrones. Nor with two other trilogies I read for the first time this year, C. S. Lewis’s Space Trilogy and The Three‐Body Problem.

Ubisoft actually sells some DRM‐free games on GOG—the first Assassin’s Creed, Far Cry 1 and 2, Rayman Origins (but not Rayman Legends).

If you have disk space to spare, I would recommend downloading GOG’s offline installers as a backup. The risk of the service disappearing or games you’ve bought being removed seems pretty low, but DRM‐free offline installers are just as good as the classic “physical copy,” provided you actually saved them!

I would love to have a voice changer so I could put video streams online without fear of having my voice cloned: https://www.forbes.com/sites/thomasbrewster/2021/10/14/huge-...

Sure, I’m nowhere near a bank director on the list of “attractive targets for voice cloning,” but who knows how widespread this attack might become in the future, and by the time one’s voice is out there on the Net, there’s no way to take it back.

I would like to use a voice changer when making phone calls to businesses too. I can totally imagine future corporations creating a new revenue stream by selling models of a known person’s voice to advertisers so they can later correlate the voice to that person.

Unfortunate that Lyrebird’s transformations seem easy to reverse. I wonder if there are any FOSS tools that make it harder to recover the original voice.

> the main thing you're getting out of the VPS is the static IP.

Yes, that and RDNS.

> If you're paying for that, why not just pay for a static IP at home?

That’s a good question. I too hear that mail providers consider IP blocks assigned to VPS providers less trustworthy than others. The reasons I don’t take the ISP/dedicated server route, aside from price, are:

• VPS providers are not tied to my physical location. If I move, I probably can’t take my ISP’s static IP with me (I may even move to somewhere they don’t service). Conversely, if I want to switch away from a local ISP, the selection of alternatives is extremely limited.

• Risk of neighboring IPs reducing the reputation of the block exists with server companies and local ISPs as well. I concede that the problem is probably worse with VPSes, but I hope to mitigate it somewhat by avoiding bottom‐of‐the‐barrel providers and by the fact that my own IP will never be used to spam.

• I’m somewhat worried about the possibility of DDOS, and VPS companies provide a lot of cheap bandwidth, so in case of attack I might be able to salvage the situation with careful firewalling on the VPS.

> You need a dedicated box, not a VPS. … Unlike some people are saying, you should never do this off a VPS if you have an interest in keeping the email secure and functioning for a long time.

I agree that hosting a mail server directly on a VPS compromises privacy and control. But there’s a better alternative: use VPSes for cheap static IPs, while hosting the server locally on hardware you physically control, using WireGuard tunnels and port forwarding to connect things. Port forward incoming SMTP over WireGuard to your real MX, and use MTA‐STS and DANE so that as many senders as possible will TLS‐encrypt mail they send you. Have your outgoing SMTP server handle DKIM signing, then send it out via WireGuard so it looks like it came from the VPS, while enforcing TLS encryption.

The VPS won’t be able to forge mail from you without your DKIM keys. It won’t be able to read your outgoing mail due to TLS. It won’t be able to read incoming mail that’s TLS encrypted. It will be able to read unencrypted mail, but the big providers that follow MTA‐STS will abort if the VPS attempts to block encrypted connections.

This has the added benefit of reducing your dependence on an external provider (the VPS company) for server setup. If you’re unhappy with a particular provider, just switch to another one. The issues associated with sending email from a brand new IP will be there, but you won’t have to set up complicated infrastructure on the new host, only a few WireGuard tunnels and firewall rules.

You don’t have to store your DKIM keys on the VPS. I keep my signing infrastructure local, and send outgoing mail over a WireGuard tunnel so it looks like it was sent from the VPS.

Spam is often sent from big providers, too. For several years I hosted my email the “middle ground” way (i.e., relaying outgoing mail via Google Workspace), and despite using DMARC correctly it was not infrequent that my emails would go to spam (even to GMail boxes) or never show up at all.

Obviously GMail is such a giant that email providers have to be very careful when blocking it, but enough spam comes from there that receivers clearly use some heuristics to block some of it. I’ve even received multiple rejection notices because the GMail server my email was sent through happened to be on a blacklist!

I switched last year to sending directly from my VPS. It was partly for privacy from Google, but moreso so I could enforce outgoing TLS. For the first few days they went to spam boxes or moderation queues, but I made sure they were rescued, and ever since I’ve had no deliverability issues sending to Google, some local ISPs, and even Microsoft (which seems crazy, as I never got a mail from my domain to show up in Outlook when I was relaying through Google).

I can only speak for my own experience, of course. But that is what I experienced.

> I just don't understand the attraction of self hosting email.

For several years I’ve hosted in the “middle ground” sense described by the OP, running my own incoming mail server and relaying outgoing mail through a big provider.

The main benefit for me (compared to using a big provider with my own domain) is personal privacy. When I used Google for mail, Google had access to so many pieces that make up my personal life: Purchase receipts. Flight itineraries. Conference registrations. Emails from my university. Emails from my realtor. Utility bills. Notifications for subscribed forum threads, GitHub repositories, Wikipedia pages. Whatever newsletters I chose to subscribe to. Theoretical access to any site with password reset by email. Running my own MX eliminates Google’s access to most of these things.

There are other some other benefits too. Free infinite aliases I can use to sign up on any website. No fear of dependence on features that might get paywalled. No sudden danger of having to migrate data to another provider.

> If you're spending more than an hour a year maintaining your self hosted email (which you will, big time!) then your Google Workspace / O365 is paid for.

Reducing my data footprint is something I care about enough to spend my spare time on.

TOTP is not as secure as WebAuthn, because if you enter the TOTP code into a phishing site, the phisher can now successfully authenticate as you. WebAuthn was specifically designed to be immune to this case: if you were to use your WebAuthn key in a phishing website, the phisher would not then be able to authenticate as you on the real site.

It’s almost as trivial with this format too, at least to guess what address is used for other services, though it has a strong advantage over using ‘+’ in GMail in that nothing will try this automatically. It’s hard to believe anyone would intentionally try to guess a different service’s email to spam to it, but even so in my setup I prefer to eliminate this possibility completely by adding a random number to the service name: experian12322@example.com, and so on, with no catchall for invalid addresses.

So far the most spam I’ve gotten has been to the address I used for Amazon (probably leaked by a third‐party seller there).

I know there are CT search services like crt.sh, but is it practical to download the raw data and search it locally? If the logs are append‐only, it feels like a perfect usecase for rsync.

> That said, the only caveat to hosting in your own house is it could suffer a fire, and your data is wiped

Well, there are other reasons to prefer using external hosting. Home connections are typically port‐filtered, have dynamic IP addresses, and have a low IP reputation, and your ISP selection is very limited. Whereas if using a VPS there are so many options that it’s easy to shop around.

But you can still self‐host while getting the benefits of a VPS. Just forward ports from the VPS over a WireGuard tunnel to your real machine. Then all the actual infrastructure is on hardware you control, and the cloud provider has no access to your TLS private keys.

Recursive resolution leaks all my DNS queries in plaintext to my ISP, the nameservers, and everyone in between; on top of that, my ISP can monitor what sites I’m viewing through SNI and server IP. If my DNS queries are encrypted and anonymized, my ISP only gets SNI and server IP. And ECH seems to be moving quickly, so within a couple of years I expect the SNI leak to be plugged.

> The longer-term solution is to wait for DoT to become prevalent in authorative servers.

That has a serious deployment problem, far more so than ECH. It’s going to be years (and years and years) before a person can successfully do recursive resolution via TLS. Is that even on anyone’s roadmap?

But that would leak all of my DNS queries in cleartext.

I use cloudflared to do DNS lookups via Cloudflare’s Tor onion. It’s weak to vulnerabilities like this one, but it disassociates my DNS lookups from myself, and TLS certificates mitigate the risk of hitting spoofed sites.

I’ve hosted a personal Matrix server (Dendrite) and web client (Element) for the last six months, with unhappy results. In practice an entirely self‐hosted Matrix stack seems to be kind of unreliable for me. Having joined just a couple of large rooms—open source project chats that migrated from IRC after the Freenode debacle—sync on login never seems to work right, either getting stuck on a loading animation, or loading messages out of order (“Friday” followed by “Today” followed by January followed by “Yesterday” followed by “Today”). The room history seems to have corrupted itself somehow, according to the logs, but I have no idea how attempt a resync from scratch besides wiping and reinitializing the whole PostgreSQL database. And from the beginning, the client randomly disconnects from the server, but when I refresh it’s connected again.

I’m no stranger to self‐hosting, and I personally run my own email server (with MTA‐STS, DANE, and mandatory TLS for outgoing mail) and Mumble for voice chat, and even Pleroma. These have all been very reliable, enough that I regularly communicate with friends this way. In contrast, my Matrix experience makes me reluctant to suggest it to anyone I know, because it’s hard enough to get anyone onto something user‐friendly like Signal.

I want to love Matrix. The idea is really cool, there is obviously a lot of work behind it, and I appreciate having something federated to compete with Signal’s model. But it just hasn’t worked for me. I hope the self‐hosted situation will improve.

I’ve seen a few venues where it’s possible to digitally buy DRM‐free movies, as a straight up downloadable 1080p MP4. Unfortunately the selection is very limited, to pretty much just a handful of indie documentaries and short films.

I know of three sources: VHX, Vimeo On Demand, and GOG. Although upon looking it appears VHX is now owned by Vimeo, so there are really only two.

https://vimeo.com/ondemand/

https://www.gog.com/movies

For personal domains, I bite the initial cost and buy the domain for 10 years, then every year top it up to 10 again. For a $20/yr domain that’s only $200 up front, and if the cost suddenly goes up or some other TLD policy changes that I hate, I have plenty of time to gradually move to a cheaper/better domain.

One thing I’m not sure of is what happens if I want to switch registrars in that time—will the full 10 years of ownership transfer to the new registrar?

To keep Google from maintaining a profile of my behavioral patterns and video viewing history based on my IP address.