Encrypted traffic interception on Hetzner and Linode targeting Jabber service
notes.valdikss.org.ru
notes.valdikss.org.ru
https://ddanchev.blogspot.com/2021/03/exposing-currently-act...
https://ddanchev.blogspot.com/2019/07/profiling-currently-ac...
https://blog.talosintelligence.com/picking-apart-remcos/
https://flashpoint.io/wp-content/uploads/Plea-Agreement-USA-...
Really interesting writeup though. I guess it's a practical example of why everyone should get a CT monitoring service!
For high risk targets, consider layering an additional auth mechanism that doesn't rely on trusted CAs: Tor onion services, SSH, or Wireguard.
> All issued SSL/TLS certificates are subject to certificate transparency.
+1. crt.sh has RSS feeds for this.
> Limit validation methods and set exact account identifiers
Using CAA is a good idea in general, but would it help in this case? The attacker would just request the exact cert configuration that is permitted by CAA. Maybe this helps if you can strengthen one validation method?
> Monitor SSL/TLS certificate changes on all your services using external service
+1. High-risk targets should be aware of what certs are valid at any time, and be checking for those.
> Monitor MAC address of default gateway for changes
A more sophisticated attack could preserve the MAC address.
> "Channel binding" is a feature in XMPP which can detect a MiTM even if the interceptor present a valid certificate.
TIL.
Author of ACME-CAA (RFC 8657) here. ACME-CAA can mitigate this because you can put a unique identifier for your ACME account in the CAA record, so it is not possible for an attacker to do this unless they can get your ACME account private key (or coerce the ACME service). This assumes you have DNSSEC-secured nameservers, of course, otherwise DNS requests can potentially also be intercepted when queried by the CA.
See RFC 8657 for a full list of security caveats. The RFC is designed to be more readable than most.
Blogpost by me with more background: https://www.devever.net/~hl/acme-caa-live
I usually brood on blogposts for days before publishing them since I care a lot about getting things accurate, but this is a bit of the moment, so here goes. Always happy to get feedback by email or IRC: https://www.devever.net/~hl/contact
atlas would be good for detecting the time Pakistan announced 0/0 or whatever.
Also, thank you for your work as well as popping in here.
Am I overstating it by seeing it as one of the most important milestones for Internet security in general over the past few years?
Do you have a take on DLT-based systems and if this is something that is (or could be) seriously discussed? It seems to me that the issues we have with PKI and certificate transparency could actually be mitigated very well if blockchain was seriously considered.
There's this thing called Zooko's triangle (https://en.wikipedia.org/wiki/Zooko%27s_triangle), basically the premise that an identifier can only have two of the following properties: secure, decentralized, human-readable.
But a blockchain can actually be used to square this triangle and get all three. The original example of this is the Namecoin project, a fork of Bitcoin using a lightly modified Bitcoin codebase which can be used as a key-value store.
The idea is that a blockchain can be used to create a decentralized database mapping keys (domain names) to values (which can be things like IP addresses, but also PKI trust anchors, similar to DANE). Thus this can eliminate the need for a CA. You can also use it to map human-readable names to .onion addresses. There is also root-of-trust transparency built in since the contents of the database is public and any changes are also public. The right to change a key-value entry belongs to the public key of the person who registered that key. Nobody can override this. Namecoin domain names use the .bit (unofficial) TLD.
It's a really neat technology and also makes things like censorship via domain suspensions, your registrar getting hacked, etc. infeasible.
Full disclosure, I previously worked on the Namecoin project, I authored the DNS resolution daemon and the technical specification for the format of DNS data in the KV store. Unfortunately public interest in the project has waned, and deployment is the real issue - you have to have the client software to be able to resolve these domain names. There is at least some conceivable possibility Tor might ship it in their bundle in the future though, so people could type example.bit in Tor Browser to get to a hidden service. The project remains active though.
Couldn't you exchange public keys with LetsEncrypt (in the web UI) and encrypt the response so you can't be MITMed? Why is http even an issue?
Edit: To be clear, this is a problem with a solution. But you asked why simply throwing a LE cert into the mix wouldn’t prevent the issue.
And no, I don't mean throwing LE certs around to prevent MITM - this whole article is about the difficulties before having an LE cert, so that's necessarily excluded.
I'm wondering "why not client certificates". They're a well established way to stop MITM, seems like a simple choice for the ownership validation step.
You should try it sometime.
Explaining and guiding people through that is the whole point.
DANE is about changing the way certificates are authenticated. DANE makes it possible to authenticate certificates without getting them issued by a well-known CA. So CAA records are not particularly relevant to DANE. You can use DANE with certificates issued by a CA, which gives you two ways to authenticate the certificate; in this situation CAA secures one path and DANE the other.
I am one of the co-authors of the DANE SRV RFC https://www.rfc-editor.org/rfc/rfc7673 which is what XMPP would use. I don’t follow XMPP development so I don’t know if it has been deployed. I would like it if DANE were more widely used, but it’s not pertinent to this attack.
There are some use cases where DANE is actually winning real victories and is actually more viable than the existing CA infrastructure - site-to-site SMTP, for example.
It is trivial to fit several compressed curve points (i.e. signatures) in a single packet, whereas you can't even fit two RSA signatures in a minimum-safe-to-assume DNS UDP reply packet after accounting for padding and ASN.1 overhead.
I get the feeling that there is some faction that really hates UDP and they sort of hijacked the DNSSEC situation to use as a lever to force people to allow DNS-over-TCP.
That seems to be backfiring, however, and DNSSEC has wound up taking a bullet for the UDP-haters.
Many very-large networks simply can't afford for their DNS traffic to be exposed to TCP's intrusion-detection malleability and slowloris (resource exhaustion) attacks. These networks appear to be simply ignoring the "thou must TCP thine DNS" edict. DNSSEC is not a good enough carrot for them. I think ditching RSA would have been a more pragmatic choice than ditching UDP or skipping DNSSEC.
When I query vjhv.verisign.com I get a response containing four 2048 bit RSA-SHA-2 signatures in 1049 bytes which is well within the EDNS MTU for unfragmented UDP, so I’m not convinced the problem is as bad as you paint it. There have been problems with EDNS trying to use fragmented UDP, but that has been reduced a lot by newer software being more cautious about message size limits for DNS over UDP.
DNS needs TCP even in the absence of DNSSEC, because there are queries you cannot resolve without it. Some operators might convince themselves they can get away without it, but they will probably suffer subtle breakage.
I was referring to the non-EDNS 512-byte limit.
Yes, you get ~2.5 times more with EDNS. Still, four records is not a lot.
> DNS needs TCP even in the absence of DNSSEC, because there are queries you cannot resolve without it.
Theoretically? Perhaps. Some would argue that connectionless DNS is valuable enough that people should not create those resource records. Before DNSSEC that was a working consensus. And with ECC it could be once again.
For signing only there are much more efficient PQ cryptosystems, with signatures around the same size as ECC. If DNSSEC ever adopts PQC it will be one of those systems.
Here are two of the earliest, and easiest to understand. There are much better ones now.
https://en.wikipedia.org/wiki/Lamport_signature#Short_keys_a...
There are a few reasons for this oddity: partly so it matches with DANE for MX records, partly to support large scale virtual hosting without reissuing certificates.
You should be able get a cert with subject names covering the server host name(s) and the service name(s).
A public CA generally has a more sophisticated relationship with their network transit provider or (hopefully) providers and can get DNSSEC actually working as intended for them.
So this means mything.example's DNS services and some public CA both need working DNSSEC, but the visitors to mything.example, such as your mom's sister or some guy who just go into mything but isn't entirely clear whether Apple make Windows - do not need DNSSEc, for them everything works exactly as before, yet the certificate acquisition step is protected from third parties.
Would that help? It depends.
Until the nation state tells a CA within their borders to issue a cert without publishing the CT logs, along with a gag order. If they don't comply they go to jail. You think Billy Bob Sysadmin wants to go to jail to protect a russian jabber server?
In the latter case, the malicious certificate + any CT log issued after that certificate should have been included are evidence of the attack that is easily verifiable and would likely cause browsers to drop the (edit) CT log unless provided with a very plausible excuse that isn't "we complied with a court/gag order".
Also, this would require the police etc. to compel 2-3 different entities: The CA and two log operators (I think one of them could be the CA itself).
Using different logs than normal would stand out, so it'd typically be two specific log operators that would have to be compelled to hide this. The US may get lucky and have jurisdiction over all of them, other countries are much less likely to.
a) Browsers aren't the only clients that matter. Will any XMPP client even look at the CT log? Probably not.
b) Browsers are not going to drop all US-based certificate authorities. And if they tried they might just get their own national security letter (reminder: all major browsers are US-based).
You can't work around the government using centralized infrastructure.
Browsers would happily accept it and IIRC they don't check against the logs.
As the other commenter already pointed out, Firefox does not require this. Safari and Chrome do. This indeed is not directly applicable to this situation, since XMPP don't involve browsers. But for websites, the parents attack scenario is not applicable.
With mild difficulty you can mirror the disk, net, and RAM of any "commodity" hardware.
Or, someone very diligently followed the orders - there was an order to set up a cert, but there was no requirement that it has to auto-renew :)
So they were told to renew the certificate, but not how many times to renew it?
One obvious possibility could be e.g. sending a notification to the previous ACME account: "hey, a new ACME account request a certificate for your domain".
I wonder why this didn't.
Their theories are interesting and would explain how they obtained the certs if true. Perhaps the take-away here is to have multiple probes on multiple providers checking all of ones TLS fingerprints and alerting on unknown fingerprints and then checking the certificate transparency log or lack thereof.
openssl s_client -servername news.ycombinator.com -connect news.ycombinator.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin
SHA1 Fingerprint=7E:49:BA:40:86:87:B3:39:66:93:94:9E:9C:45:71:85:3C:8D:95:16
There are several indications which could be used to discover the attack from day 1:
* All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names
* Limit validation methods and set exact account identifier which could issue new certificates with Certification Authority Authorization (CAA) Record Extensions for Account URI and Automatic Certificate Management Environment (ACME) Method Binding (RFC 8657) to prevent certificate issue for your domain using other certificate authorities, ACME accounts or validation methods
* Monitor SSL/TLS certificate changes on all your services using external service
* Monitor MAC address of default gateway for changes
In the case of Jabber it might be interesting to add some monitoring in the application that uses a cryptographic signed payload to list all the valid fingerprints and send an alert message to the server if something odd is happening like public key pinning but without a hard-fail. That list could be pre-loaded with new certificate fingerprints prior to being deployed. If the oddness is confirmed then perhaps add some way to tell the clients that certificate is likely forged. That way both server operator and client are aware that something evil this way comes.
[Edit] - Looks like based on the comment from MattJ100 that some Jabber servers and some clients can already do something like this.
the major certificate transparency logs are operated by several independent global companies, Apple and Google for LE. it's unlikely that they will agree to forge their global CT logs for a single government. more importantly, SCTs allow cryptographic proof of anyone lying, making such an action very dangerous for their continued participation in the WebPKI ecosystem.
So is it possible to run my own copy of crt.sh? How demanding it is (e.g. data size)?
Yes, the logs are public and I ran equivalent monitoring for a previous employer before the pandemic. You will need to consume records at the rate they're created to keep up, Let's Encrypt have stats you can look at, if you can cope with twice their typical daily throughput you'll be fine on average, but peaks will swamp you temporarily so design for that.
You can choose whether to store everything, or just stuff you consider interesting, and you can choose whether to care forever or only until expiry (so 398 days)
If you want everything (full certificates), indefinitely, that's a lot of data. Um, several terabytes per year maybe?
UPD: I was initially wrong. It looks like many CAs will issue a valid certificate to attacker capable of doing MitM.
it was a DV cert, and it was considered validated because someone was MITMing the traffic. that would have worked against any CA, nothing at all to do with Let's Encrypt.
ejabberd and Prosody support it, and a number of clients too.
Doesn't that require the server to persistently store user passwords in plaintext?
SCRAM also allows the client to verify that the server possesses the password (or a hash of it), so a MITM that just says "yep, your credentials are correct - go ahead!" can be detected.
Channel binding is an addition to SCRAM (it's usable outside of SCRAM too) that allows securely checking the TLS stream as well. Specifically it allows verification that the TLS stream is terminated by the same entity that has (a hash of) your password.
checking for new public keys in certificates when keeping private keys between renewals would also help.
i suppose any storage and possibly ram should also be considered compromised? e.g. private keys stored on disk without encryption, but also by being read into memory. especially on the vm.
In the end law enforcement can also walk up to the machine/hypervisor and steal/monitor interesting things from that as well. This is funnily the exact evil maid threat scenario so many (especially Linux people) find unrealistic.
i agree there's probably no way to keep a machine hosted at a company secure from law enforcement. also why i suggested storage and anything in ram on the machine can be considered compromised. this attack (swapping out network connection for a while to get a certificate through let's encrypt) was probably easiest/least intrusive. if it wasn't an option, the next easiest option would be taken. perhaps the options that are harder to execute are more likely to be detected, or less likely to be worthwhile.
It's not like anybody is watching or using DNSSEC like that. Also at best you might be able to detect a change but it won't prevent the attack and neither will it leave a long-term mark like CT would.
Your own hardware + continuous video monitoring is probably good enough. The idea is not to keep it secure, but to know when a breach has happened.
Kepping DNS registry, CA, and hosting in different jurisdictions could be a noticeable improvement...
I am one of the co-authors of the DANE SRV RFC https://www.rfc-editor.org/rfc/rfc7673 which is what XMPP would use. I don’t follow XMPP development so I don’t know if it has been deployed. I would like it if DANE were more widely used, but it’s not pertinent to this attack.
What I think you're advocating for is that DNSSEC should have it's own transparency log that shows when domains update or cycle their DNSSEC KSK/ZSKs which is a great idea, we would just need to get all the domain registrars on board as well which as we can already see with the transparency project, you can't get buy in from everyone.
This of course only works because there is competition. If TrustCor fails to behave, then this CA is dropped. Customers complain, but not very strongly - there are plenty of CAs around, it's annoying to have to switch to another one but still eminently doable. And TrustCor is out of business completely, all their investment is permanently gone.
With DNSSEC.. what do you do if you domain registrar does not play by the rules, say is caught double-issuing DS keys?
Answer is: likely nothing. Most orgs are not going to move .com to .io because of that -- there links, SEO, even things like business cards. People would complain, _maybe_ registrar would get a fine for 0.0001% of their annual income (but I would not count on this). And that's it, the system will stay insecure and they will continue to double-issue certificates.
The separation of CA from registrar is a _feature_, and a very helpful one. Competition is good, and we should not make our existing natural monopolies even more useful.
It is important to note that when a law enforcement with this capability would just legally take ownership of the domain instead. If the domain is under control getting domain validated certificates is trivial. There are many legal precedents with domain takedowns.
It is equally important to note that only TLDs under LE jurisdiction is ever an issue. It is not clear whether this specific attack under .ru had been ever remotely possible with any type of authenticated domains.
an attack would involve a tld operator being compelled to cooperate, or their signing keys being compromised (and then probably adding a DS record for your domain). if that were found out, it would be bad news for trust in dnssec. have there been any known case of this happening? it would be hard (impossible?) to fully detect without transparency log (like for ca certificates).
The more I think about it I would wager that Linode and Hetzner were just law enforcement having good taste in VPS providers. It's more likely to me that LE was the compelled target.
https://letsencrypt.org/documents/ISRG-Legal-Transparency-Re...
We’ve never been ordered to issue a certificate.
While I am a Let’s Encrypt employee, this message is not an official communication of Let’s Encrypt and shouldn’t be interpreted as such.
For some background I've spent a good deal of time with lawyers and C-Levels playing devils advocate trying to find a way to indirectly notify customers but it's just not legally possible in the United States of America and there is nobody that would risk violating one. People here on HN often bring up canaries but they are not compatible with a NSL.
It does not appear that NSLs compel arbitrary actions or require the recipient to actively lie about having received one.
https://www.google.com/search?q=%2290%253Ade%253A01%22+linod...
Googling for '"90:de:01" linode' and '"90:de:00" linode' indicates that addresses from these blocks have been assigned to other linode VMs recently. I sadly don't have a linode VM of my own to compare with right now but it would seem like the traffic has been routed to another VM on linode infrastructure
Obviously securely exchanging keys with an anonymous drug dealer over the Internet is error-prone though...
If you haven't heard, it has lots of problems and a lot of people recommend avoiding it (for example https://www.latacora.com/blog/2019/07/16/the-pgp-problem/ / https://news.ycombinator.com/item?id=20455780)
* https://articles.59.ca/doku.php?id=pgpfan:tpp
PGP certainly has its problems, but isn't really special compared to other similar things. The big advantage that PGP has is that it is a stable and well known standard. There is a tendency to imply that it is insecure in some way, but no real evidence seems to exist to that effect.
If you define "similar thing" as "kitchen-sink thing that tries to do everything like PGP does", then this is true, as no full alternatives exist, nor should they.
But for all practical applications? Pretty much every "other similar thing" that I have tried is _vastly_ more simple and more reliable and easier to debug and infinitely easier to script. For example, "seccure", "minisign", "age", even "ssh-keygen -Y". Especially cool are "seccure" which uses passphrases as private keys (no more private key files ever!) and "ssh-keygen -Y" which uses ssh keys which everyone already has anyway.
If you are writing a new software and thinking about integrating PGP, do yourself a favor and look for alternatives. If this is something developer-oriented, I recommend using something based on SSH keys, like git does.
most E2EE messaging services (e.g. Matrix, Signal, WhatsApp) enable verifying other people instead of devices, reducing the required verifications for one person to 1 instead of 1 per session
no, you could have multiple sessions per device, e.g. desktop client and browser tab
> That comes from the Signal protocol.
no, Signal doesn't require this
The attacker would need to set up a separate MiTM for the particular E2EE scheme used. Some of the XMPP clients I have encountered will not let you use a particular cryptographic identity unless you have explicitly claimed to have verified it.
Still a good reminder. If you have not seen and dealt with a ridiculously long number (or equivalent) you have not achieved end to end.
There's also a possibility that one would be a "search" and the other would be an "interception" with different levels of approvals requested, but I don't know what the current legal situation in Germany is right now.
On the other hand, if it is not lawful interception, I doubt Hetzner would allow it. Because that’s also against law.
but like other have pointed out this seems to have been in the hetzner network
through wire taping laws also extend to datacenter internal interconnects I mean servers of different people can communicated with each other without the traffic leaving the server so it kinda makes sense
I have all my machines' public keys, and all major git forge's SSH keys in my git-managed cfgmgmt repo.
For NixOS users: https://search.nixos.org/options?channel=23.05&show=programs...
Of course, checking the server key is better than not doing so, but any knee-jerk response to TOFU will probably make your security worse.
I don't really get why people use ssh at all if they aren't using certs. Might as well use telnet.
The same thing applies to VPN services. Why bother trying to develop tools etc that can decrypt traffic when you can just have people send their traffic to you?
They can’t use HTTPS if you don’t have a certificate yet
Also, if they cannot do secure validation then maybe they should stop issuing certificates for sites that already have a proper certificate.
For EV certs there might be more validation, but users will never see the difference between EV and DV certificates.
Or creates network/pid namespaces and puts you in them, while leaving the mitm server in the original one?
If so, the mitm could be on the same host, and wouldn't need the cooperation of the hosting provider.
I'm not sure how to check for either of these without restarting (which the admin does not seem to want to do, as it is a live service).
Whereas this attack generated new keys (and was detected!), suggesting the attacker didn't compromise the server itself.
In that case, where the redirection happened is no longer something you would be able to tell, right?
Why is it even possible to issue more than 1 certificate on the same domain via Let’s Encrypt? Shouldn't the previous certificate be revoked when a new one is issued?
More use-cases:
- You might obtain a new certificate, but deploy it gradually, so you want the old one to remain valid while you do that.
- One certificate may cover different sets of domain names. If you have a certificate for "example.com, foo.example.com" and then request a certificate for only "foo.example.com", should the earlier one be revoked? (leaving "example.com" without a certificate).
it commonly used in a "normal" way all the time
- e.g. when there are multiple data-center for the same domain (e.g. using geo-location based routing) it's a good practice to give them different certs so that if you need to revoke one the operation in other regions is unaffected
- or when rolling over from on cert to another
- or when moving certs into hardware security keys/module (HSK) you preferably do have one per HSK (so that if e.g. hardware breaks and gets replaced you can just revoce the cert for the affected HSK module not all of them), you also normaly do not keep backups to make sure it can't be leaked at all (as long as the HSK isn't hacked which is normally quite hard)
- or losing access to a cert (e.g. in the case above a HSK breaks)
Lastly the whole CA system is in the end designed to provide good security for the industry while having the backdoor of issuing certs the legal organs to allow the police some degree of wiretapping (oversimplified, it's slightly more complex then that).
Cloudflare for example, tries to optimize certificate delivery (and have backup certificates available for you just in case a CA needs to revoke theirs).
Also, on distributed systems its less safe to share private keys between the various frontends.
so you need a master server to handle the renewals, periodic sync, and to handle the case when the master goes away
this would be considerably more complicated than having a second independent certificate (assuming you've automated the entire frontend provisioning process)
For other more sensible reasons but still.
First, you want to have to have some leeway so you don't need to rotate certs at exact second the old one expires
Second, you might want to have cert-per-server rather than cert-per-domain, as that's frankly easier to implement vs having common store for certs+key
Since the certificates are nowadays renewing quite often, it's very easy to just end up ignoring these notifications. To really get the benefits, I think you should somehow combine these with logs from the legit cert renewals and only alert when something strange pops up.
[1] https://sslmate.com/help/reference/certspotter_authorization...
[2] https://github.com/SSLMate/certspotter/blob/master/man/certs...
If a log fails to publish a certificate despite issuing a receipt for it, Chrome's SCT auditing infrastructure can detect that, as it did recently with the Nessie 2023 log: https://groups.google.com/a/chromium.org/g/ct-policy/c/5x1A6... (this was also detected by monitors operated by myself and Cloudflare)
But this is all moot when it comes to XMPP, since XMPP clients don't check for SCTs, and there is no requirement for CAs to log certificates if they don't need to work in non-CT-enforcing clients. Some CAs (e.g. DigiCert) will sell you unlogged certificates - no need to compel them.
tldr: situation is good and improving with browsers, not so much with non-browser clients
But SCT validation would indeed be something we should investigate for the ecosystem, it could be beneficial for certain use cases.
Feel free to contact me (email in profile) or ask here if you want to talk more about this. I use XMPP (thank you for Prosody!) and have been involved in the CT space for many years now.
This is fine. If you're using a tool that's connecting to the internet, you probably need to monitor for updates anyway.
I'd like to see a separate package, similar to Mozilla's `ca-certificates`, but for CT that can be updated independently of the actual useragent.
People should do this, but in practice it often doesn't happen - as witnessed by the Appmattus library fiasco earlier this year.
> I'd like to see a separate package, similar to Mozilla's `ca-certificates`, but for CT that can be updated independently of the actual useragent.
Considering how slowly updates to the various ca-certificates packages propagate, this would be absolute disaster for CT.
> All issued SSL/TLS certificates are subject to certificate transparency. It is worth configuring certificate transparency monitoring, such as Cert Spotter (source on github), which will notify you by email of new certificates issued for your domain names
This seems like the kind of thing domain registrars should do for you.
openssl s_client -servername news.ycombinator.com -connect news.ycombinator.com:443 < /dev/null 2>/dev/null | openssl x509 -fingerprint -noout -in /dev/stdin
SHA1 Fingerprint=7E:49:BA:40:86:87:B3:39:66:93:94:9E:9C:45:71:85:3C:8D:95:16
The openssl s_client method is likely the most versatile for testing from different locations, ports and applications.
> All jabber.ru and xmpp.ru communications between these dates should be assumed compromised. Given the nature of the interception, the attacker have been able to execute any action as if it is executed from the authorized account, without knowing the account password. This means that the attacker could download account's roster, lifetime unencrypted server-side message history, send new messages or alter them in real time.
Id's say it would mitigate it A LOT. Though I have to agree that there is still a lot of room for improvement.
I personally prefer the approach taken by briar: your public key is your address. Pretty cut and dry, not much room for shenanigans.
> We tend to assume this is lawful interception Hetzner and Linode were forced to setup based on German police request.
> Another possible, although much more unlikely scenario is an intrusion on the internal networks of both Hetzner and Linode targeting specifically jabber.ru — much harder to believe but not entirely impossible.
And what if the attacker tricked somehow the letsencrypt challenges?
Or this is supposed to be impossible?
But for this kind of thing to happen on every connection to the server being impersonated, you either have to bring a very big and publicly noisy hammer like a BGP hijack, or have the Internet upstream of the server cooperate. If the traceroute info in the post is to be trusted, in this particular case Linode and Hetzner themselves—or perhaps their datacenter operators—seem to be performing the intercept.
Ok, now I get it. Thanks.
https://twitter.com/billmarczak/status/1710348549794185279
We need to go back to snail mail or something, this whole .well-known thing just stinks. We added layers on top like CT and while sound ideas, they don't tend to do anything unless you are Google or FB.
> Yes, the fraudulent certificate is memorialized in Certificate Transparency (CT) databases (the indelible, publicly accessible records of ~all issued TLS certs). But, most website owners don't know what CT is, have no idea how to check it, and wouldn’t know what the results meant
I think, emerDNS can be used in such critical application: https://emercoin.com/en/documentation/blockchain-services/em...
There are browser addons that replicate this behavior however and some Jabber clients/servers apparently have their own mechanism for this [1] but one can only hope everyone is using the combination of servers and clients that support this feature.
I really, really hate the idea of this kind of eavesdropping.
Interestingly, search "90:de:01" in this page: https://www.cyberciti.biz/faq/howto-linux-configuring-defaul...
^^ looks like another victim VM of the alleged mysterious interceptor :-)
second pretty much _every_ country has laws which require carriers to help them wiretap in case of an investigation with appropriate court orders
thirdly even if it went through Hetzner/Linode instead of carriers it wasn't done "a move from them" but something they where legally binding ordered to silently tolerate
lastly if as unlikely as it seems it was not lawful interception of police or similar then they (or a carrier) would have been hacked, i.e. there is absolutely no chance that server providers will do such an attack on their own violation, especially Hetzner (they also don't have the legal means to get the necessary certificates)
I think jurisdiction games is all you have, because outside of that there's going to be _someone_ close to you network-wise who will fold when faced with a lawful intercept order.
but that's why running thing at home doesn't help that much, because it also applies to carriers, too
and when it comes to sizing data it tends to not make much difference whether they physically size disk at your home or in the data center, actually if legal order for a sizure like that is confirmed by a judge it's normally applied to all the computers such a person has, both at home and in datacenters
I assume the root cause is DNS tricks.
Maybe read the article
so… not surprised. Still cool. What a time to be alive
edit: I actually thought this story was about virtual machines, but I noticed it was about physical hosts, so those have a bit more hope; but ultimately you're still trusting devices that are not physically under your control, so they cannot be trusted.
But let's say that in the case of virtual hosts: basically there is no way to ensure you have non-wiretapped connection to a virtual computer provided by hosting service. They don't even need to MitM it, they can just look at the memory of the virtual machine. The only attempt at trying to do that properly has been AMD's "Secure Encrypted Virtualization" https://www.amd.com/en/developer/sev.html, but I think it was broken by some researchers and I don't know if it could currently provide some means to do this safely. I suspect even in that case it might be challenging to install the initial operating systems so that it won't contain the MitM functionality in itself.
And even then you would be trusting AMD's implementation of the security layer.
Frankly that the wiretap was discovered this time is just a learning experience for the provider to do better next time. I doubt there are good reasons why it should be possible to detect it, if implemented competently.
so, I relied on market forces here, I hoped that major provider would protect their brand and do not do or allow to do such things, but looks like it is not the case for hetzner and linode.
the suggestion in these posts is that the German cops made them do it, which is exactly what happens to every provider in every country - they get a subpoena, they may fight it, if they lose they do whatever it is. every hosting (and transit and peering, I assume) provider in every country has an elaborate "lawful intercept" system already set up for secretly copying traffic to the feds / cops / intelligence services.
a hijacking of said LI system is what caused a huge scandal / deaths in greece ~twenty years ago: https://en.wikipedia.org/wiki/Greek_wiretapping_case_2004–05
do we know cases of silent wiretapping in Google, MS, Amazon clouds?
Do we know cases of Google/MS/Amazon fighting a targeted wiretapping subpoena and winning it? Do you think they would not have been served one?
absence of proof is a proof that your statement about "any company in any country" is just speculation without much ground.
> Do we know cases of Google/MS/Amazon fighting a targeted wiretapping subpoena and winning it?
we know somehow similar case for Apple: https://en.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption_d...
It looks like in these cases either the FBI dropped the case or the request was found to be unlawful in the first place. Arguably Apple indeed was following the law even when opposing it within the framework of the law, but what the FBI asks them to do is not the law.
We might be looking at a different story had the case fallen to the side of the government—or perhaps no story at all had that requests also contained a gag-order, which I understand is quite common in these cases and could be the key reason why people hardly ever hear of these cases.
I also wouldn't say the case of basically cracking customer device the same as providing wiretapping of a service a company provides similar at all—and even in this case it looks like the FBI would have been given access to the iCloud storage, but the problem was that the latest data was not backed up there, it needed to be retrieved directly from the device.
There were 42 digital wiretaps (not including combinations) in the year 2023 in US according to https://www.uscourts.gov/statistics-reports/wiretap-report-2... "Types of Surveillance Used, Arrests, and Convictions for Intercepts Installed". I wasn't able to find how many of them were in datacenters from that data, though..
my understanding is that courts are eligible to request wiretaps by current law, but forcing service provider to break e2e encryption is not so, that's why google, apple, telegram etc can defend such cases.
Your link says that in 180 out of 190 government could't decrypt traffic despite wiretaps.
Your only alternatives in that case would be the underground style hosters sometimes marketed as bullet-proof hosting though they are just shady resellers with bold claims. They do not typically last long. There are a couple I know of in Amsterdam that are right down the street from The Hague that have been around for a while so I suspect they are just honeypots.
This seems to be somewhat jumping the gun I think.
Given the certs and the target assuming it's an lawful interception seems reasonable.
But there is nothing there which requires Hetzner or Lindoe complying or knowing about this.
Given the nature of the attacks you can do the interception on the carrier level, and carriers being forced to comply with lawful wiretapping is pretty much anywhere in the world pretty much standard and many laws are based around that approach. Much less so around approaches involving data centers.
It's like SS7, or SWIFT. It's not bold to claim that all public phone networks and bank transfers are insecure, it's just a fact. People who know what they are know they're vulnerable.
As an aside to the design issues, social engineering any of the thousands of organizations involved with issuing certs (not just CAs) is trivial. A bored teenager could get certs issued for most domains by using nothing more than a telephone or e-mail. But that's cheating, so I don't count it.
Just grab these IP packets when CA comes to validate that you own that domain. Perhaps EV could solve that to some extent but it is never mandated.
Even if you tried to put any stuff into WHOIS to mitigate this, your hoster can serve any bullshit on this channel too.
It does look very bad and SSH approach to certificates is just infinitely better. If Jabber used SSL keys instead, they will be alerted immediately.
Come to think of it, your hoster can also find ways to steal keys directly from hardware, though.