Hacker News Reader

Comment by tptacek | Hacker News Reader

What seems to have happened here:

An Egyptian telecom company, MCS Holdings, contracted with CNNIC, the Chinese national Internet authority, to obtain a CA=TRUE certificate for use in their internal enterprise proxy --- ostensibly for use only with MCS's own hostnames. Users of MCS traversed that proxy to get to Google, at which point the proxy dutifully generated a (fake) Google certificate to bypass TLS for that connection. Google noticed.

Internal enterprise MITM proxy sounds creepy but isn't. There's a bunch of good reasons why a company would need to decrypt TLS traffic leaving their own network.

But enterprises don't need delegated CA=TRUE certificates to accomplish this. They can just roll their own self-signed root CA=TRUE certificate and built it into their machines. There is no reason a CA should need to put the entire Internet at risk solely for the convenience of a single company's IT operations.

A Chicago company called Trustwave did something similar a few years back. Are they still an HTTPS CA?

View on HN

If a company can 'put the entire Internet at risk', even if 'solely for the convenience of [their] IT operations', the system is fundamentally flawed.

As you can see, there is a lot of trust that is given to CAs. The whole cert security depends on it. The only (current) real remedy is the nuclear option - removing those CA's certs from the major browsers. Then the other side (Chinese browser vendors) can retaliate, of course. So negotiation is required to maintain detente.

> The only (current) real remedy is the nuclear option

Blockchain-based solutions like Namecoin & DNSChain would have prevented this attack without forcing people to rely on untrusted third-parties (if Google stored their domain info in a blockchain).

We compare various mechanisms here:

https://github.com/okTurtles/dnschain/blob/master/docs/Compa...

EDIT: Not sure why this comment is getting downvoted. Maybe some folks don't want this problem to be fixed? :-\

Derivatives of Moxie Marlinspike's Convergence cert plugin that allows you to assign your own trust authorities for verifying signatures. [0]

[0] https://github.com/moxie0/Convergence/network

> Derivatives of Moxie Marlinspike's Convergence cert plugin that allows you to assign your own trust authorities for verifying signatures. [0]

The only derivative of Convergence that actually addresses the problems with Convergence (ironically), is FreeSpeechMe, which btw, relies on Namecoin's blockchain.

But downvote me again for pointing out facts. lol.

"Downvote me again for pointing out facts" is the "wake up sheeple" of nerd message board discussions.

If Google stored their domain info in a blockchain how would anyone know what identifier actually belonged to Google?

The last time I brought this up you admitted there was no good solution yet, has that changed?

> If Google stored their domain info in a blockchain how would anyone know what identifier actually belonged to Google?

The same way they know that Google chose google.com and Twitter chose twitter.com instead of twitter.io or something else.

> The last time I brought this up you admitted there was no good solution yet, has that changed?

This is referring to something else I think, how to transfer .com's into a blockchain while preserving ownership, is that right? That could be done (if needed) with a centralized registrar whose job it was to vet the ownership of the names upon first registration in some namespace. After the initial registration, control would be transferred to the owner of the domain. This is again, however, specific to the case of migrating ICANN domains (if someone wants to do that).

I don't think DNSChain is a feasible project.

However, we can reduce the scope of the problem, and just focus on forcing certificates to be public. [1]

If we went a bit further, and required that the certs be in the public log for some minimum amount of time (say 6 hours), that would have made it possible to shut down MCS before they got started.

[1] http://www.certificate-transparency.org/

> I don't think DNSChain is a feasible project.

If you want to prevent MITM attacks, it's one of the only options available to you (probably the only realistic option):

https://github.com/okTurtles/dnschain/blob/master/docs/Compa...

> However, we can reduce the scope of the problem, and just focus on forcing certificates to be public. [1]

DNSChain/Blockchains already provide certificate transparency (publicly auditable log of certs issued), and they do a far better job of it than Certificate Transparency.

>DNSChain/Blockchains already provide certificate transparency (publicly auditable log of certs issued), and they do a far better job of it than Certificate Transparency.

Better in what way?

We wrote a blog post to answer this question: https://blog.okturtles.com/2015/03/certificate-transparency-...

>The CT spec allows only one SCT to accompany a certificate, making this attack feasible

No, it doesn't. It describes the format of multi-SCT on page 16, and it explains the rationale for this (basically all of the points you brought up) on page 32.

> No, it doesn't. It describes the format of multi-SCT on page 16, and it explains the rationale for this (basically all of the points you brought up) on page 32.

I see the wording wasn't very clear, so I removed the word "only" to make the meaning clearer. It now reads:

"The CT spec allows one SCT to accompany a certificate, making this attack feasible"

Good catch. :)

Would you mind elaborating on the good reasons for MITM your company web traffic? I'm not being facetious, genuinely curious.

For instance, if you are a company that handles confidential medical information (any health care organization, many insurers, every employee benefits management organization, &c), you may be required to have controls in place to ensure that nobody uses your Internet connection to exfiltrate people's PII through Google Mail.

Similarly, many investment banks and financial information firms have strict requirements to monitor all communications owing to SEC rules and insider trading regulation.

Is there any way to detect that you're being MITM'd?

You check the certificate chain and compare it to a clean version. Obviously that's going to be a solution limited to very savvy users.

That's part of the point of certificate pinning.

Would love to know this as well. I only have a high level understanding of the purpose of CA Certs, but beyond that I'm lost.

Ignorant questions ahoy:

1. Using Chrome, would you have to manually accept the MITM certificate? 2. Could such a certificate be valid across multiple domains? 3. Would it pose any threat to the computer if it was moved from the MITM network to an outside network? 4. What kind of potential problems could occur if I issued a self-signed certificate for my network?

Check the certificate store for the browser you are using. Mozilla Firefox has its own. Internet Explorer and Chrome on Windows rely on the Windows certificate store.

http://windows.microsoft.com/en-us/windows-vista/view-or-man...

Details on what certificates come with your installation of Mozilla Firefox: https://www.mozilla.org/en-US/about/governance/policies/secu...

Your organization may add their own certificates as described below: https://www.utexas.edu/its/help/user-certs/817

As far as I understand it (please someone correct me if I'm wrong):

1. In this case you would not have to manually accept anything, as the root certificate (the CNNIC cert) is already in your browser/os and the certificate chain for certs created by MCS would be OK (because their cert is signed by CNNIC).

2. As CNNIC issued them an intermediate CA cert, MCS was able to create certificates for any domain they wanted and these certificates would be considered valid by everyone that has CNNIC in the root store. So the MCS cert is not valid accross multiple domains, but it allows MCS to create certificates for every domain which kind of has the same consequences.

3. I think it would pose a threat when leaving the MITM network, but not as a consequence of having been in the MITM network. Only the root certificates are stored locally. Websites have to send a complete certificate chain that anchors their certs in one of the root certs. This means that the cert generated by MCS is not stored and therefore not used when leaving the network anymore. The danger is that this intermediate cert allows MCS to generate certs for any domain and use them outside their network, too.

4. A self signed certificate would have to be installed on the machines in the network. Otherwise users would get a certificate warning and would have to add the cert to their rootstores themselves. Other than that I think that this would grant you the same MITM-powers as this intermediate cert did for MCS, with the only restriction that you couldn't create certs for domains not in your control that would be accepted by users outside your network/that don'd have your self signed cert installed.

> Is there any way to detect that you're being MITM'd?

Yes, a project we're working on called DNSChain prevents MITM attacks for domains that have their info stored in a blockchain.

We're maintaining a list of alternatives and how they compare to DNSChain here:

https://github.com/okTurtles/dnschain/blob/master/docs/Compa...

It's the golden unsolved, perhaps unsolvable problem in crypto. Trusting trust and such. There's always a key, somewhere, that has to be shared.

One way to hack the system is if you have actual knowledge of the person you are talking to, and you assume some limited amount of tampering which can be done in real-time. For example, if I know the sound of your voice, and we want to agree on a key with no MITM, we can setup an audio channel and speak some code words to each other. Baring an adversary which can in real-time intercept and synthesize my voice convincingly speaking a different code, this is pretty secure. [1]

  [ZRTP] allows the detection of man-in-the-middle (MiTM) attacks by displaying a short
  authentication string (SAS) for the users to read and verbally compare over the phone.

Another imperfect defense is spreading over time the data that an attacker would have to intercept and modify in order to MITM. That's what Chrome is doing with their pin lists. Now an adversary would have to alter the pinning when Chrome is downloaded. Of course in this very thread we're talking about technology which can do exactly that. E.g. technology which has any hope of preventing data exfiltration, would have an easy time altering Chrome's pin-list. Of course the Chrome binaries are signed, so there's another layer to defeat, etc. etc.

So the end result is there are a lot of good technologies to prevent MITM. If you can keep the attacker out once, you can generally be confident your future conversations will be secure as well, since good protocols don't start from scratch each time, but rather "ratchet" new keys from the old as you go. [2]

One of the big trade-offs is false positives and privacy. For example, it might be nice if my browser remembered the public key of a site I visit, like HN, and let me know if it changed. Two issues are a naive implementation would also serve as a great tracker for every site I've visited, and how do I know if when I get a warning, it's a real attack and not just an expiring certificate rotating out? Now we would need a way for sites to indicate, by signing with their old key, that indeed they are switching to a new key, and complexity explodes from there.

[1] - http://blog.cryptographyengineering.com/2012/11/lets-talk-ab...

[2] - https://whispersystems.org/blog/advanced-ratcheting/

An interesting thought would be using a bloom filter to store certificate fingerprints. It would prevent someone from getting a list of all the websites/certificates a user has seen. However the significant downside is that a certificate hit could be a false positive and the user hasn't ever seen that certificate before.

> It's the golden unsolved, perhaps unsolvable problem in crypto.

I think this problem was solved fairly well by Namecoin back in 2011. Software like DNSChain [1] then makes it possible to securely access blockchains like Namecoin without having to run a full node on your phone or other device.

If you can't run your own DNSChain server (or don't have a friend's you can use), you can query two or more independent servers and make sure the responses match.

Dionysis Zyndros recently came up with a mechanism whereby you can even query a single DNSChain server (that you might not trust), and still be assured of correct replies if you received an accurate key once (we'll be publishing info on this technique soon over at blog.okturtles.com; it's somewhat similar to what you're talking about with ratcheting keys).

We maintain a comparison of various approaches here:

[1] https://github.com/okTurtles/dnschain/blob/master/docs/Compa...

Of course how could I have not mentioned the blockchain? Thank you!

Part of the trick with blockchain is validation. Everyone is not going to keep a full node, not even close, and just delegating trust is not the answer. You want to trust but verify.

I'm not sure what the state-of-the-art is these days for SPV-type verification, but I don't see anything in the current DNSChains response which would allow any kind of independent verification of the returned data.

Edit, also see: https://en.bitcoin.it/wiki/User:Gmaxwell/namecoin_that_sucks...

> Part of the trick with blockchain is validation. Everyone is not going to keep a full node, not even close, and just delegating trust is not the answer. You want to trust but verify.

Right, so hence the two techniques I mentioned in my reply: query more than one server, and/or use Dionysis' "proof of transition" (for lack of a better name).

> It's the golden unsolved, perhaps unsolvable problem in crypto.

Sorry, perhaps my question wasn't clear. I wasn't asking about MITM in the general case, I was asking about this particular case. The certificate chain for Hacker News seems to go AddTrust -> COMODO -> Another COMODO -> *.ycombinator.com. So in this case, if you're MITM'd by MCS Holdings, is MCS Holdings going to be part of the chain (after a CNNIC)?

Yes, the chain would be different. MCS Holding cannot become Comodo (Comodo = AddTrust, btw), so the chain would change to CNNIC -> MCS Holdings -> *.ycombinator.com.

For web site with authentication (e.g. bank account), protocols like SRP (Secure Remote Password) would prevent the man-in-the-middle if he doesn't know your password. SRP is a mutual authentication protocol with zero knowledge and forward secrecy, it would be nice if major browsers supported it, it's not usable without browser support.

> if you are a company that handles confidential medical information (any health care organization, many insurers, every employee benefits management organization, &c), you may be required to have controls in place to ensure that nobody uses your Internet connection to exfiltrate people's PII through Google Mail.

Yes, but what are those controls? You check every packet to see if it contains any information from one of your databases?

What if the person sending the data just applies a simple obfuscation technique to the data, or just tunnels through some other encryption scheme?

Edit: downmods? really?

Search for things like “data loss appliance”. As an example, when BlueCoat isn't helping repressive regimes spy on their citizens, they're helping businesses watch every outgoing packet:

https://www.bluecoat.com/products/dlp

“Blue Coat DLP allows you to easily create policies that analyze the data source, content, destination and more.

…

accurate data “fingerprinting” capabilities, in addition to powerful keyword, pattern, and regular expression support, so you can create precision policies to effectively secure your data while minimizing false positives.”

Sure, the every HN reader might have questions about this but I'd bet a LOT of C-level executives are receptive to this.

What questions do you have? Limiting the scope of discovery to fingerprinted data is a pretty good way to limit "incidental" data filtering/discovery.

Oh, sorry, I meant questions like the ones raised about how someone might try to smuggle data past such filters or some of the security aspects of having a single point with access to everything.

I certainly agree that if you have a requirement to watch outbound data like this, having a system to selectively capture it is much better than simply attempting to record everything.

You don't check for every record in your database, you create a regex (or multiple regexes) which matches the patterns you don't want leaked. This is how I've seen Data Loss Prevention done in the Sophos UTM.

Yes, if even the simplest obfuscation technique is employed, this system falls flat on its face. (Shh don't tell the regulators)

Sophos is a low-end solution. Higher end solutions (e.g. Vontu) do in fact let you detect on individual records or groups of non-regex detectable groups of records using fingerprinting.

Simple answer: Yes, they basically check every packet, or at least as many as they can. No, DLP isn't perfect, and it doesn't always work. This should not be a shocker.

Notes:

1) Modern DLP solutions have some pretty sophisticated obfuscation detection tech. Like almost all of these kinds of technologies, they're looking for the 80% case, not the 99% case.

2) Tunneling out encrypted tunnels is subject to traffic analysis techniques. It's not as uncommon as one might suspect to detect out-of-band ex-filtration of many different types this way.

I guess my hangup was on the claim that such a scheme could 'ensure' prevention of ex-filtration which, frankly, seemed laughably impossible.

Please, point out any systems which have believeable claims for doing this. In my experience most 'DLP' systems do no such thing, they are just like the bit of string which stops you stealing pens at the bank, basically theatre.

Automatic analysis to statistically detect hidden channels is a research topic, it can be used to put bounds on the exfil rate but not reliably detect it.

What if the site you're accessing is required to have controls in place to ensure that nobody can intercept the communication between the user and the site? I'd expect that to be the case when the site handles things like confidential medical information.

When you are using your employer's computing equipment on your employer's premises, you have limited right to privacy.

In a typical deployment of MitM tech (e.g. Bluecoat, Websense, etc.), things like personal banking, health care sites, etc., are exempted from the interception policy to avoid personal privacy issues and HR headaches. This can be overridden in the local policy of course, but I've rarely seen that in practice (anecdote isn't fact, blah, blah).

Be aware that the site you're going to may be MitM'ing sessions to meet other compliance regulations (e.g. SOX in the financial sector).

> In a typical deployment of MitM tech (e.g. Bluecoat, Websense, etc.), things like personal banking, health care sites, etc., are exempted from the interception policy to avoid personal privacy issues and HR headaches.

How does it know? Does it have a list of all "personal banking, healthcare sites, etc" from the whole world? How is that list kept up-to-date? What happens if the site the employee is accessing is missing from the list? What happens if the employee knows these sites aren't monitored and finds a way to use them to bypass the monitoring?

> Be aware that the site you're going to may be MitM'ing sessions to meet other compliance regulations (e.g. SOX in the financial sector).

If it's the site itself, is it really a MITM? And even if they technically use a MITM, does it really matter, since the site would have access to the plaintext anyways?

All of the reasons that benefit the company. Data Loss Prevention, web filtering, malware detection, take your pick.

While there may well be good reasons to MitM HTTPS on a corporate network, the case isn't quite so clear for secretly doing so. IIRC chrome disables altogether certificate pin violation notifications if the offending certificate is locally installed. While it probably is the right move to forgo the giant security warning in that case, I don't see why they can't use an "orange light" type indicator and in the popup explain corporate monitoring (with a further note that if you aren't on an institutional computer you probably have a virus). While corporate IT departments could recompile chromium sans message, or just ban chrome altogether, why make it easy for them?

If you are using corporate hardware it should not really be secret that your traffic is being man in the middled, this should be assumed. I think it makes sense for a corporation to be able to whitelist their certificate to make intermediate network devices appear as trusted, because they are. I do agree that having a little asterisk or something for the lock ice to indicate that you are using a certificate that falls outside of the default trust store could be beneficial.

The stuff I work with has the option to pop up a warning/comfort page to explain the practice/policies etc every N hours - seems to be a good thing to use.

One reason would be an Intrusion Detection System: make sure no malware gets downloaded, or to analyze traffic for stuff that looks like malware phoning home.

http://en.wikipedia.org/wiki/Intrusion_detection_system

A good friend of mine works for a major electronics manufacturer (trust me, you'd recognize the name) where some employees exfiltrated 200,000 internal design documents/artifacts to a competitor that they were planning on joining. As a result, no one can use gmail/gchat from work machines or VPNs anymore. I think a MITM proxy that detected giant file exfils but allowed people to send gchat messages to their family would be a huge plus in that environment.

Signing another "CA=TRUE" cert seems like it should be a very restricted and audited operation, right? Is it out of the question to say that all such certs should be cleared by 3rd parties (like Mozilla and MS), on pain of revocation? Or is there a large use case outside of CA infrastructure I'm unaware of?

Google's response here seems a bit weak.

I would agree; there's really no reason that all major browsers couldn't ship with a complete list of all acceptable CA=TRUE certificates, intermediate or otherwise.

Unlikely and would cause problems. Parent was suggesting that they should be cleared separately without having to update browsers. I like the certificate transparency idea better though, and I wonder if it is possible to refuse new certs via public endpoints but allow certs to be manually added to the logs and SCTs to be manually issued, in case going that far is needed.

What problems, precisely? Sure, it would prevent current CAs from selling sub-CA certificates without coordinating with browser vendors. That's the point. What's a legitimate use case for doing so?

Yes, but the point is that this "coordinating" can be done without users having to update browsers themselves.

All the major browsers have automatic updates these days. And if the coordination doesn't include a browser whitelist, it has no teeth.

I just edited my comment to mention the certificate transparency solution.

Presumably there's some reason they aren't just don't this already. I mean, I sure hope so otherwise it sounds like a rather big flaw for no reason.

Trustwave is still in the Mozilla CA bundle: https://wiki.mozilla.org/CA:IncludedCAs

At the end of the bug discussion here: https://bugzilla.mozilla.org/show_bug.cgi?id=724929 it was decided to give Trustwave a reprieve. Mozilla policy was updated to explicitly forbid such usage, and each of the CA was required to verify that they were complying with the new policy or state when they would come into compliance.

CNNIC did so here: https://docs.google.com/spreadsheet/pub?key=0Ah-tHXMAwqU3dGx...

Kathleen Wilson's comment on the bug was: https://bugzilla.mozilla.org/show_bug.cgi?id=724929#c66 "My intent is to make it clear that this type of behavior will not be tolerated for subCAs chaining to roots in NSS, give all CAs fair warning and a grace period, and state the consequences if such behavior is found after that grace period."

However, over 14th months later when it came out the ANSSI (aka the French government) was doing the exact same thing, rather than revoking the root certificate Mozilla decided to limit them to issuing certificates to: .fr, .gp, .gf, .mq, .re, .yt, .pm, .bl, .mf, .wf, .pf, .nc, .tf

which AFAICT essentially acquiesces in MitM French firefox users that go to French websites.

I wonder if the response will be the same here.

> Mozilla decided to limit them to issuing certificates to: .fr, .gp, .gf, .mq, .re, .yt, .pm, .bl, .mf, .wf, .pf, .nc, .tf

Is there any way to do the same, manually, for the other "national" CAs? I woudln't mind if CNNIC handed out a certificate for every .cn domain out there, but if they ever try to sign one for an Egyptian entity (or even worse, a .com domain), I want to see a big red warning. Ditto for the Japanese and Taiwanese governments, which Firefox also seems to trust unconditionally.

I actually do this to some extent, as I don't quite trust the NIC of my own government. I told my browser not to trust it, so whenever I try to visit a government website, I get a big red warning. I override the warning after confirming that I am indeed visiting a government website protected with a government certificate. But if the government NIC ever tried to show me a certificate for a non-government website, I would know immediately. This works, but it's inconvenient, so I'd love to be able to restrict any given CA to subdomains of specific TLDs and/or second-level domains.

So is the Chinese who signed the certificate, probably.

Too big to fail and all that jazz.

> But enterprises don't need delegated CA=TRUE certificates to accomplish this. They can just roll their own self-signed root CA=TRUE certificate and built it into their machines. There is no reason a CA should need to put the entire Internet at risk solely for the convenience of a single company's IT operations.

Doing it this way, however, means that they don't need to worry about pushing their self-signed certificate out to all their machines, right?

That could be done easily with group policies on Windows machines which would take care of, at least, Internet Explorer. I haven't used a Windows machine in a long time, though, so I'll ask: do the other major browsers use the built-in certificate store? If not, they'd still have to address the problem of getting their self-signed certificate "trusted" by Firefox and Chrome, for example.

Firefox does not use the built-in certificate store. Mozilla has their own trusted CA list.

Chrome does, Firefox doesn't. I'm not sure if there's a way to mass-push certs to FF with AD

Not directly via AD. But you can add it to the certificate database in the template Firefox profile.

> Users of MCS traversed that proxy to get to Google, at which point the proxy dutifully generated a (fake) Google certificate to bypass TLS for that connection. Google noticed.

I'm curious about the mechanism of Google noticing - was Chrome side-channeling information about it's cert to Google? Because if it was a true MITM proxy, google would never have talked to the browser directly to know what cert the browser was being presented. That's kinda how the whole MITM thing is dangerous - it's invisible to both sides if done correctly...

Google Chrome automatically reports back to Google if a certificate appears for Google and it is not issued by Google's own intermediary. It also blocks it from ever loading via HPKP.

Chrome ships with a list of CAs allowed to issue Google certificates. If Chrome encounters a Google certificate signed by some other root authority, it phones home.

http://blog.chromium.org/2011/06/new-chromium-security-featu...

If you're on a machine which has self-signed root certificates in its trusted store does this mean that all bets are off? Can you achieve authentication, integrity and confidentiality despite having an adversarial root certificate on your machine (for example, if all network connections go via proxy which do MITM on TLS connections and DPI)?

In the blog post it says that CNNIC issued the CA=TRUE on the basis that MCS H only use it for domains that they have registered.. Wouldn't it be better to just issue a CA cert with name constrains extensions?

Why do we have that extension if nobody uses it :( ?

Reply on news.ycombinator.com