Every Byte of a TLS Connection Explained and Reproduced
tls.ulfheim.net
tls.ulfheim.net
EDIT: I'm putting a CDN in place.
I didn't even get a chance to explain the (normal, server-side) x509 certificate signing much because it just kept taking over the document. TLS is complex enough that just explaining the happy path is 400kb of HTML.
(Isn't that what every site turns into eventually, a custom CMS?)
As it is it's all tcpflow, hexdump, and vim.
[0] https://serializethoughts.com/2014/07/27/dissecting-tls-clie...
Nice.
Very nice work !
I may make a version where the bytes used for lengths are highlighted, since it feels like so many bytes are lengths; look at the SNI extension, which has three 16-bit lengths, I know why they're there, but SNI probably shouldn't be a list, and even if it was a list, an extension that consists solely of a list has a list of the length of the extension, you shouldn't need two bytes for that, and if we recast sni into just a type and a string, the string is clearly going to take the rest of the extension length, so it doesn't need a two byte length either.
I recently had to implement two way auth over 1.2 and this would have saved much hair pulling. (and who does TWO way auth over an MPLS connection. Turns out, us).
And don't get me started about the hassles of obtaining signed certificates that are actually usable for client auth...
What sort of clients were you authenticating? The Web PKI needs to be trusted by random people from the whole world, but most mutually authenticated systems have a relatively small number of clients which are known to the server operator out-of-band. So probably the Web PKI is not the right choice. Instead you (the server operator or some neutral facilitator if it's a group of providers operating services for the same clients) should operate a CA for this purpose, not piggyback on the Web PKI.
One reason not to use the Web PKI if you aren't actually part of the public Internet is that we, to put it bluntly, don't give a shit about people who do that. Running a PKI is expensive (not just in dollar terms, it needs a bunch of smart, motivated people who are morally upright or it's worthless), and this one is ours, so it obeys our rules.
If you have your own PKI (or just one CA) you set the rules. Fifty year certificates for 1024-bit RSA? Why not. A current passport photograph baked into every certificate? Sure. Want the issuer to mint the keys and keep a copy? Do as you please. All those things are prohibited in the Web PKI.
Totally curious, it sounds like you were going for a traditionally signed certificate approach. So you had clients that you didn't "own" -- normal Joe Public -- using your service? If so, that is definitely way outside the norm. If not, why didn't you just sign your own certs?
I might be able to shed some light on there, as this topic isn't actually that hard. It just requires some very careful thinking.
He did a great job of demonstrating every byte and jitter on the wire and how it related to the underlying BSD TCP/IP stack.
It's a bit like having a low-level debugger. If you're happy with low-level C, and you're looking at a debugger and it says variable 'k' which you know is a uint8_t which loops from 0 to 5, currently has the value 65 you should say to yourself. "Hmm, I bet that the compiler used the same place to store variable 'c' that's a single byte from a text string, so this is just the capital letter A and the compiler has realised it doesn't need to store the value of k even if it's technically in scope..." rather than "OMG my loop variable somehow massively exceeded its expected range, maybe cosmic rays have damaged the RAM".
With TLS for example if you give it a whole TLS 1.2 sesssion, Wireshark will say oh, this is TLS 1.2. Fine. But if you show it only a TLS 1.2 connection that failed, Wireshark will say "Oh, this is TLS 1.0". Why? Well, the low-level protocol has been bodged over the years because of crappy middleboxes, so Wireshark doesn't actually know for sure, and rather than say "I don't know yet, I need to see more of the connection" it says TLS 1.0
This can be a problem because you'll get amateurs saying "Our system can't talk to your server because you only do TLS 1.0" and you say "No. You are wrong" and they say "Look, here's a Wireshark trace" and sure enough Wireshark is telling them it's TLS 1.0 because their system has disconnected early (e.g. because they disabled all the crypto algorithms you allow), and so Wireshark wasn't sure and labels it TLS 1.0 rather than TLS 1.2
This is going to happen again with TLS 1.3. TLS 1.3 deliberately says "Hi I'm TLS 1.2" (middleboxes again) and so that's what Wireshark will report (until you get a newer version that knows to look inside the supported_versions extension field for the version) and so you can bet that amateurs are going to say "Your service only does TLS 1.2" when actually their connection failed for some reason and they don't understand how to read the Wireshark trace.
Writeup/comment: http://www.moserware.com/2008/04/towards-moores-law-software...
Comment thread: https://news.ycombinator.com/item?id=846028
The vpri texts: http://www.vpri.org/writings.php
Appendix e and section "A Tiny TCP/IP Using Non-deterministic Parsing" of "STEPS Toward The Reinvention of Programming: First Year Progress Report, December 2007.":
The first couple drafts took forever to figure out, and I got a bunch of stuff wrong. This guide would've saved a ton of time back then.
Now do a version with DTLS (one more message type, couple more fields on existing types, and logic concerning retries). Also, now do a TLS 1.3 one.
I originally used an AEAD cipher but found it was impossible to demonstrate on the command line (openssl enc refuses to do AEAD because it can't confirm the authentication in a streaming context).
A friend asked me to demonstrate ALPN in this but as I looked into it I found it was distracting, as there's already so much going on and any new feature required digression. Maybe next time!
As for 1.3 my next project was going to be implementing it rather than documenting it. Just a throwaway implementation, nothing you'd want to use.
Since you read the Go source, you might like [0]. I will say I personally think Go could have done better. I think it's too compact, too hidden, too disorganized, too underdocumented, and too inflexible/non-extensible. I began to pick some of it apart for a DTLS impl I started at [1], but have put on temporary hold yesterday due to other work obligations.
0 - https://github.com/cloudflare/tls-tris 1 - https://github.com/cretz/go-dtls
This website lays things out really nicely, would love to have more protocols :)
Using Diffie-Hellman to generate a shared key with each party's private key and the other party's public key is the part that amazed me most when I was trying to understand the handshake back then.
I think it's important to mention that even with ephemeral cipher suites, the server's ephemeral public key is signed using the server's certificate private key and verified by the client, since otherwise one would be able to MITM the key exchange.
# in your javascript console, paste this:
[].forEach.call(document.querySelectorAll(".record, .calculation"), function(el){el.classList.add("selected")});
[].forEach.call(document.querySelectorAll(".record, .calculation"), function(el){el.classList.add("annotate")});
[].forEach.call(document.querySelectorAll("codesample"), function(el){el.classList.add("show")});
Because the other party needed to know you'd picked this particular random data to make the keys, the messages from them encrypted with those keys couldn't possibly have been pre-recorded / replayed.
In the ephemeral Diffie Hellman modes both parties contribute to the key anyway so this isn't as important, but with old school RSA the random values are the only thing preventing Replay attacks.
TLS 1.3 capable servers also scribble "DOWNGRD" in part of the random field if a client message says it can't do TLS 1.3. If a TLS 1.3 client sees that unusual "random" choice it knows bad guys tampered with the connection (attempted a downgrade attack). If bad guys just change the values, they won't match between client and server and the connection aborts. Older clients think nothing of the unusual random value and carry on as before.
I haven't looked at the spec in detail, but does this mean that random generation has to specifically exclude that "sentinel value", lest it accidentally occur?
Still, the likelihood of this happening by chance is miniscule.
So it's one in 2^64 random connections
Also the client isn't even checking for possible downgrade if it got the protocol version it wanted (if I wanted TLS 1.3 and I got TLS 1.3 that is not a downgrade). So if "One in every 16 billion billion connections fails" is unacceptable, upgrade your servers and the problem vanishes.
*Edited an incorrect UDP reference out based on the below comment.
https://security.stackexchange.com/questions/22711/is-it-a-b...
The reason I ask in this thread is that this thread treats some of the low-level minimum traffic necessary between clients and servers.
It's not clear what you're asking. Gmail obviously runs over TLS. It also seem pretty nippy to me, but TLS only has a minor impact on the speed.
OK. (my experience since the redesign is the opposite.)
can you (or anyone) put this into quantitative terms? How much are we talking about here? I realize this is a bit off-topic, but the topic is "every byte explained", so I think the people who are interested are in the right place to discuss it.
You'll want to go to the performance tab, and then record and reload the page (ctrl-shift-e). For me, about 3/4ths of the time is spent 'scripting' which is 'JS and DOM work'.
It's not terribly enlightening, though, because all the JS is minified and takes some work to understand.
And much of the rest of the records might go away completely if the session were resumed from the client's memory (this wasn't demonstrated here).
It gives a nice strings-like view of raw data.
Thank you.
ulfheim.net - no problem
At least according to my megacorp threat filter. I have never actually seen something blocked before, and it's a shame because the page would be great to share with my team.
The only things I can think of:
- it doesn't like the hostname (tls?)
- the hostname is new, and has no reputation
- too much h4cking contentLikely explanations for your experience:
1. Your clock is wrong. If your system currently thinks this is Thursday 11 October for example, that's a problem, 'cos this is Friday 12 October.
2. There's some subtle configuration error on their server (seems unlikely as it looks to be just a generic AWS setup) that results in the wrong certificate being presented.
3. Your OS or browser trust store lacks the root CA "DST Root CA X3" operated by IdenTrust. If you didn't deliberately choose to do this, you should investigate as most likely you aren't getting important security updates.
All three causes can often by diagnosed by closely examining the detailed error reported in a browser e.g. SEC_ERROR_EXPIRED_CERTIFICATE
it mightve been invalid while he set up his cdn / reverse proxy?