Decrypting your own HTTPS traffic with Wireshark | Hacker News Reader

Decrypting your own HTTPS traffic with Wireshark

trickster.dev

531 pointsrl198798 comments

As the blog post points out, this requires coordinating Wireshark's "(Pre-)Master Secret Log filename" / tls.keylog_file preference with you web browser's environment, so a while back I wrote a Wireshark Lua plugin that automates as much of this as possible. It pops up a window that lets you launch Chrome or Firefox with the SSLKEYLOGFILE environment variable set to the correct value: https://gitlab.com/-/snippets/2156054

Thank you for this

I think this would be interesting to have in Wireshark itself, have you considered contributing it back upstream?

I'll see what I can do. :)

Awesome, thanks!

I was trying to hack/reverse engineer some IoT API calls, and I really wish there was some middle ground between "Don't trust any network, encrypt everything end to end" and "I can MitM anything I want". Everything on my network is a (sometimes literally) black box sending and receiving who knows what. As a network operator, I should be able to MitM everything on my network. But I don't want people to MitM me when I'm on a public network, so I'm at a loss on how to fix this.

On public network you use tunnels aka end-to-end-encryption. On your private network you control (theoretically) everything.

"Private" and "public" are relative terms. From the perspective of an IoT smart thermostat talking to its mothership, your home network is a public connection, and it's successfully using E2E encryption to protect its private data from you -- even though you theoretically own it.

Nice! It's a subtle shift but I think your framing is key.

Most b2b gear will play well with a proxy server, and that will allow you to explicitly mitm everything going to the internet. Just don't allow anything else out.

One annoying problem is certificate pinning, if you have deices that join your network and others, but it's doable.

The real problem is devices sold to consumers that don't allow you to install custom certs or allow you to configure a proxy. There should probably be a law that says every network device should have a means for the owner to decrypt the traffic on the fly.

Maybe with some locally installed b2b software, but any kind of hardware device or appliance has trusted root CAs. Unless you have access to one of those, nothing will trust your MitM setup (at least, I couldn't get Nest equipment to connect via a MitM proxy).

Maybe there could be some kind standard DHCP option for a trusted root CA on the network. For user controlled devices (i.e. phones), they wouldn't trust it, but any kind of vendor managed device would. Probably too naive of a solution.

That totally defeats the purpose of TLS. On most home networks, any device can act as a DHCP server. Indeed, a lot of those “limit internet time for kids” devices depend on ARP and/or DHCP spoofing to work. I’m glad the security model of the last couple of decades (trust every device on the LAN) is slowly dying.

Oh and adding a nerdy feature to home routers is never gonna happen. Consider how long it took for IPv6 to be enabled by default.

A problem is that the easier you make it to sneak in other trusted root CAs, the more you open attack surfaces which malicious parties can exploit.

As a device manufacturer that uses certificate pinning, do you really want to trust whatever cheap crap router people have at home to provide a trusted root CA? Let's say that one gets hacked (which happens too often..), you are now also wide open. Sure, it's the owners responsibility, and with certificate pinning that's not configurable, you are depriving responsible owners of the possibility to decrypt their own traffic, but we at least need to acknowledge that there is a balance to strike here, and it's not obvious what the perfect solution would be.

Maybe an opt-out of certificate pinning that is so "hard" to reach that it's really only for power users that know what they are doing. (And with "hard" I mean more than some window popping up that says "Are you sure you want to do this? You'll void all warranty blah blah" -- people have already been conditioned to just click away certificate warnings in their browser; the same people would just click anything to just go on with their lives..)

A physical switch on the device would be a good solution. Physical access is mostly "game over" anyway.

It needs to be something that requires certain skill. Maybe a dip switch inside the enclosure. Average Joe won't dare and it's too much work anyway. At least if you just try random things to get your IoT device to work if something is off.

I'd love to have that.

It could just be a text file that you have to modify with vi over ssh. Even if the SSH server is only available if configured through the scary pop up in the gui, maybe even require a public key be uploaded for authentication. Not too difficult, yet technical enough that no one is going to do it without thinking.

The point of a physical switch would be to restrict access to someone with physical access-- ideally the owner.

Also guarantees that you may need to break some seal that then can be proof for tampering, warranty-wise. (Which otoh some users may not like.)

There is no "balance", this is purely a question of ownership. If I can't inspect or modify the traffic, it's not mine, I'm just "renting" perpetually.

Now for the last 7 years I've been more on the dev side of devops, but I can't remember seeing any network appliances being sold for "enterprise" that didn't have an option to enter a proxy for updates, or cloud services, or whatever it needed to phone home for. This could also be that for the last 15 years I've been working for very large non-tech companies that just wouldn't budge on giving a random network appliance direct access to the internet. So the vendors either added the feature for us, or those features didn't work, and maybe we didn't buy it at all. I just did some quick searches and found proxy settings for F5, juniper, cisco, infoblox, and tufin appliances.

Now if the problem is that they have an option to configure a proxy, but don't trust your internal cert for the proxy itself, then the answer is to use a publicly signed cert on your internal proxy server. Even if it's just a let's encrypt one. It does feel dirty using public namespace for something that is only ever internal, but it works fine.

Decrypting traffic arguably allows you to rewrite traffic as part of the proxy process. So (for example) Samsung's advertising by using DNS over HTTPS would be defeated. As nice as it would be I don't really see IOT manufacturers offering up a niche feature that would potentially lessen revenue. Outliers like HN readers aside consumers haven't really revolted over progressively more invasive monitoring via their IOT devices.

From an SSL/TLS perspective you could have an HTTPS proxy on your network proxying all your traffic and impersonating all websites, by using a certificate authority trusted by your devices.

You'd be the sole owner of that root cert / its private key, so you wouldn't have to worry about people in public networks mitm'ing you, but you'd mitm everything on your own network (that is, the traffic coming from devices where you trusted that custom CA).

Unless you want to do that with devices that don't explicitly trust your custom CA, but in that case, it's a feature that this isn't possible.

I know how to do it, you just can't do it unless you control the device. Which for the IoT devices I'm trying to "hack"/reverse engineer the API server like Nest, I cant change anything.

But this is a decision you in effect made, those devices needn't even use TLS, they could use Jim Bob's secret MITM-defeating crypto thing and you can't "hack" that either, you bought devices that are resistant to being reverse engineered.

Governments could legislate of course, there could be a rule that means you can't sell Nest because it doesn't have some "hack" feature. After all the EU mandated people provide a sane charge port on phones and it more or less worked, even if you don't live in the EU your phone likely no longer has some arbitrary custom charge port on it, and you can buy replacement chargers or keep the one from a previous phone, reducing waste.

You also could legislate from the far end, for public services, you could say if I own a device using my credentials to do stuff, I get to have the agreed key material so that I can decrypt messages after the fact. That's a huge pain to implement, most of the ways to do it make everybody's security a bit worse, but it isn't technically impossible.

But technologically we can't change the mathematics of the encryption, Diffie-Hellman works, two parties can agree a secret (which in effect becomes a symmetric key) without either of them uttering the secret.

"decision you in effect made" == "you were basically coerced"

What IoT devices have you been coerced into allowing onto your network?

Exactly. Even Android now with Android 12 is difficult. I spent a bunch of time I didn't really have trying to get my Android to proxy through my Burp suite, which is something I used to be able to do easily. But now it's extremely difficult, and the use of HSTS makes it basically impossible for all big tech sites, especially Google owned, but even HN. All I want is to be able to see who my device is using my network to call, and what they're saying. It used to be that the tech was there to serve me, but now it seems the other way around.

"All I want is to be able to see who my device is using my network to call, and what they're saying."

Instead of Burp, I use Haproxy running in Termux. I have not run into any problems with HSTS.

The part about being able to see "what they're saying" is interesting. Assuming the people running "tech" companies wants users to trust them, the user's data/information being transferred to the "tech" company should be available to the user. After all, it is the user's network, it is the user's computer and it is the user's data/information, not the "tech" company's. It is also the the user's bandwidth. "Tech" companies do not pay the costs of transferring the data/information, users foot the bill.

A cautious user could decide that if the "tech" company will not share with her the deatils of the user's data/information being sent to the mothership, then she will just block the requests. In that case, installing NetGuard available from F-Droid will allow the user to list and optionally block DNS/HTTP requests. It falls short of instecting the contents of the requests, but it does allow the user to export lists of all DNS/HTTP requests. It can also export a list of all domains that the user allows to be resolved. This is useful for determining what domains apps need to access and creating whitelists.

The simple(ish) solution is to install a firewall on your network and configure it to MITM all the HTTPS traffic and simply deny anything it can't MITM. I understand this to be standard practice in many corporate environments. The device/app/whatever either trusts your CA or the traffic gets blocked. I don't bother to do so myself but I think it's a good position to take. My network is mine.

That is what I was doing (and still do). I wanted to learn how to do it on a smaller form factor device. For example, imagine a firewall in the form factor of a smartphone. I have seen at least one other person on HN who says they are using a pocket-sized gateway/firewall.

Intentional "MITM" is obviously a common topic that gets discussed on HN from time to time. What bothers me about HN commenters on this topic is that they signal that they are familiar with what needs to be done, i.e., to MITM their own traffic, and want us to know they understand how one "could" do it, however it is unclear that they themselves are actually doing it. I sometimes wonder if this is tacit approval of TLS being used (strategically perhaps) to lock users out of monitoring their own computers and computer networks. Hopefully there are many HN readers who do monitor ther own traffic but are generally not commenting about it.

Sysadmins, private network operators, and device owners are just a subset of malicious attackers and "evil" nation states.

The average user doesn't know to care. They keep buying this garbage and, in turn, financing the lobby against consumer protection laws for embedded devices.

It makes me want to give up on technology and go live in a shack in the woods.

Is it any comfort that we might be staring down financial collapse that crushes consumer electronics?

Doubtful. Collapse and revolution are just tools of the market. The market which now has eyes and ears in your trashcan.

You don't own the communication as a network operator, unless you can also prove that you have control over the device. IoT devices sometime rely on sending sensitive data to their backends. This data may include API keys, client authentication secrets and such. By having access to that communication, you may be able to spoof identity of the IoT device from a PC or a hacked-device. Not very desirable from IoT vendor point of view.

> You don't own the communication as a network operator, unless you can also prove that you have control over the device.

Sure, but devices that you don't have control over shouldn't be permitted to communicate anywhere at all ever.

> IoT devices sometime rely on sending sensitive data to their backends.

Such as literally anything.

> This data may include API keys, client authentication secrets and such.

Yeah that sounds like sensitive data alright. It's all data that the owner of the device should have but often do not because of misplaced corporate greed.

> By having access to that communication, you may be able to spoof identity of the IoT device from a PC or a hacked-device.

So what? If I own the device and it's on my network then I have every right to everything on the device.

> Not very desirable from IoT vendor point of view.

And that right there is exactly the problem. IoT vendors don't want to really give up ownership of something they've sold.

There can be legitimate reasons for restricting access to IoT traffic, and data may not be user data at all. Imagine an IoT light bulb, that can be controlled via voice assistants and IoT vendor's mobile app. This bulb talks to some services. IoT vendor developed those services just for this product, If an adversary/hacker can see the communication protocol, and API keys, he can use same service to control, his own ESP32 project. He can even abuse the service in ways it was not originally intended. For example a competitor can ship counterfeit products, that work with your service.

This idea is used in some "corporate security" tools. For example companies which use ZScaler install zscaler certificates on employees machines and use ZScaler servers as MitM kind of proxy which allows them to scan all traffic, including SSL/TLS.

I've been running a MITM proxy for many years to do filtering (including adblocking). Fortunately I don't have any IoT to worry about.

I can't tell if you know this or not, but TFA is not about MitM. Wireshark isn't acting as a proxy, it's using keys saved by the client application to decrypt the traffic after the fact.

I know, it's just related to something I was trying to do recently (MitM some IoT devices to inspect the traffic with Wireshark). I wish there was some way to get the PMS on remote devices via a proxy or configuration option.

Arguably this is still MitM since Wireshark needs to set up a network tap. MitM does not necessarily require modifying the data. This would work just as well captured at the switch or ISP level if you still had access to the keys.

No, that's not what MitM is. The machine needs to rely traffic whether it modifies it or not. There's nothing in the middle here, clearly. It's saving packets and decrypting later, this is just eavesdropping, it's like a wire tap.

Easy. Set up your own PKI and trust it. Keep the private key safe and nobody else can use it.

Only problem is that on Android 7 and above apps can choose to ignore any added certificates in the user store and the system store is off limits. Only way you can do it is by managing the phone in COBO or COPE mode with Android Enterprise. Which requires an expensive MDM system and a factory reset to enroll it.

Also the app can do certificate pinning which is even worse to get around. You'll have to modify the actual app.

I wouldn't be surprised if this rigmarole was the result of app development companies trying to keep secret how much data they exfiltrate.

You don't need a factory reset to enrol in MDM, you can create a managed "work" profile and leave the unmanaged user profile alone.

That is correct, you can enroll into BYOD / regular work profile mode without reset. But for COPE mode and COBO mode you must factory reset to enroll.. because they are triggered in the startup Wizard by tapping 5 times on the background and then scanning the QR code (or by entering the token at the Google setup screen). These are fully managed modes.

And those modes are the only ones that allow you to push certificates to the system store. Normal BYOD work profile mode doesn't allow you to do that anymore. They only go into the user store where apps are free to ignore them - since Android 7. This mode is only meant for user owned phones and thus Google wants to limit what you as an admin can do. And manually you can't add them anymore either.

So that's why I mentioned the factory reset there.. It really is needed if you want to add root CAs to the system store.

(I managed 60.000 phones, 40k of which Android until a couple months ago :)

That makes perfect sense though, because presumably the system store would affect all profiles. Sure the app can ignore the user CA store, but it can ignore the system one too and just bring its own CA roots.

You certainly still get the "Your organisation may monitor network traffic in your work profile" warning on Android 12 when there's a CA added to the work profile.

> managing the phone in COBO or COPE mode with Android Enterprise

Better solution IMO is flash Lineage, preferably a variant with signature spoofing, and refuse to use proprietary garbage that pins certificates. If that means no streaming services or whatever it is then fine, I'll just pirate instead thank you very much. Why should I put money towards supporting things that erode my rights?

True. I do the same. But I like to do some research on security and privacy (this is my job now) and for that it's very handy to be able to intercept SSL traffic from mainstream apps.

Are you me? This perfectly describe something I have been wanting to do recently. I really do wish that everything that connect to "MY" network I can inspect freely.

This stupid wireless lightbulb I have want to ping home to its manufacturer and I have no way to stop it. I had to go through the trouble of putting it on its own network and blocking it from everything. Very annoying. In the end I returned it because I couldn't trust it.

Neat, I didn't know about SSLKEYLOGFILE, seems like a great way to debug apps that support it without installing custom root certificates.

How would one go about doing that for Android apps? I guess android apps don't respect that flag?

As long as there's no certificate pinning implemented in the apps (which can be defeated with some effort) I would go with a different approach and use mitmproxy. See: https://www.trickster.dev/post/setting-up-mitmproxy-with-and...

Can use frida-sslkeylog to extract the keys on Android, and put it into Wireshark: https://github.com/saleemrashid/frida-sslkeylog

This require a rooted phone. Or you can patch the app with objection, so you don’t need the root: https://github.com/sensepost/objection/wiki/Patching-Android...

Can download the APK from places like https://apkpure.com/

apkpure.com has a track record of serving malicious APKs.

If you run it in a sandbox environment it's probably okay-ish, but if you already have the app installed from a trusted source on your phone, you can grab the APK file with `adb pull`.

Did not know that. Thanks

Yeah I've had good success with patching apps. If the app complains the phone is an emulator, just patch the byte code that is checking for emulators too :)

Unfortunately I could not patch any apps yet. They open with a blank screen and crash after a few seconds. objection can't connect to the server. My phone doesn't seem to be easily rootable, so I'm kinda stuck for a while.

You can add a self-signed trusted root ca, and set a system proxy in Android and apps will (usually) use that, although the Google and other big-tech apps will certificate pin and/or use HSTS, so they won't trust your cert. Also newer versions of Android make it a huge PITA.

Pop Quiz: Who remembers Wireshark’s original name?

What a fun tool for over two decades!

ethersomething??

> Pop Quiz: Who remembers Wireshark’s original name?

That's a real question for the ethernet people ;)

Hehe, I've never used it when it was called ethereal.

Love it! :)

I still remember how flashed I was seeing the byte-by-byte analysis of a network package for the first time -- it was a real eye opening moment for me.

Hate to be that guy, but network packet. But yeah, it's quite the revelation especially when you realize other people can also sniff the network and capture the unencrypted traffic. Remember thinking nobody would waste so much time and energy. How young and naive I was back then.

Hate to be that guy, but Wireshark captures and decodes protocols starting at the data layer. So in the most common case what you see is the decode of the ethernet frame which in the majority of cases but not all includes a network packet inside it which Wireshark further decodes. If you were capturing from an ATM network you would see ATM cells being decoded. For some protocols the data layer unit is also referred to as a packet in which case you would see the data layer packet which includes the network packet. But yeah, it is pretty cool no matter what you want to call things.

I believe the parent was pointing out a typo, package should have been parent. I don’t think he was saying that it only has capabilities from the network layer onwards.

The original domain name is crypto bait now.

Wireshark and its predecessors have made it easy to visualize how protocols actually work, and peel back the magic that makes the internet work.

Between the ability to "follow TCP stream" or the "conversation" views between two hosts, wireshark has helped connect the abstract and the observed / real behavior of networks. Which seems to make the difference for people trying to get beyond the basics of networking in workshops / training sessions.

The built-in protocol dissectors are also a big help for teaching.

Some common situations won't work well out of the box (multi interface PCAPs), or I need to read the manual again... but with a bit of scripting / light processing of the PCAP it works very well.

also very useful for reverse engineering devices using a custom protocol over USB bulk data (in conjunction with a Windows/OSX VM and the usbmon kernel module).

dude- wireshark was the name from a really, really long time ago.. that is real trivia

Whether it was a "really, really" long time ago or not is dependent on how old you are.

I remember replacing Novell LANalyzer for Windows with Ethereal, so it doesn't seem that long ago to me. Of course I also remember replacing The Sniffer from Network General which ran under DOS with LANalyzer for Windows.

Initially I dismissed this comment as I remember it changing name and I'm still in my twenties, but it turns out that it got renamed the same year I actually switched to GNU/Linux, so I was very close to missing the original name :)

Right after TCP/IP class around 2009, I haven't touched it since. Zero gas Ethereal!?

Ethereal. I was working a job where I had to use it daily when they changed the name to Wireshark. Around 2006-2007 IIRC. I never got used to the new name and left that job shortly afterwards.

Wireshark is an objectively better name though, it's for the best.

But real “hackers” used Ettercap instead!

Real hackers use tshark :) [1] I try to get my students doing everything from commandline, creating pcap files. When they finally see the GUI of Wireshark they think they died and went to heaven.

[1] It's actually the capture part of Wireshark

Apologies for "well, actually"-ing here, but the capture part of Wireshark is dumpcap and the various extcap utilities (sshdump, ciscodump, etc). TShark is the command line interface to Wireshark's dissection engine, which, as you point out can be a really useful thing to have. It ships with a bunch of other command line utilities as well, such as editcap, mergecap, and capinfos: https://www.wireshark.org/docs/man-pages/

> sshdump

Thanks for that handy one I'd missed until now!

Ethereal. But I thought it was pronounced Ether Real. I'm not sure I'd encountered the word ethereal in any other context at that point (pre-teen/young teen).

I still call it Ethereal sometimes. For some reason the name change really confused me and it took ages to get used to the new name.

It's a wonderful tool and I'm very happy our network technology teacher showed us this back in high school.

For academic purposes (in grad-school as part of my networks class), I have used T-Shark before to monitor packets.

(Shameless self-promotion) I had written a blog post about it back then. Here goes [1]

[1] https://arjkb.gitlab.io/post/2017-11-27-capture-packets-usin...

I prefer mitmproxy

You can have both! mitmproxy will dump TLS master secrets for any app: https://docs.mitmproxy.org/stable/howto-wireshark-tls/

Something like mitmproxy can be easier, but the Wireshark approach has the advantage of not changing the wire traffic. That is, some problems could go away when you introduce a proxy.

How does this work with perfect forward secrecy, does curl save all the keys? Or is it disabled?

First step the article gives: `export SSLKEYLOGFILE=~/.sslkeyfile`

This will tell curl, among many other applications, to save the necessary keys.

Yes, I saw this and tried that, but when running curl only saw one key. I just expected more.

I can't figure out how to do virtualized android devices? any tips.

Today I got everything working using BlueStacks.

You still have to root the emulated device as you would with a regular device. But it works

how would this work for custom applications? Say I have a C program which does openssl handshakes inside. How can we implement the same SSLKEYLOGFILE functionality in it

This blog is awful to read. Wireshark will decrypt your ssl traffic automatic if you tell it to.

With which keys?

confidently incorrect

How would one go about doing this? I've never been able to get it to do that.

Wireshark doesn't come with a built in way to extract the dynamically exhanged TLS keys from applications and use that to decrypt the traffic, so no it does not do what you claim.

(Sorry folks, was replying to the (at the time) only other comment about wireshark automatically doing decryption - edit)

This doesn't work if you use DH, and you should be using DH.

To work with DH, you need to run your browser with a SSL key logfile, as discussed here:

https://security.stackexchange.com/questions/35639/decryptin...

That's exactly what this article covers.

Reply on news.ycombinator.com