Hacker News Reader

Comment by yjftsjthsd-h | Hacker News Reader

Do you mean to say that your biggest frustration with HTTPS on .internal is that it requires a private certificate authority? Because I'm running plain HTTP to .internal sites and it works fine.

View on HN

Try running anything more complicated than a plain and basic web server! See what happens if you attempt to serve something that browsers deem to require a mandatory "Secure Context", so they will reject running it when using HTTP.

For example, you won't be able to run internal videocalls (no access to webcams!), or a web page able to scan QR codes.

Here's the full list:

* https://developer.mozilla.org/en-US/docs/Web/Security/Secure...

A true hassle for internal testing between hosts, to be honest. I just cannot run an in-development video app on my PC and connect from a phone or laptop to do some testing, without first worrying about certs at a point in development where they are superfluous and a loss of time.

localhost is a secure context. so.. presumably we're just waiting for .internal to be added to the white list.

Unlikely. Localhost can be a secure context because localhost traffic doesn't leave your local machine; .internal names have no guarantees about where they go (not inconceivable that some particularly "creative" admin might have .internal names that resolve to something on the public internet).

One can resolve "localhost" (even via an upstream resolver) to an arbitrary IP address. At least on my Linux system "localhost" only seems to be specially treated by systemd-resolved (with a cursory attempt I didn't succeed in getting it to use an upstream resolver for it).

So it's not a rock-hard guarantee that traffic to localhost never leaves your system. It would be unconventional and uncommon for it to, though, except for the likes of us who like to ssh-tunnel all kinds of things on our loopback interfaces :-)

The sweet spot of security vs convenience, in the case of browsers and awarding "secure origin status" for .internal, could perhaps be on a dynamic case by case basis at connect time:

- check if it's using a self-signed cert - offer TOFU procedure if so - if not, verify as usual

Maaaaybe check whether the connection is to an RFC1918 private range address as well. Maybe. It would break proxying and tunneling. But perhaps that'd be a good thing.

This would just be for browsers, for the single purpose of enabling things like serviceworkers and other "secure origin"-only features, on this new .internal domain.

> One can resolve "localhost" (even via an upstream resolver) to an arbitrary IP address. At least on my Linux system "localhost" only seems to be specially treated by systemd-resolved (with a cursory attempt I didn't succeed in getting it to use an upstream resolver for it).

The secure context spec [1] addresses this-- localhost should only be considered potentially trustworthy if the agent complies with specific name resolution rules to guarantee that it never resolves to anything except the host's loopback interface.

[1] https://w3c.github.io/webappsec-secure-contexts/#localhost

localhost is pretty special in that it's like the only domain typically defined in a default /etc/hosts.

No, you can't. Besides the /etc/hosts point mentioned in the sibling, localhost is often hard-coded to use 127.0.0.1 without doing an actual DNS lookup.

Years back I ran into a issue at work because somebody named their computer "localhost" on a network with automatic DNS registration. Because of DNS search path configuration it would resolve. So, "localhost" ended up resolving to something other than an address on 127.0.0.0/8! It was a fun discovery and fixed soon after I reported it.

No. The concept of a DMZ died decades ago. You could still be MITM within your company intranet. Any system designed these days should follow zero-trust principles.

Sure, but people still need to test things, and HTTPS greatly complicates things. Browsers' refusal to make it poasible to run anything unencrypted when you know what you're doing is extremely annoying, and has caused significant losses of productivity throughout the industry.

If they're so worried about users getting duped to activate the insecure mode, they could at least make it a compiler option and provide an entirely separate download in a separate place.

Also, don't get me started on HSTS and HSTS preloading making it impossible to inspect your own traffic with entities like Google. It's shameful that Firefox is even more strict about this idiocy than Chrome.

Indeed. Nothing enrages me more as a user when my browser refuses to load a page and doesn't give me any way to override it.

Whose computer is this? I guess the machine I purchased doesn't belong to me, but instead belongs to the developer of the browser, who has absolutely no idea what I'm trying to do, what my background is and qualifications and what my needs are? It seems absurd to give that person the ultimate say over me on my system, especially if they're going to give me some BS about protecting me from myself for my own good or something like that. Yet, that is clearly the direction things are headed.

To inspect your own traffic you can use SSLKEYLOGFILE and then load it into wireshark.

Most apps don't support SSLKEYLOGFILE. OpenSSL, the most popular TLS library, doesn't support it.

OpenSSL does provide a callback mechanism to allow for key logging, but the application does have to opt in. IIRC, at least Curl does support it by default.

Yes, there are ways to do keylogging with OpenSSL. Even if the app doesn't support it, you can do it with LD_PRELOAD and external libraries that call those callbacks. But it's still a whole lot more work than just an env var, and then just not having all these problems in the first place, by avoiding unnecessary encryption. And it probably won't work on mobile.

> The concept of a DMZ died decades ago.

That is very much not true. Most corporate networks I've ever been on trust the internal network. Whether or not you think they should, they do.

Doesn't matter for mixed content, like e.g. when you run a client-side only app that happens to be loaded from a public domain over HTTPS, and want it to call out to an API endpoint running locally. HTTP won't fly. And good luck reverse-proxying it without a public CA cert either.

A lot of services default to HTTPS. For instance, try setting up an internal Gitlab instance with runners, pipelines, and package/container registries that actually works. It's an absolute nightmare, and some things outright won't work. And if you want to pull images from HTTP registries with Docker, you have enable that on every instance for each registry separately. You'd be better off registering a real domain, using Let's Encrypt with the DNS challenge, and setting up an internal DNS for your services. That is literally an order of magnitude less work than setting up HTTP.

I consider HTTPS to be easier to run - you get less trouble in the end.

As mentioned, some browser features are HTTPS only. You get security warnings on HTTP. Many tools now default to HTTPS by default - like newer SQL Server drivers. Dev env must resemble prod very closely so having HTTP in DEV and HTTPS in prod is asking for pain and trouble. It forces you to have some kind of expiration registry/monitoring and renewal procedures. And you happen to go throught dev env first and gain confidence and then prod.

Then there are systems where client certificate is mandatory and you want to familiarize yourself already in dev/test env.

Some systems even need additional configuration to allow OAuth via HTTP and that makes me feel dirty thus I rather not do it. Why do it if PROD won't have HTTP? And if one didn't know such configuration must be done, you'd be troubleshooting that system and figuring out why it doesn't work with my simple setup?

Yeah, we have internal CA set up, so issuing certs are pretty easy and mostly automated and once you go HTTPS all in, you get the experience why/how things work and why they may not and got more experience to troubleshoot HTTPS stuff. You have no choice actually - the world has moved to TLS secured protocols and there is no way around getting yourself familiar with security certificates.

At my first job out of college we built an API and a couple official clients for it. The testing endpoint used self-signed certs so we had to selectively configure clients to support it. Right before product launch we caught that one of our apps was ignoring certificate verification in production too due to a bug. Ever since then I've tried to run publicly valid certificates on all endpoints to eliminate those classes of bugs. I still run into accidentally disabled cert validation doing security audits, it's a common mistake.

There's some every packet shall be encrypted, even in minimal private VPCs lore going on. I'm blaming PCI-DSS.

The big problem with running unencrypted HTTP on a LAN is that it's terribly easy for (most) LANs to be compromised.

Let's start with the obvious; wifi. If you're visiting a company and ask the receptionist for the wifi password you'll likely get it.

Next are eternity ports. Sitting waiting in a meeting room, plug your laptop into the ethernet port and you're in.

And of course it's not just hardware, any software running on any machine makes the LAN just as vulnerable.

Sure, you can design a LAN to be secure. You can make sure there's no way to get onto it. But the -developer- and -network maintainer- are 2 different guys, or more likely different departments. As a developer are you convinced the LAN will be as secure in 10 years as it is today? 5 years? 1 year after that new intern arrives and takes over maintainence 6 weeks in?

What starts out as "minimal private VPC" grows, changes, is fluid. Treating it as secure today is one thing. Trusting it to remain secure 10 years from now is another.

In 99.9% of cases your LAN traffic should be secure. This us the message -developers- need to hear. Don't rely on some other department to secure your system. Do it yourself.

Well said. I used to be of the mindset that if I ran VLANs I could at least segregate the good guys from the evil AliExpress wifi connected toasters. Now everything feels like it could become hostile at any moment and so, on that basis, we all share the same network with shields up as if it were the plain, scary Internet. It feels a lot safer.

I guess my toaster is going to hack my printer someday, but at least it won’t get into my properly-secured laptop that makes no assumptions the local network is “safe”.

For most purposes, when wishing for non-HTTPS, we are talking about development or maybe a staging server of some sort. Maybe if we had state secrets people would be trying to plug into the lan to snoop the traffic, but for 99.99% of developers the traffic between a testing instance and them is the most worthless thing ever. Worst case you might find out what features we will release to the app in 2 weeks. The conflation of “SSL” with “cybersecurity” is unfortunate.

The big issue with encrypted HTTP on the local LAN is that you’re stuck running a certificate authority, ignoring TLS validation, or exposing parts of your network in the name of transparency.

Running certificate authority is one of those a minute to learn, lifetime to master scenarios.

You are often trading “people can sniff my network scenario” to a “compromise the CA someone setup 10 years ago that we don’t touch” scenario.

I agree that setting up a self-signed CA is hard, and harder to keep going.

However DNS challenge allow for you to map an internal address to an IP number. The only real information that leaks is the subnet address of my LAN. And given the choice of that or unencrypted traffic I'll take that all day long.

Also, make sure your TLS certificates are hard-coded/pinned in your application binary. Just like the network, you really cannot trust what is happening on the user's system.

This way you can ensure you as the developer have full control over your applications' network communication; by requiring client certificates issued by a CA you control, you can assert there is no MITM even if a sysadmin, user, or malware tries to install a proxy root CA on the system.

Finally, you can add binary obfuscation / anticheat mechanisms used commonly in video games to ensure that even if someone is familiar with the application in question they cannot alter the certificates your application will accept.

Lots of e.g. mobile banking apps, etc. do this for maximal security guarantees.

In practice pinning tends to be very "best effort", if not outright disadvantageous.

All our apps had to auto-disable pinning less than a year after the build date, because if the user hadn't updated the app by the time we had to renew all our certs... they'd be locked out.

Also dealt with the fallout from a lovely little internet-of-things device that baked cert pinning into the firmware, but after a year on store shelves the clock battery ran out, so they booted up in 1970 and decided the pinned certs wouldn't become valid for ~50 years :D

Pinning is very complex, there is always the chance that you forget to update the pins and perform a denial of service against your own users. At the point where the device itself is compromised, you can’t really assert to anything. Furthermore, there is always the risk that your developers implement pinning incorrectly and introduce a chain validation failure.

Lots of apps use the anticheat/obfuscation mechanisms added by mobile apps are also trivial to bypass using instrumentation - ie frida codeshare. I know you aren’t implying that people should use client side controls to protect an app running on a device and an environment that they control, but in my experience even some technical folk will try and to do this

At some point you have to wonder if your app even matters that much.

The App probably not. The server maybe, the data probably.

This is way overkill, unless you are making a nuclear rocket launch application. If you can not trust the system root CA, the whole internet breaks down.

You will also increase the risk that your already understaffed ops-team messes up and creates even worse exposure or outages, while they are trying to figure out what ssl-keygen does.

Exactly what an NSA puppet account would say!

Don't believe the hype. Remember the smiley from "SSL added and removed here"

https://blog.encrypt.me/2013/11/05/ssl-added-and-removed-her...

This "NSA puppet" is all for encrypting traffic between networks.

;-)

Blame leaked documents from the intelligence services.

No one really bothered until it was revealed that organisations like the NSA were exfiltrating unencrypted internal traffic from companies like Google with programs like PRISM.

Echelon was known about before Google was even a thing. I remember people adding Usenet headers with certain keywords. Wasn’t much, but it was honest work.

Hoping datacenter to datacenter links are secure is how the NSA popped Google.

Turn on crypto, don’t be lazy

Pretty sure state-level actors sniffing datacenter traffic is literally the very last of your security issues.

This kind of theater actively harms your organization's security, not helps it. Do people not do risk analysis anymore?

Taking defense in depth measures like using https on the local network is "theatre" that "actively harms your organization's security"? That seems like an extreme opinion to me.

Picking some reasonable best practices like using https everywhere for the sake of maintaining a good security posture doesn't mean that you're "not doing risk analysis".

I have seen people disabling all cert validation in an application because SSL was simultaneously required and no proper CA was provided for internal things. The net effect was thus that even the traffic going to the internet was no longer validated.

It’s not theatre, it’s real security. And state level actors are absolutely not the only one capable of man in the middle attacks.

You have:

- employees at ISPs

- employees at the hosting company

- accidental network misconfigurations

- one of your own compromised machines now part of a ransomware group

- the port you thought was “just for internal” that a dev now opens for some quick testing from a dev box

Putting anything in open comms is one of the dumbest things you can do as an engineer. Do your job and clean that shit up.

It’s funny you mention risk analysis, plaintext traffic is one of the easiest things to compromise.

NSA sniffs all traffic through various internet choke points in what's known as upstream surveillance. It's not just data center traffic.

https://www.eff.org/pages/upstream-prism

These kind of risks are obvious, real, and extensively documented stuff. I can't imagine why anyone serious about improving security for everyone would want to downplay and ridicule it.

Found the NSA goon.

Seriously, your statement is demonstrably wrong. That's exactly the sort of traffic the NSA actively seeks to exploit.

Caring excessively about certain metrics while neglecting real security is harmful.

Encrypting all network traffic between endpoints does nothing to actively harm security.

That's some "it's okay to keep my finger on the trigger when the gun is unloaded" energy.

If you're on a laptop or phone that switches between WiFi networks then you are potentially spilling session cookies and other data unencrypted onto other networks that also happen to resolve .internal. HTTPS encrypts connections, but it also authenticates servers. The later is important too.

A lot of modern web features now require HTTPS.

Reply on news.ycombinator.com