Automatic cipher suite ordering in Go’s crypto/tls | Hacker News Reader

Automatic cipher suite ordering in Go’s crypto/tls

go.dev

122 pointsFiloSottile47 comments

This seems at odds with the Go 1 compatibility promise. It's trivial to see that the go1.17 upgrade will break any program that relied on 'PreferServerCipherSuites' to run correctly.

The Go 1 compatibility promise (https://golang.org/doc/go1compat) does have an exception for "new security issues that come to light", but this really doesn't seem like it's a new security issue, but rather closer to a defense in depth measure.

I think the correct Go-like way to handle this is to deprecate the entire 'crypto/tls' package and tell users to use the 'v2/crypto/tls' package, akin to how go modules work.

That new package could then not have the 'PreferServerCipherSuites' option at all, which would make it very clear when a user migrates that this change is occurring, and if they relied on it they must make an appropriate change to their code too.

Obviously, the entire world would have to shift over (since a 'v2/crypto/tls.Conn' would not be compatible with a 'crypto/tls.Conn', it's a struct not an interface), but that also seems fair. It's what the go team tells users to do in the case of module versioning, so it seems quite consistent.

I find it surprising there's no mention of backwards compatibility or the go1compat page in the CL or linked issues.

If you encounter such a program that requires PreferServerCipherSuites to run correctly, please open an issue and we’ll look into it as a potential regression.

More in general, it’s unavoidable for packages that implement living protocols like TLS and HTTP to have a higher level concept of backwards compatibility: if the wire behavior of the package were to never change, Go would soon stop working with most of the ecosystem, and wouldn’t implement the likes of HTTP/2 and TLS 1.3. Many standard libraries went that way, and ended up less useful and safe.

I don't know of any programs that will break from this specific change, though I will be unsurprising if some exist.

I agree that the http and tls packages have made frequent changes that are technically exceptions to the go1compat promises, where there are changes that are technically breaking, but are "in spirit" with the intent of the package. I agree that it's unavoidable to change them over time in this way.

The reality of this has been that http-related changes semi-frequently break my programs when I update go versions.

To me, the fact that http and tls are living protocols is an argument for them being separate versioned modules (i.e. golang.orx/x/net/http), not that we should be causing breaking changes and hand-waving it as being an improvement.

Other packages, like os, sync, etc, have remained stable, and I have no issue with those being versioned alongside the go distribution.

For packages like http and tls, that have made frequent breaking changes, it does seem like they should have been deprecated and replaced with either golang.org/x/vX/net/http, or some in-stdlib-versioned-thing.

Why not deprecate crypto/tls and move it out of the stdlib such that users can get benefits of updating the compiler without the risk of code breakage? It also seems like a benefit that users could then consume crypto/tls updates on a faster cadence without needing go point releases.

That’s generally something that might be desirable for a number of reasons but it’s an extremely delicate change because of how stdlib packages import each other, and how types from x/ repos vendored back into the stdlib are not compatible with the original ones. You quickly have to ask questions like “do we let the main module upgrade the version of x/ repos used by the stdlib?” Maybe eventually we’ll find a way to do this cleanly and decide to do it, but large changes like this take time.

This change will break my (and others) scanning the internet and enumerating supported cipher suites and server behavior.

At least other footguns in Go are stuck into unsafe or something similar. Completely removing it will just force people to maintain local forks.

My first intuition for how to do that would be to only enable one cipher suite at a time, which still works with this change. Still, do feel welcome to open an issue, I can’t promise any specific outcome but we’ll look at it.

I’ll mention that we had to ignore the requirements of diagnostic and scanning tools in the past, and you might be better served by a fork like BoGo from the BoringSSL test suite. Fundamentally, what an application wants (“make a secure connection”) is at odds with what a test tool wants (“make a connection potentially broken in one of a thousand different ways, and tell me how it went”).

It's already pretty common for people using Go's (excellent) TLS library for scanners to fork it; it's probably what you should do, because there's lots of opportunity for instrumentation that the library easily hosts but doesn't provide out of the box.

Are you already lacking the ability to scan TLS 1.3 ciphers? Because it sounds like Go never provided the ability to order them.

I vaguely recall go’s safe but unusual TLS cipher suite default being one of v2ray’s early flaws. V2ray was a tunnel designed to look like normal HTTPS traffic to censors, and they were getting very easily sniffed out until they copied the suite preference from a popular browser.

https://github.com/v2ray/discussion/issues/704 is the actual issue; my memory could be wrong.

Go 1 compatibility promise is not about std lib API at large: a set of core APIs, the "standard packages" of the Go library.

Also none of the std libs are versioned, and I don't see any time soon a v2 for it.

> none of the std libs are versioned, and I don't see any time soon a v2 for it.

There is one precedent: the syscall package (https://golang.org/s/go1.4-syscall)

It was deprecated and replaced with the versioned 'golang.org/x/sys/unix' package instead.

The reason the syscall package was changed to an x package was because the x package could be versioned better, and because the stdlib package had to make breaking changes.

I'm more or less just asking why we don't do the same thing for tls, http, and any other packages that, like syscall, have issues that cannot be fixed in a backwards compatible way.

It seems like we've ended up making inconsistent decisions here, which ultimately lead to go programs breaking during go compiler upgrades... which is a thing that isn't supposed to happen.

From that page:

> Go 1 defines two things: first, the specification of the language; and second, the specification of a set of core APIs, the "standard packages" of the Go library.

...

> The APIs may grow, acquiring new packages and features, but not in a way that breaks existing Go 1 code.

So yes, it does apply to the standard library, AFAICT

I don't undertsand it that way, it's not all the std APIs, just some of them ( the core ones ).

Where is this list defined?

If it's not defined specifically, it would seem the intent is the stdlib.

Summary: The Golang team is deciding what ranked order TLS cipher suites should be used in. You are not able to decide what cipher suites to use, the Golang team sets that in the code and will update it as they see fit.

My take on this is that Filippo is taking a heavy handed approach here. This works for the majority of "dev write code fast ship it over the wall" scenarios. But it also falls apart in a couple scenarios:

* Companies/govt agencies which mandate the use of certain algorithms, such as RSA, and both sides of the connection are running Golang code. Now you need to vendor the TLS library to allow for what the company wants.

* If an issue is discovered with an algorithm it would be really nice to be able to turn off the algorithm in production until the issue can be patched. I'm not sure if this is possible with the current set of changes.

_o/ the summary is close, but there's a key detail that might have not come across: you can still enable and disable TLS 1.0-1.2 cipher suites. What we take care of is the preference order in which they are picked.

The two scenarios you present aren’t really affected by this change. However, I want to address the latter: if we fail to make it possible for Go applications to promptly roll out software security fixes, being able to change TLS configuration will be barely a band-aid. I actually can’t remember the last time that would have helped secure a Go application faster.

Thanks for the clarification. Being able to disable cipher suites in TLS <= 1.2 takes care of a lot of my concerns. It does still remain a concern for 1.3.

I still do worry about the lack of ability to disable cipher suites. Does that extend to not being able to say "I will only accept certificates signed with RSA?" Because if you are forced to allow EC certificates then there are a couple points in the past where this could have been an issue ( https://www.cvedetails.com/cve/CVE-2019-6486/ and https://www.cvedetails.com/cve/CVE-2021-3114/ ). Furthermore if we imagine a future where an issue is found with how one of the ciphersuites is implemented (say ChaCha20 is done incorrectly and we only want to allow AES-GCM) then being able to disable the known bad ciphersuite is a great band-aid.

Let's say that you are providing a Go application to a customer. The customer can configure the application with a config file that controls TLS params. If you can configure the TLS ciphersuites and an issue is found the customer turns off the ciphersuite until they can upgrade. The alternative is that you go through a chain of: vendor gets new Golang -> rebuilds application -> does in-house testing to make sure nothing else came in the new Golang that broke stuff -> customer gets new application -> customer does their own in-house testing -> finally it gets deployed. It's a way longer cycle if you rely on upstream to push changes in code.

Why do TLS issues need to get patched faster than any other security vulnerability? Or said another way, why is a slow deploy cycle ok for non-TLS vulnerabilities?

So far I haven’t heard any good reason. TLS issues are not more common than other components, at least in Go. Maybe people are just a little traumatized by how often other TLS stacks broke in the past?

We maintain two Go versions with very conservative patch releases specifically to allow quick deployment of critical and security fixes. If that still doesn’t allow for quick patching of security vulnerabilities, it sounds like a process issue (on our side or on the application side), not a reason to make TLS special.

> why is a slow deploy cycle ok for non-TLS vulnerabilities?

Okay, this is a fair point. I'd say that TLS is something where fixing it faster than the baseline is possible, if it had configurable knobs. In general we should of course strive to have fast deploy cycles for security vulnerabilities of all kinds.

Any government agency that mandates use of specific TLS algorithms (like whitelist) almost certainly requires FIPS cryptography (or classified cryptography), so you won’t be using crypto/TLS anyway.

As a security/cryptography engineer, I love this change: for cryptography, it’s clear that more knobs == more problems.

As a developer, however, I dislike it: more knobs it’s easier to get the code to do what I need it to do.

Personally, I think it’s a good choice by Filippo.

Isn't it mostly banks that want RSA key exchange, so their exfiltration detectors can sniff TLS streams?

Shouldn't matter in this case, anyhow, as they would be disabling ephemeral KEX suites entirely.

That's not so much to do with RSA as it does with Perfect Forward Secrecy. In fact, the banks tried to make their own separate TLS standard instead of using TLS 1.3: https://www.eff.org/deeplinks/2019/02/ets-isnt-tls-and-you-s...

From "Go Crypto and Kubernetes — FIPS 140–2 and FedRAMP Compliance" (2021) https://gokulchandrapr.medium.com/go-crypto-and-kubernetes-f... :

> If a vendor wants to supply cloud-based services to the US Federal Government, then they have to get FedRAMP approval. This certification process covers a whole host of security issues, but is very specific about its requirements on cryptography: usage of FIPS 140–2 validated modules wherever cryptography is needed, these encryption standards protect the cryptographic module from being cracked, altered, or otherwise tampered with. FIPS 140–2 validated encryption is a prerequisite for FedRAMP. [...]

> [...] Go Cryptography and Kubernetes — FIPS 140–2 Kubernetes is a Go project, as are most of the Kubernetes subcomponents and ecosystem. Golang has a crypto standard library, Golang Crypto which fulfills almost all the application crypto needs (TLS stack implementation for HTTPS servers and clients all the way to HMAC or any other primitive that are needed to make signatures to verify hashes, encrypt messages.). Go has made a different choice compared to most languages, which usually come with links or wrappers for OpenSSL or simply don’t provide any cryptography in the standard library (Rust doesn’t have standard library cryptography, JavaScript only has web crypto, Python doesn’t come with a crypto standard library). [...]

> The native go crypto is not FIPS compliant and there are few open proposals to facilitate Go code to meet FIPS requirements. Users can use prominent go compilers/toolsets backed by FIPS validated SSL libraries provided by Google or Redhat which enables Go to bypass the standard library cryptographic routines and instead call into a FIPS 140–2 validated cryptographic library. These toolsets are available as container images, where users can use the same to compile any Go based applications. [...]

> When a RHEL system is booted in FIPS mode, Go will instead call into OpenSSL via a new package that bridges between Go and OpenSSL. This also can be manually enabled by setting `GOLANG_FIPS=1`. The Go Toolset is available as a container image that can be downloaded from Red Hat Container Registry. Red Hat mentions that this as a new feature built on top of existing upstream work (BoringSSL). [...]

> To be FIPS 140–2 compliant, the module must use FIPS 140–2 complaint algorithms, ciphers, key establishment methods, and other protection profiles.

> FIPS-approved algorithms do change at times; not extremely frequently, but more often than they come out with a new version of FIPS 140. [...]

> Some of the fundamental requirements (not limited to) are as follows:

> [...] Support for TLS 1.0 and TLS 1.1 is now deprecated (only allowed in certain cases). TLS 1.3 is the preferred option, while TLS 1.2 is only tolerated.

> [...] DSA/RSA/ECDSA are only approved for key generation/signature.

> [...] The 0-RTT option in TLS 1.3 should be avoided.

Was there lag between the release of TLS 1.3 and an updated release of FIPS 140? @18f @DefenseDigital Can those systems be upgraded as easily?

NIST just moves slowly is all.

Common Criteria (NIAP) still only allows TLSv1.2 and TLSv1.1.

I'll note that this follows the general trend of the Golang Security team trying to remove all the sharp edges on security code so that people don't cut themselves. I won't say that this is a uniformly bad thing, but it does make things harder.

Another change in the same vein was removing official support for x/crypto/openpgp. There were no maintainers, it required too much effort, and pgp is in general a PITA to use. Totally understand. However this means that anyone working with signed git commits or signed rpm/deb packages is now using a forked library due to upstream dropping support.

> but it does make things harder.

Unknowingly doing bad things harder, yes.

It would break in my use case, for sure. We routinely initiate TLS connections to crappy devices which have TLS implementations broken in ways that cause them to fail if you select the wrong cipher suite, even when the server otherwise claims to support it. Our ability to force this on a per-service basis is crucial to our success. We do not have the ability to fix the broken devices nor the luxury of not connecting to them.

Can you open an issue elaborating on this? I’d expect the ability to disable broken cipher suites to be enough to work around this, and that is not changing. (What’s going away is controlling the order in which enabled cipher suites are picked.)

Why yes let's make it more secure by taking power away from the developers, that'll surely work.

Note, that this is for TLS 1.0-2 and TLS 1.3 only has two options since it was implemented. For TLS 1.3...

> In Go 1.16, we started actively preferring ChaCha20Poly1305 cipher suites over AES-GCM on the server when we detect that either the client or the server lacks hardware support for AES-GCM. This is because AES-GCM is hard to implement efficiently and securely without dedicated hardware support (such as the AES-NI and CLMUL instruction sets).

@FiloSottile will ktls make it some day in Go crypto/tls ? (https://blog.filippo.io/playing-with-kernel-tls-in-linux-4-1...)

Great work, thank you.

I am always against taking choice away from developers. The setting of sane, up-to-the-minute defaults, which are always helpful, is not at all what happened here.

And, banal recommendations such as "just fork it then" are either not really serious or not in good faith. Should third-party developers really be encouraged to fork a key security component of the standard library, just because it no longer operates the way it used to?

Personally, I welcome choosing better suites for me, but only as a default, and silently overriding my carefully chosen settings will never be ok.

They did remove the ability to control ordering for TLS 1.2 cipher suites, but that’s only because there’s no logical reason anyone should change it. TLS 1.3 never allowed controlling this because it is unnecessary.

When it comes to a crypto library, you really, really do not want configuration options beyond what is necessary. More configuration options is more opportunities for misconfigurations, and given the knowledge and skills required to fully understand the implications of these decisions as well as the consequences for doing it wrong, it is better to go without.

These are lessons learned from the bad old days of crypto libraries. If the cipher ordering needs to be (objectively) suboptimal in your use case, it sounds like something is broken and you should actually expect to need to vendor that.

The only reason you can even still control TLS 1.2 cipher suites is because of the fact that it might sometimes still be useful for legacy reasons to enable known-broken ciphers. There’s really no other logical reason to keep that option.

There's no reason to be condescending to someone who has real and legitimate concerns about this decision, which was apparently taken without any discussion or input from the broader security or dev community.

I could be wrong, but this overall decision to take the developer out of the loop seems to be assuming that:

1) the Go developers know the current correct, precise ordering better than anyone else. I wonder if DJB or Wietse or Schneier would always agree with all of these (and future) recommendations.

2) that perfect knowledge for this version continues to be correct into the indefinite future.

Next year, you might still be using Go 1.17 in production for various compatibility reasons, and a severe vuln is found in one of the cipher suites. (Let's say ChaCha20, as hard as it might be at this moment to believe that, or maybe heartbleed/crime/beast/etc never happened. Qualys SSL Labs' tests are constantly changing precisely because our knowledge about what is vulnerable changes.)

But, now, it is impossible for you to fix that and take the vulnerable ciphersuite out of contention, short of forking crypto/tls.

I’m not being condescending.

Anyways, if a serious exploit is found in the TLS stack, I would expect what you would want to do is update to the inevitable security release for your current stable version of Go. They do in fact do security releases, so it shouldn’t be a huge problem.

That leaves the only logical reason to need this lever as basically just one thing: you are dealing with broken software and need the cipher suite order to be suboptimal. And yes, I expect someone to need to fork the TLS stack for that, unless it’s common enough that many people would need to do that.

If my TLS stack is not updating in response to new security threats, that really poses a bigger problem to me than not having the levers to try to mitigate them on my own.

The only people who are being encouraged to fork crypto/tls are people building TLS testing tools. Nobody is suggesting that people building ordinary applications should run a forked Go TLS.

Honestly, this sounds like a terrible decision. Preventing bad decisions by ensuring that no dev ever needs to set the cipher order in the first place would be a good idea, but ignoring flags and removing developer control because Google Knows Best is a terrible idea.

It's open source. If you are capable of making such decisions on encryption with founded confidence you can also vendor the library and adapt it.

The reality is most developers (almost every developer I know) won't spend particularly much time on deciding settings for encryption - and honestly if they spent 10x the amount they did doing research they would still only base their opinions on things they find online like blog posts (not mathematics or government whitepapers).

I don't believe the approach is bad. It doesn't take the ability away from you to choose. It just adds a hurdle (vendoring the library), which imo stops a lot of bad decisions from overconfident developers.

[edit]: typo

Sounds like game over for google.

That said, there is trend even outside of google to move away from high levels of configurability in these sort of security sensitive areas. I'd be curious what a system like wireguard lets you do in terms of bit twiddling.

OpenSSL was sort of famous for letting your turn just about every knob you wanted. Not sure that was a ton better.

And I have heard some concern that folks like apple put so much into attack surface on things like imessage (memoji's?) for messages from unknown and untrusted users (ie, new users you aren't texting with can send entire set of stuff imessage supports which is a TON vs just being text only to start). So even in other areas dialing down things might be helpful.

"I'd be curious what a system like wireguard lets you do in terms of bit twiddling."

WireGuard doesn't allow you to do anything in terms of bit twiddling. This is one of the claimed benefits of WireGuard over OpenVPN or IPsec.

To pile on: WireGuard doesn't even have ciphersuites, exactly because of the arguments Filippo makes in this post. You get Curve25519 and ChaPoly, and that's it.

Is it clear that then that as OP says this is a terrible idea?

It seems then in keeping with ideas being followed elsewhere.

Casually (not a security expert) this makes a let of sense to me. If I'm copying my stack overflow code out for my security stuff, maybe have fewer footguns?

Not sure this is a terrible idea then.

You should do whatever Filippo says you should do. Not trying to be flippant. :)

Filippant?

I agree with Google that applications shouldn't normally be setting these parameters manually and I agree with their chosen defaults, but if developers are going through the effort of setting these values they're clearly doing it for a reason.

That reason could be "the API doesn't allow developers to easily access the defaults" but the fix for that would be something completely different.

Even in OpenSSL you don't get that much choice in TLS 1.3 because the number of usable ciphers and key exchange methods has been severely reduced.

> so there is no need to delegate a choice to the application

Until there is a need. Can you see the future? Do you know for a fact that there will never, ever be a single edge case where being able to change the order might be beneficial?

This is nanny-state moral-idealist development. It's one thing to pick defaults. It's another to choose a design decision that literally screws people over, because either you thought you were smarter than everyone, or you didn't care if you screwed anyone over.

  What if a cipher suite turns out to be broken?
  Just like any other vulnerability, it will be fixed in security release for all supported Go versions

Ah. So the millions of statically-compiled Go apps just all need to be recompiled, for every version and platform they were ever released for, and re-downloaded. For supported Go versions.

  Why not make TLS 1.3 cipher suites configurable?
  Conversely, there is no tradeoff to make with TLS 1.3, as all its cipher suites provide strong security

Until a cipher suite turns out to be broken. Like they pointed out in the first question.

Reply on news.ycombinator.com