Because in golang everything is UTF-8. After all, Rob Pike authored both golang and utf-8.
Now, what are the security implications of this?
(beside the chunk thingy)
Now, what are the security implications of this?
(beside the chunk thingy)
That's a terrible reason. HTTP header values are not necessarily UTF-8, and an implementation that blindly assumes they are is broken.
> Now, what are the security implications of this? (beside the chunk thingy)
Many variations of the "chunk thingy" where values considered safe in one context may be unsafe in another context.
Like so many parts of Golang, this is a good example of how "good enough" engineering often really isn't good enough. HTTP is pretty thoroughly specified, and the HTTP stack included in a language should be thoroughly tested against that spec. If you can't achieve that, just leave it out of the stdlib!
Actually, they are explicitly specified as ASCII ("follow the same generic format as that given in Section 3.1 of RFC 822"): https://tools.ietf.org/html/rfc822#section-3.1
They are not, only new headers added after RFC 7320 are ASCII, pre-existing ones may contain high-bit characters which implementations compliant with 7230 version of HTTP/1.1 SHOULD not parse. They stopped short from making it a MUST so that applications compliant with RFC 2616 (where they were to be treated as ISO-8859-1!) was allowed. Or, more simply, they stopped themselves from making RFC 7230 describe new, incompatible HTTP/1.2.
Consider this: all strings are utf-8, so working with pure ASCII is much more work in golang.
It leads to all kind of issues, not only security problems.
The whole "it would be more work" as a justification for not doing things properly is one of the roots of the more general issues with Go I alluded to in my post above.
3) Go specification requires UTF-8 in only one place: the encoding of Go source files string literals.
2) Pure ASCII is a subset of UTF-8, ie. pure ASCII _is_ valid UTF-8. There is _no_ more work and there are _no_ security problems when working with pure ASCII as UTF-8, because that's what it is.
That said, I tweeted this rather than posting a whitepaper because I haven't thoroughly explored the impact. I've shared my guess about what other issues you'll find here: https://news.ycombinator.com/item?id=25858015
And I posted a full whitepaper/presentation on the technique here: https://portswigger.net/research/http-desync-attacks-request...
[0] https://daringfireball.net/linked/2020/05/02/psychic-paper
You would think an old http header is a string. It's not. It looks like a string though :)
Actually its not guaranteed to be UTF-8, only a sequence of bytes with some convenience methods: https://golang.org/ref/spec#String_types
However, the existence of the 'strings' package and the 'range' operator give the impression that all strings are UTF-8.
This is a very, very sloppy way to handle text, and will inevitably lead to a huge number of mistakes and broken software.
Any well-designed modern language really should make a strong separation between raw byte buffers and text strings. They are very different things, and one can definitely not trivially be converted to the other.
> In a go a string is an immutable byte slice assumed to be valid utf8, with associated methods and users which all assume that slice of memory is valid utf8.
Which is in my opinion the worst choice you could make when designing a type. If I'm working in a typed language because of so-called "type safety", if I can't trust my types to actually enforce their guarantees, what can I trust ?
This is not a defence of Go but more of a warning to Go programmers.