When str.lower() is a security vulnerability in Python

"if you have a software system that contains two different implementations of IDNA 2003 processing user input"

Is that a real thing though? Is someone doing that?

That would be a vulnerability in the IDNA filter that they’re responsible for fixing.

I wouldn't call this a "vulnerability", I'd call it "a thing that can potentially turn into a vulnerability, more often it can turn into an obscure bug, and most often it is just a quirk".

In particular, if my corporate security team started just mass-flagging all instances of "str.lower" as "security bugs" I would be having a talk with their manager about their threshold for what constitutes a "security bug". Their job is arguably to be more sensitive to that than most engineers, but not that sensitive. It would be like flagging all instances of string concatenation as a vulnerability... and I say that as the guy who would like to eliminate simple string concatenation from programming languages, already a very extreme position on that operation, because of it being at the root cause of so many vulnerabilities... but simply flagging every use as a "vulnerability" is way too sensitive. A demonstration of the ability to use it to bypass some sort of security barrier is necessary to call any specific instance a "vulnerability".

And string concatenation has caused orders of magnitude more actual, verified vulnerabilities than incorrect case folding has.

I guess that's why you shouldn't be _validating_ things, and then praying them on as-is. "Parsing is validation" is (also) the idea that after validation, your data is in a format that reflects and enforces what's known about it.

"Safe strings" is an example of that idea. Not always possible or practical, but always worth considering if you're doing "validation" as a function.

> A straightforward way to exploit an implementation differential like this is if you have a software system that contains two different implementations of IDNA 2003 processing user input.

Apparently, if this behavior depends on Unicode version, you have a bug over time even if it's a single implementation, *if you process the same input twice*.

The correct fix is to process it once and store the result.

It is situational, but it very much seems like a thing you'd squirrel away and bring out when you find a system where the differential is helpful.

DNS names are a thing where Sales is going to tell the Engineer that they can't issue the customers randomized ASCII names like abxuewrf.my-thing.example because real customers want to write our-brand-name.my-thing.example instead - even though you already know bad guys will choose billing.my-thing.example and name-of-bank.my-thing.example and every other unintended bad choice even before we realise about likelihood of these confusion bugs in software like Python.

I can see how this would be a vulnerability in the context of a security researcher that wants to exaggerate their findings in order to get paid a bounty.

From what I understand the vulnerability is not on lower() but on using different Unicode rules version for it

That sounds to me like a better explanation

IDN is a security flaw masquerading as a standard.

Not all Unicode characters are permitted in DNS, so the parser differential need to exist within the allowed set of Unicode characters. For example, a parser differential on a "space" unicode character won't impact a list of trusted hostnames, since space are not allowed in the first place.

Looking at examples where it could matter, German Sharp S (ß) is allowed in domain names and so a list of trusted hostnames could be fooled if one hostname had a double ss idn-encoded name and the attacker registered a second domain with ß, but there is a caveat here. Looking at how .eu registry handle it, they will not allow two domains to only have the difference between ss and ß, unless they are the same registrant and explicitly request to have both variants (https://eurid.eu/en/knowledge-centre/domain-names-with-speci...). Thus the attack vector will depend on if the registry has protection against it or not.

It's a big deal in anything dealing with permitted/excluded names because once you take the insane C/Posix locale rules into account you end up with nondeterministic comparison results: compare a with b, get a match, another thread changes the locale setting, compare the same a with b, get a non-match. You've also got nontransitive case-changes, e.g. toupper('ı') = 'I', tolower('I') = 'i', 'ı' != 'i'. This means that an attacker can get around any name-based checks by choosing a character set that compares however they want it to, which is why a lot of OSS security projects implement their own guaranteed-deterministic comparison functions, e.g. WireGuard's built-in replacement for the ctype.h macros/functions.

That fragment doesn't mean much in isolation. You've just said that they didn't modify str.lower (because "that would be bonkers") but you've posted a fragment which, for all we know, is part of the str.lower implementation.

Yeah. I can understand the confusion, though. The title claims the issue was in lower(). Though the problem was actually in encode('idna')'s usage of lower().

The article would probably get far fewer clicks if it were named "when encode('idna') is a security vulnerability"

Yeah, that's what I figured, but my worry upon seeing this is "what happens if lower() changes again and people forget to update the list of exceptions?".

Unless they have unit testing on the entire Unicode code space to ensure what they're doing is always identical to 3.2.0.

5.1 adds uppercase ẞ which can fold to either ss or lowercase ß depending on the chosen algorithm.