That’s the normal way lexers work, given “tight” token definitions. They continue adding to the current token until an invalid (for the current token type) character is reached, and then begin parsing a new token starting with the “invalid” (but now valid for the next token) character (or the next non-whitespace character).

“1or2” is lexed into “1” (integer) followed by “or2” (identifier), which is valid on the lexer level but then fails on the grammar level.

The lexer unfortunately is a greedy token matcher. As soon as the 0xf "made sense" to it, and 0xfo did not - it did the same thing it would do in the case of something like 0xf+3. Except the + was an `or` in this case which is kosher. There is an idempotent step you can take where extra spaces are added before the AST is formed to make this sort of thing easier. The good news is, with a decent lint / format flow - these sorts are easy to catch.

Probably a lexer bug. "foo"or should never be processed as "foo" and token OR

Not by design, fortunately: https://bugs.python.org/issue43833

It is not strictly speaking a bug, since it works as intended. But it is clearly a counter-intuitive behavior and could be improved. Making 0xfor a syntax error would definitely be an improvement.

But requiring whitespace between all tokens is not an acceptable solution, since "2+2" should work. Always equiring whitespace between alphanumerical characters in different tokens would make sense.

For pretty much the same reason that "1+2" is not parsed as an invalid integer.

Yes, that is basically how C works, though the C standard describes it in a weird way. There is the notion of a “preprocessor number” which starts with a digit and allows any characters that can appear in a number or identifier. At a later step in parsing it is converted to a strict number, which would fail in a case like 0xfor.

This is the kinda thing that helps illustrate just how powerfully dynamic Python is. Almost nothing is statically bound. Super cool if you want to write code that inspects some dynamic state and has special behavior. But the cost is that there's very little static checking capability.

This code "should" result in a NameError or something similar. But resolving that identifier to a name won't happen until we decide to evaluate the right-hand side of the `or`.

Thankfully, most python static checkers make some mostly-sane assumptions and will flag this code as an error, despite the subtle possibility of legally creating this name using some other Python/CPython features.

Type annotations are a great idea but in practice I don't see them used much. Should be great for dev teams working on bigger Python projects.

"2+2" and "2 + 2" is the same in Python, so whitespace is not significant between otherwise unambigous tokens in Python.

Whitespace in Python is significant to indicate statements and block.

This is the kind of thing you could end up with from an automated test case reducer. Start with a typo that causes odd behavior (which might be interpreted as a CPython 'bug') and use something like creduce to whittle it down to the minimal case.

My guess is that they were trying to do soemthing like [x for x in some_list], but had some unfortunate typos.

Other languages don't enforce whitespace around logical operators. Like `0xf||1+2` is perfectly valid javascript.

I can see how this could slip by, it's possible that original python used a non-alphabetical token as the logical operator, then it was swapped out at some later time for or and and.

If you want confusion with a valid expression:

    [0x1for x in (1,2,3)]

looks like it should parse as

    [0x1 for x in (1,2,3)]

but instead parses as

    [0x1f or x in (1,2,3)]

I would have expected the english boolean operators to require whitespace to parse correctly, personally. "0xf||x" makes sense. "0xforx" makes no sense.