In Python, `[0xfor x in (1, 2, 3)]` returns `[15]`

Python: Semantic whitespace is the one true way

Also Python: Meh, you can leave out the whitespace, I'll figure it out

Yeah, FORTRAN made some bad choices. In its defense, it first appeared in 1957. There wasn't much language design or compiler writing knowledge back in those days.

When we were in college learning FORTRAN, students would ask for help from the teaching assistants at the computer center. One big problem was FORTRAN allowed horrible spaghetti code because GOTO statements could be used anywhere. It was easy to jump into and out from loops. The TAs had a tough job.

It didn't help when people would deliberately mess with the TAs by asking about code with language features similar to those in the article you linked. Something like IIRC after 47 years:

      DO 15 I = (1, 100)
         purported loop stuff goes here
   15    final statement of loop

That is also not a loop control statement. Instead it is an assignment to a complex number.

Because the hexadecimal parsing rule [0] happens before [1] the rule that parses names [2]:

    % cat 0xfor.py
    0xfor 1
    % python -m tokenize -e '0xfor.py'
    0,0-0,0:            ENCODING       'utf-8'
    1,0-1,3:            NUMBER         '0xf'
    1,3-1,5:            NAME           'or'
    1,6-1,7:            NUMBER         '1'
    1,7-1,8:            NEWLINE        '\n'
    2,0-2,0:            ENDMARKER      ''

Also, literals have to be evaluated before names, otherwise you could overwrite them:

    >>> 0xzzz = 1
    File "<stdin>", line 1
        0xzzz = 1
        ^
    SyntaxError: invalid hexadecimal literal
    >>> 0xf = 1
    File "<stdin>", line 1
        0xf = 1
        ^
    SyntaxError: cannot assign to literal

[0]: https://docs.python.org/3/library/stdtypes.html#float.fromhe...

[1]: https://github.com/python/cpython/blob/5ce227f3a767e6e44e7c4...

[2]: https://github.com/python/cpython/blob/5ce227f3a767e6e44e7c4...

Because `0 xfor` is a syntax error regardless -- nowhere in the grammar are you allowed to separate two non-keywords with a space. 0xf is a number, 0xfo is not, so the lexer breaks the token before the o.

This isn't a bug. It's a slightly loose grammar spec that can surprise python users who don't play golf.

The lexer reads the longest tokens it can (the "maximum munch" rule mentioned in another answer). Once it has read the 0, it has decided to read a number. It will then proceed to read the longest possible number at this position, which is 0xf. This is simpler to specify and implement than the alternative that would require looking ahead to see where to split.

Because of maximal munch.

Enforce consistent formatting, e.g. with black (https://github.com/psf/black) this gets formatted to [0xF or x in (1, 2, 3)].

This issue is caught by a linter and reports, "PEP 8: E225 missing whitespace around operator".

That’s the normal way lexers work, given “tight” token definitions. They continue adding to the current token until an invalid (for the current token type) character is reached, and then begin parsing a new token starting with the “invalid” (but now valid for the next token) character (or the next non-whitespace character).

“1or2” is lexed into “1” (integer) followed by “or2” (identifier), which is valid on the lexer level but then fails on the grammar level.

The lexer unfortunately is a greedy token matcher. As soon as the 0xf "made sense" to it, and 0xfo did not - it did the same thing it would do in the case of something like 0xf+3. Except the + was an `or` in this case which is kosher. There is an idempotent step you can take where extra spaces are added before the AST is formed to make this sort of thing easier. The good news is, with a decent lint / format flow - these sorts are easy to catch.

Probably a lexer bug. "foo"or should never be processed as "foo" and token OR

Not by design, fortunately: https://bugs.python.org/issue43833

It is not strictly speaking a bug, since it works as intended. But it is clearly a counter-intuitive behavior and could be improved. Making 0xfor a syntax error would definitely be an improvement.

But requiring whitespace between all tokens is not an acceptable solution, since "2+2" should work. Always equiring whitespace between alphanumerical characters in different tokens would make sense.

For pretty much the same reason that "1+2" is not parsed as an invalid integer.

Yes, that is basically how C works, though the C standard describes it in a weird way. There is the notion of a “preprocessor number” which starts with a digit and allows any characters that can appear in a number or identifier. At a later step in parsing it is converted to a strict number, which would fail in a case like 0xfor.

This is the kinda thing that helps illustrate just how powerfully dynamic Python is. Almost nothing is statically bound. Super cool if you want to write code that inspects some dynamic state and has special behavior. But the cost is that there's very little static checking capability.

This code "should" result in a NameError or something similar. But resolving that identifier to a name won't happen until we decide to evaluate the right-hand side of the `or`.

Thankfully, most python static checkers make some mostly-sane assumptions and will flag this code as an error, despite the subtle possibility of legally creating this name using some other Python/CPython features.

Type annotations are a great idea but in practice I don't see them used much. Should be great for dev teams working on bigger Python projects.

"2+2" and "2 + 2" is the same in Python, so whitespace is not significant between otherwise unambigous tokens in Python.

Whitespace in Python is significant to indicate statements and block.

This is the kind of thing you could end up with from an automated test case reducer. Start with a typo that causes odd behavior (which might be interpreted as a CPython 'bug') and use something like creduce to whittle it down to the minimal case.

My guess is that they were trying to do soemthing like [x for x in some_list], but had some unfortunate typos.

Other languages don't enforce whitespace around logical operators. Like `0xf||1+2` is perfectly valid javascript.

I can see how this could slip by, it's possible that original python used a non-alphabetical token as the logical operator, then it was swapped out at some later time for or and and.

If you want confusion with a valid expression:

    [0x1for x in (1,2,3)]

looks like it should parse as

    [0x1 for x in (1,2,3)]

but instead parses as

    [0x1f or x in (1,2,3)]

I would have expected the english boolean operators to require whitespace to parse correctly, personally. "0xf||x" makes sense. "0xforx" makes no sense.

You have to recognize the specific case to throw the syntax error. Sounds like they're going to fix that.

I guess people don't remember PL/I very much: PL/I keywords are not reserved words, so it is possible to use them in a program in other than their keyword context. DO DO=1 TO TO BY BY;END=END+1;END; is a valid program.

Though you don't want to throw a syntax error on "1+2"

Why is it "clearly" an error? This syntax has been around forever and is widely used by code golfers, so if it were unintentional, I'm sure the Python devs would've fixed it by now.