Hacker News Reader

Comment by css | Hacker News Reader

> Whitespace is needed between two tokens only if their concatenation could otherwise be interpreted as a different token (e.g., ab is one token, but a b is two tokens).

https://docs.python.org/3/reference/lexical_analysis.html#wh...

View on HN

I find that to be a really odd choice when OTOH tabs/indents are required for block structure.

Python: Semantic whitespace is the one true way

Also Python: Meh, you can leave out the whitespace, I'll figure it out

Similarly, Python is supposed to be a language that's easy to learn and easy to read code in. Pythonistas take pride in that, as IMO they should. With that in mind, preserving a counterintuitive behavior that exists solely because of how the lexer was implemented seems inconsistent at best. A language with a more relaxed attitude toward inscrutable brevity - e.g. Perl - can use that excuse. I don't think Python should.

I first learned Python in the 2.6 days, and have been developing in it fairly regularly over the 15 or so years since.

I've never encountered this issue before.

The reason I've never encountered this issue, or even needed to know it was an issue is because I use good development tools. 1) Pycharm highlighted the or in orange, making it clear that it was being interpreted as a keyword, 2) I use a linter (as we all should) which explicitly highlights the lack of whitespace around the token as an issue (PEP 8: E225), and 3) I use a code formatter (which we all should) which, again, highlights this statement as requests that I fix it.

Python has a lot of flaws. This one is down near the bottom of ones that are even really worth talking about.

Started with 1.5 myself. I've never encountered this issue, nor would I expect to. It might be down near the bottom of the list, but it's still a wart and it showed up here for us to talk about. You seemed to think it was worth talking about at even greater length than I did. So do you just believe it's not worth other people expressing opinions?

This showed up on the front page because it's a novelty that even people who have been using the language for over a decade would never encounter.

You're taking my statement way too personally and out of context. You and everyone else can express your opinions all you want.

As a topic in the list of "flaws with Python," I don't think it's a very important or insightful topic because it's not something people really run into. This just doesn't fit there.

This is a novelty, pure and simple. Filed under "weird programming hacks you won't expect" This would be fine. It's like, `([]+![])[+!+[]+!+[]+!+[]][([]+{})]` in javascript. Weird, kind of interesting, but it's not really a "flaw," at least not the sense of something that will burn you unexpectedly.

This is amazing to me and I would never have suspected such a thing.

Reminds me of Fortran.

https://arstechnica.com/civis/viewtopic.php?t=862715

Yeah, FORTRAN made some bad choices. In its defense, it first appeared in 1957. There wasn't much language design or compiler writing knowledge back in those days.

When we were in college learning FORTRAN, students would ask for help from the teaching assistants at the computer center. One big problem was FORTRAN allowed horrible spaghetti code because GOTO statements could be used anywhere. It was easy to jump into and out from loops. The TAs had a tough job.

It didn't help when people would deliberately mess with the TAs by asking about code with language features similar to those in the article you linked. Something like IIRC after 47 years:

      DO 15 I = (1, 100)
         purported loop stuff goes here
   15    final statement of loop

That is also not a loop control statement. Instead it is an assignment to a complex number.

But why "0xf or" rather than "0 xfor" (where xfor is a valid variable name) as long as we only care about splitting between valid tokens?

Because the hexadecimal parsing rule [0] happens before [1] the rule that parses names [2]:

    % cat 0xfor.py
    0xfor 1
    % python -m tokenize -e '0xfor.py'
    0,0-0,0:            ENCODING       'utf-8'
    1,0-1,3:            NUMBER         '0xf'
    1,3-1,5:            NAME           'or'
    1,6-1,7:            NUMBER         '1'
    1,7-1,8:            NEWLINE        '\n'
    2,0-2,0:            ENDMARKER      ''

Also, literals have to be evaluated before names, otherwise you could overwrite them:

    >>> 0xzzz = 1
    File "<stdin>", line 1
        0xzzz = 1
        ^
    SyntaxError: invalid hexadecimal literal
    >>> 0xf = 1
    File "<stdin>", line 1
        0xf = 1
        ^
    SyntaxError: cannot assign to literal

[0]: https://docs.python.org/3/library/stdtypes.html#float.fromhe...

[1]: https://github.com/python/cpython/blob/5ce227f3a767e6e44e7c4...

[2]: https://github.com/python/cpython/blob/5ce227f3a767e6e44e7c4...

You seem to suggest some sort of differences in rule priorities. I don't think they are prioritized. It's just that Python reads left to right, and the first thing it sees looks like the start of a number, so it starts parsing a number. It doesn't reason like "I have to parse like this, otherwise you could overwrite literals".

I’m suggesting that it has to happen in a specific order, which it does. Regardless of the direction the tokenization process reads, the rules can’t be evaluated concurrently.

> the first thing it sees looks like the start of a number

Yes, because it checks if the token is a number before checking if it is a name.

Names cannot start with a decimal digit, so 0xfor can never be a name. It doesn't matter in which order the rules are checked.

> 0xfor can never be a name

But to know if a token is a name, it has to check, and that check never happens because the tokenizer yields when the number case hits. You can attach a debugger and see for yourself.

> It doesn't matter in which order the rules are checked.

Nowhere did I claim otherwise. The rules _do_ happen in a deterministic order, which I posted the source code for.

>> It doesn't matter in which order the rules are checked.

> Nowhere did I claim otherwise.

You claimed otherwise when you wrote "Because the hexadecimal parsing rule [0] happens before [1] the rule that parses names [2]", and when you wrote "literals have to be evaluated before names", and when you wrote "the rules can’t be evaluated concurrently". That is, you claimed otherwise in every single one of your posts in this subthread.

The check whether a token is a number happens before the check whether a token is a name, as evinced by both the source code and a trivial debugger session.

You are conflating the fact that there is an order to the rules with a strawman that the rules _must_ be evaluated in a specific order. All of the quotes you pulled are facts (Python checks if a token is a number before it checks if it is a name, Python checks if a token is a literal before it checks if it is a name, and the rules are not evaluated concurrently), but your strawman is making a different claim.

> a strawman that the rules _must_ be evaluated in a specific order.

I'm not the one who wrote "literals have to be evaluated before names".

> rules are not evaluated concurrently

Strawman yourself. I didn't say they were evaluated concurrently, I said that your claim that they can't be evaluated concurrently was wrong. They could be, because they are disjoint, and regardless of order only one of them can match at input starting with a '0' character.

Anyway, I'm done here.

Because `0 xfor` is a syntax error regardless -- nowhere in the grammar are you allowed to separate two non-keywords with a space. 0xf is a number, 0xfo is not, so the lexer breaks the token before the o.

This isn't a bug. It's a slightly loose grammar spec that can surprise python users who don't play golf.

Because of maximal munch.

The lexer reads the longest tokens it can (the "maximum munch" rule mentioned in another answer). Once it has read the 0, it has decided to read a number. It will then proceed to read the longest possible number at this position, which is 0xf. This is simpler to specify and implement than the alternative that would require looking ahead to see where to split.

Any way to lint this pattern away? Say what you will about the language and lexer complexity, backwards compatibility. This is a code smell if I have ever seen one.

Enforce consistent formatting, e.g. with black (https://github.com/psf/black) this gets formatted to [0xF or x in (1, 2, 3)].

This issue is caught by a linter and reports, "PEP 8: E225 missing whitespace around operator".

Some cases are, but there are still plenty of patterns that (currently) are not. e.g. `1if 1else 2`

Reply on news.ycombinator.com