Hacker News Reader

Comment by css | Hacker News Reader

Because the hexadecimal parsing rule [0] happens before [1] the rule that parses names [2]:

    % cat 0xfor.py
    0xfor 1
    % python -m tokenize -e '0xfor.py'
    0,0-0,0:            ENCODING       'utf-8'
    1,0-1,3:            NUMBER         '0xf'
    1,3-1,5:            NAME           'or'
    1,6-1,7:            NUMBER         '1'
    1,7-1,8:            NEWLINE        '\n'
    2,0-2,0:            ENDMARKER      ''

Also, literals have to be evaluated before names, otherwise you could overwrite them:

    >>> 0xzzz = 1
    File "<stdin>", line 1
        0xzzz = 1
        ^
    SyntaxError: invalid hexadecimal literal
    >>> 0xf = 1
    File "<stdin>", line 1
        0xf = 1
        ^
    SyntaxError: cannot assign to literal

[0]: https://docs.python.org/3/library/stdtypes.html#float.fromhe...

[1]: https://github.com/python/cpython/blob/5ce227f3a767e6e44e7c4...

[2]: https://github.com/python/cpython/blob/5ce227f3a767e6e44e7c4...

View on HN

You seem to suggest some sort of differences in rule priorities. I don't think they are prioritized. It's just that Python reads left to right, and the first thing it sees looks like the start of a number, so it starts parsing a number. It doesn't reason like "I have to parse like this, otherwise you could overwrite literals".

I’m suggesting that it has to happen in a specific order, which it does. Regardless of the direction the tokenization process reads, the rules can’t be evaluated concurrently.

> the first thing it sees looks like the start of a number

Yes, because it checks if the token is a number before checking if it is a name.

Names cannot start with a decimal digit, so 0xfor can never be a name. It doesn't matter in which order the rules are checked.

> 0xfor can never be a name

But to know if a token is a name, it has to check, and that check never happens because the tokenizer yields when the number case hits. You can attach a debugger and see for yourself.

> It doesn't matter in which order the rules are checked.

Nowhere did I claim otherwise. The rules _do_ happen in a deterministic order, which I posted the source code for.

>> It doesn't matter in which order the rules are checked.

> Nowhere did I claim otherwise.

You claimed otherwise when you wrote "Because the hexadecimal parsing rule [0] happens before [1] the rule that parses names [2]", and when you wrote "literals have to be evaluated before names", and when you wrote "the rules can’t be evaluated concurrently". That is, you claimed otherwise in every single one of your posts in this subthread.

The check whether a token is a number happens before the check whether a token is a name, as evinced by both the source code and a trivial debugger session.

You are conflating the fact that there is an order to the rules with a strawman that the rules _must_ be evaluated in a specific order. All of the quotes you pulled are facts (Python checks if a token is a number before it checks if it is a name, Python checks if a token is a literal before it checks if it is a name, and the rules are not evaluated concurrently), but your strawman is making a different claim.

> a strawman that the rules _must_ be evaluated in a specific order.

I'm not the one who wrote "literals have to be evaluated before names".

> rules are not evaluated concurrently

Strawman yourself. I didn't say they were evaluated concurrently, I said that your claim that they can't be evaluated concurrently was wrong. They could be, because they are disjoint, and regardless of order only one of them can match at input starting with a '0' character.

Anyway, I'm done here.

Reply on news.ycombinator.com