Hacker News Reader

Comment by ufo | Hacker News Reader

In cases like this, the parser and lexer can often produce a better error message if they are written to accept a more lax input and then check it for errors.

For example, instead of faithfully implementing the grammar from the specification, allow numbers with leading zeroes and then produce an error for them.

Another situation where this comes up is parsing language keywords. Instead of writing a separate lexer rule for every keyword, write a single rule for keyword-or-identifier, and then use a hash table lookup inside of that to determine if it is a keyword or identifier.

View on HN

This seems more flexible to me somehow.

Using the approach in TFA, can the laser handle tokens that are prefixes of other tokens? Or even tokens that share prefixes?

Lexing "truefalse" as two adjacent tokens "true" and "false" seems slightly crazier than just lexing it as one (meaningless) token.

Usually, if there is a token that is a prefix of another the longer token wins. For example, in Javascript --x is parsed as the decrement operator instead of as two unary "-" operators.

With that rule, the "leading zero" problem in the article should have been caught properly as well. Why wasn't it?

edit: I guess because "01" isn't a valid token.

"--" is also invalid in json, as it doesn't deal with mutations.

The parent was talking about js.

> instead of faithfully implementing the grammar from the specification, allow numbers with leading zeroes and then produce an error for them.

But that's the problem. The tokenizer doesn't talk to the grammar parser (and vice versa)

The tokenizer could understand numbers with leading zeroes and throw an error there.

Something to think about: do languages - not json - interpret -1.2 as [MINUS][NUMBER] or just [NUMBER]. Or how does languages deal with 1.0-2.0 compared to 1.0+-2.0

Exactly. I was referring to the grammar the tokenizer uses for tokenizing numbers. Instead of faithfully copying the JSON spec, you could use a simpler rule that also accepts leading zeros.

As for the leading "-", in languages that have expressions it is common to parse the "-" as a prefix operator because that covers both negative numbers (-1.0) and negating variables (-x).

But in JSON there are no expressions like 1.0-2.0 so the leading "-" is parsed as part of the number.

If you know you are expecting an expression, then it is easy for the parser to understand ‘-‘ as a unary negation if it is the first symbol, otherwise it must be a binary subtraction operator.

    Expr: ‘-‘ Expr {
            return -1 * $2;
            }
        | Expr ‘-‘ Expr {
            return $1 - $2;
            }
        | ‘(‘ Expr ‘)’ {
            return $2;
            }
        | ...

Reply on news.ycombinator.com