Hacker News Reader

Comment by ridiculous_fish | Hacker News Reader

Shells have somewhat unusual parsing requirements. For example "if" is a keyword when used as `if echo` but not `echo if`.

So you either need to implement the lexer hack, or have a "string" token type which is disambiguated by the parser (which is what fish-shell does).

https://en.wikipedia.org/wiki/Lexer_hack

View on HN

That's no problem in many modern lexers as they usually have a "state" so when you encounter "echo" you can switch to a new state and that state may have different token parsing rules. So "if" in the "echo" state could be a string literal whereas it may be a keyword in the initial state.

Lex/Flex takes care of that mostly for you which is one of the benefits of using a well worn lexer generator and not rolling your own.

unless i miss something this should not be an issue. the lexer could parse if as an IF token, and the parser could treat tags as STRING || IF ( || other keywords… )

That seems like it'd get really awkward pretty quickly. "if" isn't unique in this regard; there are about a hundred shell builtins, and all of them can be used as an argument to a command. (For example, "echo then complete command while true history" is a valid shell command consisting entirely of names of builtins, and the only keyword in it is the leading "echo".)

You'd have to `|| EVERY_KEYWORD_IN_LANG`, and then if you ever add a keyword, now you're updating that list there, and anywhere else you've used it.

As the "Lexer hack" Wiki page says, this is only a problem if you're lexing in the first place. If you just parse the grammar, this isn't a problem.

The problem lies with shells extensive usage of barewords. If you could eliminate the requirement for any bareword to be treated as a string then parsing shell code would then become much simpler...but also few people would want to use it because nobody wants to write the following in their interactive shell:

    git "commit" "-am" "message"

    ls "-l"

 etc

Reply on news.ycombinator.com