Hacker News Reader

Comment by layer8 | Hacker News Reader

The author may have missed that lexing C is actually context-sensitive, i.e. you need a symbol table: https://en.wikipedia.org/wiki/Lexer_hack

Of course, for syntax highlighting this is only relevant if you want to highlight the multiplication operator differently from the dereferencing operator, or declarations differently from expressions.

More generally, however, I find it useful to highlight (say) types differently from variables or functions, which in some (most?) popular languages requires full parsing and symbol table information. Some IDEs therefore implement two levels of syntax highlighting, a basic one that only requires lexical information, and an extended one that kicks in when full grammar and type information becomes available.

View on HN

> this is only relevant if you want to highlight the multiplication operator differently from the dereferencing operator

Can you mention one editor which does that?

I could be stretching the definition of "does" but the newfound(?) tree-sitter support in Emacs[1] I believe would allow that since it for sure understands the distinction but I don't possess enough font-lock ninjary to actually, for real, bind a different color to the distinct usages

  /* given foo.c */
  int main() {
    int a, *b;
    a = 5 * 10;
    b = &a;
    printf("a is %d\n", *b);
  }

and then M-x c-ts-mode followed by navigating to each * and invoking M-x treesit-inspect-node-at-point in turn produces, respectively:

  (declaration declarator: (pointer_declarator "*"))

  right: (binary_expression operator: "*")

  arguments: (argument_list (pointer_expression operator: "*"))

1: https://www.emacswiki.org/emacs/Tree-sitter

These examples are unambiguous. Try with something more spicy like

  return (A)*(B);

which depends on A being a type or a variable.

I don't think they implied there is. The sentence you quoted is essentially "this is relevant for their article about weird lexical syntax, but (almost definitely) not relevant to their original problem of syntax highlighting".

hey

I don't think the lexer hack is relevant in this instance. The lexer hack just refers to the ambiguity of `A * B` and whether that should be parsed as a variable declaration or an expression. If you're building a syntax tree, then this matters, but AFAICT all the author needs is a sequence of tokens and not a syntax tree. Maybe "parser hack" would be a better name for it.

I’d be shocked if jart didn’t know this, but it seems unlikely that an LLM would generate one of these most vexing parses, unless explicitly asked

I think you're thinking of something different to the issue in the parent comment. The most vexing parse is, as the name suggests, a problem at the parsing stage rather than the earlier lexing phase. Unlike the referenced lexing problem, it does't require any hack for compilers to deal with it. That's because it's not really a problem for the compiler; it's humans that find it surprising.

Given all the things that were new to the author in the article, I wouldn’t be shocked at all. There’s just a huge number of things to know, or to have come across.

Justine is proficient in C, she is the author of a libc (cosmopolitan) among other things, like Actually Portable Executables [1].

I would expect her to know C quite well, and that's probably an understatement.

[1] https://justine.lol/ape.html

Reply on news.ycombinator.com