Hacker News Reader

Comment by kristo | Hacker News Reader

Why don’t we just check ASTs in to git instead of text files ?

I think this is the key question, and doing this will enable numerous interface improvements for people working with software (Eg: separation of content and presents means that everybody can use their favorite view — three-space-tabs, extra-long lines, whatever).

The challenge is that version control on text files (list of lines) is fairly straight forward and language agnostic, while OTOH the VCS would have to run after the language parser processes on the text file, and then be able to interface with the AST generated by the parser (for multiple languages). I presume this would become easy if different languages have the same parsing setup, or we have language/project-specific VCS tools. While the answers are not obvious (to me, at least) this definitely seems like the right direction to move in, instead of getting hung up on plain text and formatting issues.

Storing an AST instead of text might be a way to eliminate parsing ambiguity of angle brackets for generic arguments (as opposed to comparisons). It might still be visually ambiguous, but I don't think it's a problem in practice (if the display forces whitespace around comparisons)?

There are some cases where formatters like Prettier don't have one canonical representation for an AST, in particular behavior around line breaks (multiple empty lines are allowed, a line break after an "{" opening brace in an object literal will force it to span multiple lines, etc)

But yeah, otherwise everyone could just have their own preferred formatter/settings and we could commit whatever representation is most optimal for diffing to version control (e.x. multi-line object literals, trailing commas in arrays/args, etc).

You could use something more compact like a binary AST but for compatibility with "legacy" tools that deal with text I think it would be better to use a more verbose text-based representation.

I believe unison does something along those lines:

https://www.unisonweb.org/

I believe that's called "lisp".

Which lisps don't use text files for source code?

I kind of like the ability to have comments, though. The tools I've worked with have issues round-tripping them though ASTs.

I have thought about this a bit in the past, and I think the answer is that the AST for a given blob of text can be different depending on which version of the language you interpret it with. consider 'print "foo"' in python 2.7:

    Python 2.7.15 (default, Feb 14 2019, 16:26:56)
    [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.10.44.4)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import ast
    >>> ast.dump(ast.parse('print "foo"'))
    "Module(body=[Print(dest=None, values=[Str(s='foo')], nl=True)])"

but in python 3.7:

    Python 3.7.2 (default, Feb 14 2019, 16:12:42)
    [Clang 10.0.0 (clang-1000.10.44.4)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import ast
    >>> ast.parse('print "foo"')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Users/philsnow/.pyenv/versions/3.7.2/lib/python3.7/ast.py", line 35, in parse
        return compile(source, filename, mode, PyCF_ONLY_AST)
      File "<unknown>", line 1
        print "foo"
                  ^
    SyntaxError: Missing parentheses in call to 'print'. Did you mean print("foo")?

nothing changed in the text, but the AST is different. What should get committed to this hypothetical "git-ast"?

The answer probably lies in creating a different abstraction , different building blocks. In git we have bytes -> blobs -> trees -> commits, where the bytes of a given file get annotated with a file name (and mode bits etc etc, gross posix things) by the tree object.

In git-ast, maybe the abstraction is bytes -> lexemes (which annotates the bytes with a language version, and thus lexer version) -> tree? I haven't gone down the thought experiment that far.

I suspect you're probably right, but that example isn't the greatest because Python 3 isn't source compatible with Python 2. The source code text itself isn't even backwards compatible in that case.

That was maybe not a great example, pick any language change where a new keyword was introduced that wasn't previously a keyword or reserved word.

I tried to make an example with `with`, because that was introduced between 2.4 and 2.5 IIRC, and 2.4 and 2.5 aren't nearly as far apart as 2.* and 3.*, but I didn't find an easy way to install a 2.4 python (it didn't compile in pyenv on my mac, and I looked at a couple docker images before giving up).

But you're describing a benefit of storing an AST. Instead of the code suddenly changing meaning, you have an AST that says what the meaning should be.

The worst case is that you export it with the old version of the converter, and then the code changes meaning. You have silent corruption. But if you stored raw text this would always happen.

The good case is that you export it with the new version of the converter. It will notice that a variable name matches a keyword, and alert you to fix it.

Since even minor releases can introduce new syntax that previously failed to parse, this happens more than just on major version numbers. For example: async/await, formatting string literals, underscores in numeric literals, the list goes on and on. So you'll have to upgrade your formatter. And when you upgrade it, it may start formatting something differently than it previously did, even if that chunk had no new syntax.

Use one that won't change how it formats on a whim?

Because that requires either compiler support, or a plug-in to be written for each language and language version. But, I guess you can always fall back to text files in the worst case.

Sounds like a great idea, or binary representation of lexer tokens, the AST is larger in size than the lexer tokens and the AST can be easily generated from the tokens

Reply on news.ycombinator.com