Hacker News Reader

Comment by rav | Hacker News Reader

Using the line/column information in Python's ast objects, you can sort of, almost recreate the concrete syntax, and I've tried to do so in my toy project https://github.com/Mortal/fstrings which converts old %-style formatting to the new Python 3.6 f-string syntax. It's not feature complete, but it mostly works.

View on HN

There's redbaron, which provides a full syntax tree, and is designed for making such changes: https://github.com/PyCQA/redbaron

Unfortunately, redbaron targets Python 2, which doesn't have the f-strings added in Python 3.6. The project looks quite nifty otherwise.

That looks very cool! Thanks for the pointer.

While line/column works nicely if no modification was done on the AST, it works pretty bad with modification.

Furthermore, you can't do cosmetic (e.g., indentation) changes on the AST that will reflect back on the source code (preserving user, i.e., non-modified, style decisions).

The Microsoft example (thinking of Roslyn) offers a great lossless syntax tree written with immutable data structures.

The lib2to3 library provides concrete syntax trees.

Yes I just mentioned lib2to3 in the post, that's probably the closest example. (There are also some Clang-based tools to convert C++ 03 to 11 to 14, etc.)

Although I believe that you need different terminology. A concrete syntax tree doesn't necessarily represent comments and whitespace. It represents the derivation of the string from the grammar. These are totally different things and I give an example in the article.

The CST is kind of wasteful and I'm not sure why you would use it other than not wanting to write semantic actions to prune the tree on the fly. If anyone has insight into this, please let me know.

Reply on news.ycombinator.com