Hacker News Reader

Comment by treesciencebot | Hacker News Reader

I haven't used bowler, but from what I can see it is using lib2to3 (through fissix), which can't parse newer Python code (parenthesized context managers, new pattern matching statement etc.) due to it being using a LL(1) parser. The regular ast on the other hand is always compatible with the current grammar, since that is what CPython internally consumes. It is also more handy to work with since majority of the linters already using it, so it would be very easy to port rules.

View on HN

(Author of Bowler/maintainer of fissix)

The main concern with using the ast module from stdlib is that you lose all formatting/whitespace/comments in your syntax tree, so writing any changes back to the original sources requires doing a lot more extra work to preserve the original formatting, put back comments, etc. This is entire point of lib2to3/fissix and LibCST, allowing large scale CST manipulation while preserving all of those comments and formatting. We do recognize the limitations of lib2to3/fissix, though, so there have been some backburner plans to move Bowler onto LibCST, as well as building a PEG-based parser for LibCST specifically to enable support for 3.10 and future syntax/grammar changes. But of course, this is very difficult to give any ETA or target for release.

Did you consider PyCQA/RedBaron (which is based upon PyCQA/baron, an AST implementation which preserves comments and whitespace)? https://redbaron.readthedocs.io/en/latest/

It was considered, but the initial goals of Bowler was to a) build on top of lib2to3 in an attempt to get broader upstream support for maintaining it as an "official" cst module, which fizzled out, and b) to use some of the more complex matching semantics that lib2to3 enabled, which baron and other cst alternatives don't really attempt to cover.

Things like "find any function that uses kwargs" or "find any class that defines a method named `bar`" can be easily expressed with lib2to3's matching grammar, and no other CST that I'm aware of (that isn't itself based on lib2to3) has equivalent functionality. This is something we wanted to add to LibCST, but haven't had the time to focus on given other priorities. Meanwhile, we used LibCST to write a safer alternative to isort: https://usort.readthedocs.io

Rog. I think CodeQL (GitHub acquired Semmle and QL in 2019) supports those types of queries; probably atop lib2to3 as well. https://codeql.github.com/docs/writing-codeql-queries/introd...

From https://news.ycombinator.com/item?id=24511280 :

> Additional lists of static analysis, dynamic analysis, SAST, DAST, and other source code analysis tools […]

Indeed! I also would suggest people to use a CST implementation (parso / LibCST) instead of refactor if they intend do large scale refactors, but from what I can see in my previous attempts (e.g teyit, a unittest assertion formatter) when you deal with small code fragments (a single expression, or a small statement) then you generally don't need to worry much about the style. The only concern is the literals (especially strings, which there are a few different variations of the same AST) where you could resurrect them back from the token stream (which the CustomUnparser representative in refactor allows).

The real start point for this project was to find / replace all type(<literal>)'s in CPython codebase with type(type(<literal>)) (e.g type('') would become type(str)) which is very light weight transformation, and I was able to write a script which did it without having any major problems about style on over 2000 files. Here it is for the reference: https://github.com/isidentical/refactor/blob/master/examples...

Also one thing to note here is that; in the last couple of years, thanks to black (and yapf), the adoptance of code formatters have really increased which is very nice for custom refactoring tools like refactor since the end-code would be refactored anyways so that means if you convert a multi line call, or a list to a single line version then the formatter you use probably reformat that segment anyways.

But thanks for authoring Bowler! It is a very cool project.

Reply on news.ycombinator.com