Refactoring Python with Tree-sitter and Jedi

These sorts of cases are why I wrote srgn [0]. It's based on tree-sitter too. Calling it as

     cat file.py | srgn --py def --py identifiers 'database' 'db'

will replace all mentions of `database` inside identifiers inside (only!) function definitions (`def`) with `db`.

An input like

    import database
    import pytest


    @pytest.fixture()
    def test_a(database):
        return database


    def test_b(database):
        return database


    database = "database"


    class database:
        pass

is turned into

    import database
    import pytest


    @pytest.fixture()
    def test_a(db):
        return db


    def test_b(db):
        return db


    database = "database"


    class database:
        pass

which seems roughly like what the author is after. Mentions of "database" outside function definitions are not modified. That sort of logic I always found hard to replicate in basic GNU-like tools. If run without stdin, the above command runs recursively, in-place (careful with that one!).

Note: I just wrote this, and version 0.13.2 is required for the above to work.

[0]: https://github.com/alexpovel/srgn

Interesting use of treesitter. But I’m a little surprised that treesitters built in query language wasn’t used.

There’s no need to manually iterate through the tree, and use if statements to select nodes. Instead you can just write a couple of simple queries (and even use treesitters web UI to test the queries), and have the treesitter just provide all the nodes for you.

https://tree-sitter.github.io/tree-sitter/using-parsers#patt...

I’ve been looking at codemod tools recently, just as a way to extend my editing toolbox. I came across https://ast-grep.github.io/, which looks like it might address part of this problem. My initial test case was to locate all calls to a method where a specific argument was ‘true’, and it handled that well - that’s the kind of thing an IDE seems to struggle with. I’m not yet sure whether it could handle renaming a variable though.

I guess what I’m looking for is something that

* can carry out the kind of refactorings usually reserved for an IDE

* has a concise language for representing the refactorings so new ones can be built quite easily

* can apply the same refactoring in multiple places, with some kind of search language (addressing the task of renaming a test parameter in multiple methods)

* ideally does this across multiple programming languages

Interesting refactor!

This is trivial with codegen.com. Syntax below:

  # Iterate through all files in the codebase
  for file in codebase.files:
      # Check for functions with the pytest.fixture decorator
      for function in file.functions:
          if any(d.name == "fixture" for d in function.decorators):
              # Rename the 'db' parameter to 'database'
              db_param = function.get_parameter("db")
              if db_param:
                  db_param.set_name("database")
                  # Log the modification
                  print(f"Modified {function.name}")

Live example: https://www.codegen.sh/codemod/4697/public/diff

Tree-sitter is really powerful, but it's worth people learning a few methods they prefer to use because there are going to be situations where one method works better than another. Things I have found useful in the past include

- perl -pi -e 's/foo/bar/g' files

"-pi" means "in place edit" so it will change the files in place. If you have a purely mechanical change like he's doing here it's a very reasonable choice. If you're not as much of a cowboy as I am, you can specify a suffix and it will back the files up, so something like

perl -p -i.bak -e 's/db/database/g' py

For example then all your original '.py' files will be copied to '.py.bak' and the new renamed versions will be '.py'

For vim users (I know emacs has the same thing but I don't remember the exact invocation because it has been >20years since I used emacs as my main editor) it's worth knowing the "global" command. So you can execute a particular command only on lines that match some regex. So say you want to delete all the lines which mention cheese

:%g/cheese/d

Say you want to replace "db" with "database" but only on lines which start with "def"

:%g/^def/s/db/database/

OK cool. Now if you go 'vim *py' you can do ":argdo g/^def/s/db/database/ | update" and it will perform that global command across all the files in the arg list and save the ones which have changed.

Nice (simple) introduction to the tree sitter APIs.

If you're looking for a higher level interface, GritQL[0] is built on top of tree-sitter and could handle the same refactor with this query:

  language python

  `def $_($_): $_` as $func where $func <: contains `database` => `db`

[0] https://github.com/getgrit/gritql

I wrote up a Semgrep rule as a comparison to add! (also tree-sitter based, `pip install Semgrep`, https://github.com/semgrep/semgrep, or play with live editor link: https://semgrep.dev/playground/s/nJ4rY)

    pattern: |-
       def $FUNC(..., database, ...):
           $...BODY
    fix: |-
      def $FUNC(..., db, ...):
          $...BODY

Everyone's tossing in the name of other third-party packages, but have you explored the language section from Python's standard library?

https://docs.python.org/3/library/language.html

Would’ve been easy with fastmod: https://github.com/facebookincubator/fastmod

There is a Python library/tool called Bowler (https://pybowler.io/docs/basics-intro) that allows selecting and transforming elements on a concrete syntax tree. From my limited experience with it, I guess it would have been a nice fit for this refactoring.

This can also be achieved via ast-grep by the command `sg -p 'database' -r 'db'`.

https://ast-grep.github.io/guide/rewrite-code.html

I've always wanted to do mechanical refactors and recently ran into the problem the author ran into where tree-sitter can't write back the AST as source. Is there an alternative that is able to do this for most programming languages?

There are several straightforward ways to do this without needing Tree-sitter or Jedi.

Here are two approaches in Emacs.

https://emacs.stackexchange.com/a/69571

https://rigsomelight.com/2010/02/14/emacs-interactively-find...

what are some other tools like jedi? it would be cool to have a list of the favored tool for each language, or a meta-tool.

there's tsmod at least https://github.com/WolkSoftware/tsmod

i've heard of fastmod, codemod but never used them.

Was it possible to have Python parse its own ast and rename those variables?

Use a query expression instead.

I'm wondering if this would be fairly easy to do with AI?

I wonder if the author has ever heard something called an IDE?