Clarifying exceptions and visualizing tensor operations in deep learning code

Be sure to read the last section - really deep work involved, including writing a Python parser! Here's a snippet:

"I used the built-in Python tokenizer, but built my own parser for the subset of statements (assignments, return) and operations supported by TensorSensor. There is a built-in Python parser, but it generates a different AST than I wanted; plus, filtering the built-in tree structure for just the parts I care about would be about the same amount of work as writing the parser. I built the parser using simple recursive descent, rather than adding another dependency (my friend antlr) to TensorSensor.

Next, we need to augment the AST with the values of all subexpressions so it looks like the AST from the previous section. This is a straightforward bottom-up walk of the AST calling eval() on each subexpression, saving the result in the associated node. In order to execute the various bits of the expression in the proper context after an exception, though, TensorSensor has to walk the call stack back down to the specific context that directly invoked the tensor library. The trick is not chasing the call stack too far, down into the tensor library.

Once the AST is augmented with partial values, we need to find the smallest subexpressions that evaluate to tensors for visualization purposes. That corresponds to the deepest subtrees that evaluate to tensors, which is the role of tsensor.analysis.smallest_matrix_subexpr()."

(Note: the author is also the creator of ANTLR.)

One of the biggest challenges when writing code to implement deep learning networks is getting all of the tensor (matrix and vector) dimensions to line up properly, even when using predefined network layers. This article describes a new library called TensorSensor that clarifies exceptions by augmenting messages and visualizing Python code to indicate the shape of tensor variables. It works with Tensorflow, PyTorch, and Numpy, as well as higher-level libraries like Keras and fastai.

This looks like a neat tool, especially for students taking courses like CS 231N! I imagine this already works in Jupyter Notebook? Perhaps this could be integrated into an extension for the VSCode, maybe in the debugger.

This looks really neat, but I've only felt the need for such a tool with tensorflow. With pytorch and numpy you can just use ipdb or any other debugger to be parachuted right at the exception and inspect shape and value of all the variables you need.

BTW, here's a link to the twitter thread.

https://twitter.com/the_antlr_guy/status/1313569854918590465

I believe problems solved by this tool generally disappear when using named tensors