Debugging Lisp: fix and resume a program from any point in stack | Hacker News Reader

Debugging Lisp: fix and resume a program from any point in stack

lisp-journey.gitlab.io

85 pointstodsacerdoti27 comments

I'm wondering whether it would be possible to build some debugger for Python to fix that!

For example, I modified my IPython config to always activate https://ipython.org/ipython-doc/3/config/extensions/autorelo...

It works well for modules with redefined functions. So if you do

    from module import f1, f2
    
    ans1 = f1()
    ans2 = f2(ans1)

and f2 fails, then you can just modify the code of module and relaunch `ans2 = f2(ans1)`.

What is not available is reloading classes, for example if it looks like:

    from module import Class
    
    c = Class()
    c.f1()
    c.f2()

and there is a bug in f2, then the class won't be redefined.

I guess there is no perfect way to do it (for example what if f2 needs some variables defined in Class.__init__), but the situation is the same for Lisp with its dynamic typing.

So maybe the situation would be to have some command like `%reload Class`.

The other obstacle is restarting from a frame, and I understand that Python's standard exception handling doesn't allow that. But pdb exists so there should be a solution.

I suppose you could create a new class, with any fixes and wrap the existing object in the new class. It will kinda work, due to the duck typing, but it's not that elegant.

For the restarting, yeah, I don't know either.

in ipython just running %debug will get you in pdb in the particular frame that threw, there you can modify stuff using normal python, and then you can tell pdb to continue

> I guess there is no perfect way to do it (for example what if f2 needs some variables defined in Class.__init__), but the situation is the same for Lisp with its dynamic typing.

CL has the generic function update-instance-for-redefined-class[1] for this, so you can customize what happens to instances when a class gets redefined. I think most developers don't actually define methods for it all that often, but it's there if the defaults don't work for you.

[1]: http://www.lispworks.com/documentation/HyperSpec/Body/f_upda...

> the situation is the same for Lisp with its dynamic typing

Common Lisp and Smalltalk (quintessential dynamic systems) both track the existing instances of a class. If you update the class definition then they can update the existing instances of the class. In CL you'd want to specialize `update-instance-for-redefined-class` to handle the update.

http://clhs.lisp.se/Body/f_upda_1.htm#update-instance-for-re...

Most CL implementations don’t “track” the instances so much as have them point at a proxy for the real class object. When redefining a class, the old proxy object is marked as stale and then updated to point at the new class. This allows the runtime to lazily update instances to the new class (using the method you linked to) as they are encountered. As a result, you can have instances from several different class generations kicking around in memory.

Lisp has a lot of capabilities here that Python doesn't. It has restartable exceptions, the ability to redefine functions and whole classes at runtime, the ability to write new functions (/variables/classes etc) in the process of those redefinitions. Its debuggers are written in Lisp (completely extensible by you at runtime) offering a REPL right at the debug site.

Python is a completely different beast.

Python is dynamic and other than restartable exceptions I think it offers all of the features you mention. Indeed it's fairly common in the Python world to take advantage of that dynamic nature by distributing application security patches as 'monkey patches' [1] as code which runs to dynamically update other code.

[1] https://en.wikipedia.org/wiki/Monkey_patch

Not only can you redefine classes and functions at runtime in CL, my understanding is that all references to them will automatically be updated to the new code (I think unless it’s compiled?). It was already mentioned that Python’s classes come up short here, and I bet there are lots of cases where functions, too, miss the mark.

All implementations can update the function because all function calls are resolved at runtime (e.g. the symbol is examined). I don’t know what happens with inlines and macros though — don’t know if any implementations recompile those.

In most implementations, redefining an inline function or macro just means you may have some callers using the old code. SBCL warns if you redefine an inline function but doesn’t re-compile old code. Do note that the language has a different view on the matter though.

In its list of assumptions compilers are allowed to make about the code they receive, we can see…

> The definition of a function that is defined and declared inline in the compilation environment must be the same at run time.

> Within a function named F, the compiler may (but is not required to) assume that an apparent recursive call to a function named F refers to the same definition of F, unless that function has been declared notinline. The consequences of redefining such a recursively defined function F while it is executing are undefined.

> A call within a file to a named function that is defined in the same file refers to that function, unless that function has been declared notinline. The consequences are unspecified if functions are redefined individually at run time or multiply defined in the same file.

http://clhs.lisp.se/Body/03_bbc.htm

So, technically the compiler is allowed to remove runtime function call indirection for functions defined in the same file and/or recursive calls (unless you use a NOTINLINE declaration). In practice, you probably won’t see that happen unless you crank up the optimizations when compiling.

You are right that the language explicitly allows redefining functions in general

> Except in the situations explicitly listed above, a function defined in the evaluation environment is permitted to have a different definition or a different signature at run time, and the run-time definition prevails.

The thing is the CLOS has specific support for updating existing class instances at runtime.

Many languages do not. I don't know if Python does that.

It's quite something to claim Python has features that basically very few if any languages have, languages that Python has decided to ignore. Python barely even has a workable REPL.

Python is dynamic and everything is ultimately some sort of object (including functions and classes) and can be modified. Taking the features mentioned one by one:

> the ability to redefine functions

In Python you have two options here, you can either update the module to have a new function:

    >>> mymodule.myfunc = mynewfunc

But any existing references to `myfunc` will not be updated (e.g. if you did `import myfunc from mymoudle` instead of `import mymodule`).

Alternatively you can update the function __code__:

    >>> def f1(): return 1
    ... 
    >>> def f2(): return 2
    ... 
    >>> def f3(): return f1()
    ... 
    >>> f3()
    1
    >>> f1.__code__ = f2.__code__
    >>> f3()
    2

> and whole classes at runtime,

You can take a similar approach with Python classes:

    >>> class A: pass
    ... 
    >>> class B(A): pass
    ... 
    >>> b = B()
    >>> A._a = '_a'
    >>> A.a = lambda self: self._a
    >>> b.a()
    '_a'

You can also update SomeClass.__bases__ should you wish.

> the ability to write new functions (/variables/classes etc) in the process of those redefinitions.

As python is dynamic this can all be done at runtime. If you're not in a REPL you can call compile/exec/__import__ from your code.

> Its debuggers are written in Lisp (completely extensible by you at runtime) offering a REPL right at the debug site.

The Python debugger is written in Python and can be switched out. It's python code so like anything else can be changed at runtime. If you want something flashier than pdb you could use the IPython debugger: https://pypi.org/project/ipdb/

Edit: Want to modify an existing running python program but you didn't think to expose Python's dynamic extensibility before hand? No worries, just use parasite and inject arbitrary code into a running python process! https://pyrasite.readthedocs.io/en/latest/

Even Java IDE debuggers have long had the ability to redefine methods and classes at runtime (with some limitations).

There’s a difference between running a debugger on a program and running a debugger in a program.

Some of the basic hot-loading features can probably be approximated in Python. Here's some from my bookmarks(i've not tried them personally)

- https://github.com/breuleux/jurigged

- https://github.com/reloadware/reloadium

I would imagine lisp can do this on a whole different level. Emacs seems like a testament to that. Basically the entire editor feels like eval()

jurigged is excellent.

PDB has the interact command that drops you into a full blown python repl at the debug point. Changing an existing method on a class isn't a big deal. Redefining a class is possible as well but updating existing instances to the new class requires metaprogramming tomfoolery.

Hi everyone, happy to share more about my Common Lisp journey if you have any question.

IMHO, if CL got their handler system a tad tiny step further and better, they would have landed in a full algebraic effect territory, with a total flexibility of resuming your „throwers“ and not only restarting them. Paired with unparalleled CLOS capabilities that still drives context oriented programming, aspects, subjective dispatch research, CL would have been a religion by now.

I don’t quite understand what you mean by the distinction between resume and restart here. CL condition handlers allow code that signals a condition (sort of equivalent to “throws an exception) to continue on as if the condition was never signaled. You just need a handler-bind somewhere in the stack that doesn’t perform an exit. You can use this to do things like catch a type error and convert the input value to the expected type on the fly.

Can I have REPL that does not get stuck into debug loop? All I want is to read the error message and start again.

"Howto get out of debug-loop" is very much the same question as "howto quit vim". Takes a while to understand the cryptic "ABORT :R3" is actually the way out.

You can. First, you don't always need to type all this (is that a CCL or LispWorks prompt?). In SBCL, use only the restart number: "0". In your editor, use "q".

In SBCL, you can disable the interactive debugger. Put this in your .sbclrc:

    (defun print-condition-hook (condition hook)
      "Print this error message (condition) and abort the current operation."
      (declare (ignore hook))
      (princ condition)
      (clear-input)
      (abort))

    (setf *debugger-hook* #'print-condition-hook)

It only prints the error messages. It could be useful when experimenting with a bare-bones REPL on the terminal. For that purpose, see also sbcli and cl-repl on Github.

You can do the same thing in j, using the 'cut back' debugger command. It is a useful and powerful facility indeed.

People using Emacs without paredit or lispy to automatically close parentheses / do semi-structural editing should be put in jail! ; )

I like using M-( insert-parentheses in vanilla emacs when composing a sexp.

Reply on news.ycombinator.com