Beefing up the Python Shell to build apps faster and DRYer | Hacker News Reader

Beefing up the Python Shell to build apps faster and DRYer

benplesser.com

66 pointsbjpless24 comments

To make this even more robust, steal a clever mechanism from Smalltalk -- the Change Log. Basically, all code changes and basically all of the manual manipulation of state was kept in a transactional log, so you could be as daring as you'd like with exploratory programming without the fear of losing any code. (Mandatory for Smalltalk, because all coding was interactive runtime exploratory programming.)

For iPython, you'd want to implement one change log per source file, so for a source.py file, there'd be a source.py.log file in the same directory. Runtime changes would first write a patch to the source.py.log file at parse/compile time. Then if you ever crash iPython after doing an hour of exploratory coding, iPython could reload your changes after the source.py file. (Actually, you'd want a little ui to come up, giving you the option of leaving off the last few changes, so you can omit something that causes a crash.)

Would automating Git be too slow?

There are problems with writing back out to source files and keeping pointers in source files from debugging sessions indexed correctly. The change log mechanism handily avoids this.

There is also a wonderful extension to IPython, namely autoreload: http://ipython.org/ipython-doc/dev/config/extensions/autorel...

I use it with level 1 and %aimport magic all the time and it's really sweet, especially with %ed. It probably does not work well with Django models however. You could use deep_reload: http://ipython.org/ipython-doc/stable/api/generated/IPython.... but I'm not sure if it would be enough.

I was having major issues with deep_reload and Django.

Erm, wow. I didn't even know about IPython extensions. Or %ed for that matter. Great tips, thanks!

EDIT: You seriously just changed my life with %autoreload

If your interpreter takes 3 seconds to start up there's something seriously wrong. Hopefully that's taking into account the human time to hit ^D and an up arrow, plus whatever time it takes to do all your imports or whatever. Assuming that was what was meant, then you probably really just want to learn about `python -i`.

I agree that the real solution here is "write unit tests", but even if you like an interactive interpreter, seriously, just restart it, it should take a half a second at most (and yes, reload() is broken by design, though sometimes it happens not to matter).

3 seconds includes closing AND starting the IPython shell. Typing "exit" or ctrl + d + return adds a little time and is generally annoying.

The builtin Python shell is admittedly faster to start/stop but IPython's features rock.

Ah, OK, fair enough.

FWIW you might also like checking out https://github.com/ludios/Pyquitter. I use it occasionally when I want something like this, you could rig it to start iPython's main().

It also doesn't use inotify but that'd be fairly trivial to add.

Ah I like it, thanks. From a quick glance, it seems like it just restarts the entire child process on change.

The big (but admittedly) dangerous feature of my script is that you can maintain state inside of the embedded shell on module reload.

I like pyquitter looks like a cool generalization, though, of the basic idea.

For those who didn't know (like me), from the python man-page:

  -i     When  a  script  is passed as first argument or the -c option is
         used, enter interactive mode after executing the script  or  the
         command.  It does not read the $PYTHONSTARTUP file.  This can be
         useful to inspect global variables  or  a  stack  trace  when  a
         script raises an exception.

http://man.cx/python

I think doctest and a UI around the doctest concept (there have been a couple) is a better approach. There was one whose name I can't remember – r...something. Unfortunately it used wxWindows, and was a pain to install... native UIs suck.

But with a Doctest model you just develop a script, and if you change something rerun the script, starting where you left off (assuming the script reruns – if it doesn't then you probably want to start where it fails). You can extend the script without reloading still, but changing the past requires starting over... but it's just CPU cycles, assuming you aren't doing something computationally expensive. But even if so, you could have a kind of cross-session Pickling memoize function if you don't want to recalculate things.

These reloading tricks are fragile and break in weird ways, like it won't fix badly initialized data, or classes that can't be upgraded because of state changes. It leads ultimately to a distrust of the environment, until you throw your hands up and go back to the old restart-frequently model like everyone else. Recursive reloading certainly isn't new, but it's never satisfying.

seems a hard way to go to not write unit tests. FWIW, I use the shell too for exploratory coding but the moment that I find myself making changes and restarting the shell, chances are it's time to write it in a unit test.

I write unit tests. This is still very useful to me...sometimes even in the process of writing those tests.

no argument from me on how useful it is.

I usually just drop into pdb wherever I want to prototype my tests.

Yes, especially with nose's --pdb and --pdb-failures options.

or in ipython just type pdb

pdb is a bit scary. Is there a way to drop "into pdb into ipython"?

There is ipdb.

"ipdb exports functions to access the IPython debugger, which features tab completion, syntax highlighting, better tracebacks, better introspection with the same interface as the pdb module."

http://pypi.python.org/pypi/ipdb

Or, you know, you could try not reinventing the wheel and use emacs.

How does emacs deal with:

  | Django models.py modules can’t be reloaded
  | normally due to the AppCache singleton

Do you mean Pony-Mode? Could you be a bit more specific?

Or Smalltalk, since we're apparently throwing out the idea of using Python entirely.

Reply on news.ycombinator.com