On Repl-Driven Programming (2020) | Hacker News Reader

On Repl-Driven Programming (2020)

mikelevins.github.io

100 pointsivanvas52 comments

Used Clojure for ~4 years, doing REPL-driven development in Emacs. Have used Ruby and Java since. Maybe it's a rosy retrospection, but in terms of program production I think that was the most productive I've ever been. Falling into a flow state was much easier. Translating complex ideas into code was much easier. Building constructs up piece by piece felt natural and smooth. Sometimes I worry I'll never work in a LISP again.

Do you mean "falling into flow state was much easier" when you switched to java/ruby?

No, sorry, I meant it was easier back when I was using Clojure. Even invoking Java unit tests in the IDE or referencing a Ruby REPL in another window, while fast, are enough to break my concentration. Those seconds of context switching can derail my train of thought. At least in my case, building a piece of logic incrementally with near instant and continuous feedback from a REPL-integrated editor in Clojure just felt like an extension of my internal thought process.

Not to make you feel too bad, but based on your sentence you may appreciate a recent addition to my personal REPL toolkit in Clojure which I very much enjoy.

You may remember the rich comment forms in Clojure development as a communication tool to your future self and others. Now someone made them testable https://github.com/hyperfiddle/rcf#readme I feel this is even closer to the platonic ideal Clojure way of serializing your thoughts to file.

I also love Clojure and find it a very productive language. But surely the article makes it clear it falls very short of ‘real’ Lisp environments? There’s not much that Emacs and CIDER do that is so different to modes for other languages mentioned in the article with REPLs. If you’ve ever had stale protocol/record definitions in a Clojure REPL you know it doesn’t provide the features described. The whole ‘reloaded’ workflow is built around these limitations.

All that said I’ve never really liked this idea of editing as you go with extreme late binding in the debugger. Just seems a recipe for non-reproducible code. I love being able to explore and experiment in the REPL, but ultimately everything coalesces into code and tests in files. These give me far more confidence to move forward than a long lived REPL session.

There’s stuff like Reveal[0] which is interesting but I’ve only ever found it a marginal upgrade on pretty printing, at the cost that it sits outside Emacs so slows me down anyway.

0: https://vlaaad.github.io/reveal/

On Repl-Driven Programming - https://news.ycombinator.com/item?id=25620256 - Jan 2021 (206 comments)

Objective c could be. You can have undeveloped method … if you can make any unbanked message invoke debugger and continue ?

Not so sure about swift playground.

One surprise may be mvs … it is not a Repl. But the idea is that the system cannot be stopped. The sysplex cannot be down because you have a sub-system or just a program error. You have to look at any core dump and fix the system whilst it is running. Obviously not development-Repl. Waterfall and punch card, thank you. But production-Repl.

For production-Repl, basically all current systems needed it. There is no stop of your game world. Or may be yours but not the underlying world. It continues to run even on function upgrade or bugs etc.

The real question is this production-Repl kind (not exactly Repl but the feature of keep the image running, debug or change one part of it and propagate the change without breaking the system) is really hard.

Changing a s/370 control block or jes2 exit lately? Good luck and thanks you for all the fishes, that is why I go away.

A peculiar workflow of REPL-driven programming environments is that the running program is eventually "saved" by writing the state of the virtual machine to disk. There may be no source code to rebuild from, if the program was built interactively. It has its appeal, but it's kind of a nightmare from a maintainability standpoint.

I write Lisp for a living (in a couple of different forms), and I've never seen this. You do occasionally get problems where the state drifts a bit and you end up with order dependencies in files, but everybody I know works on source code and pushes it back and forth to the REPL via their editor.

> There may be no source code to rebuild from, if the program was built interactively.

If someone did that, they'd be a fool. Common Lisp and Smalltalk, the quintessential REPL-driven languages, have supported storing and loading source from files since their standards were published, and before then since the standards weren't created out of thin air.

That was true of InterLisp-D and the SmallTalk environments, but not the case of the systems that begat CommonLisp (MACLISP, the lispms, franzlisp, etc).

In other words it was a cultural thing, not a property of the languages. And even with the systems for which that was possible it was also possible to start with a fresh band and load code into it.

Moreover, Smalltalk and Interlisp interactions are recorded in files automatically--the changes file, the sources file, and the working image (which also contains tools--e.g. Interlisp's Masterscope--for keeping track of changes).

The notion that you'll lose your work if you use the normal means of interacting with the repl is a misunderstanding of how such systems work.

No.

I've been doing repl-driven development with Common Lisp compilers for about 33 years now. I have never ever worked in the way that you describe, and have never witnessed anyone else working that way, either.

It would be a nightmare for maintainability if anyone worked that way, but if they do, I haven't ever seen it.

I do write the state of the runtime to image files from time to time for various purposes. I assure you, though, that all the code I write lives in files that are maintained under revision control, and it's been that way for many years, and in all that time I've never been in danger of losing any work because of forgetting to write an image file.

To be fair, this general idea was at the core of how many versions of (at least GNU) emacs were created: launch minimal version, load lots of stuff, dump. Fire up dump in the future.

Nevertheless, this still does not add up to precisely what the GP described.

That's not for developing emacs, but for accelerating emacs startup.

Smalltalk in contrast has GP's model of saving and reloading the VM state.

Most Smalltalk systems and most Common Lisp systems and Interlisp and Factor, and, for that matter, some Schemes and the version of Dylan that Apple used to develop the bauhaus OS, all offer the convenience of starting up from a saved image and of saving a new image to preserve incremental changes for next time.

Interlisp and most Smalltalk systems also offer the convenience of automatic change tracking, writing your changes to the sources file and the changes file, and providing in-image tools for browsing and managing changes, including things like Interlisp’s Masterscope and Pharo and Squeak’s Monticello. These tools make life more convenient for people who prefer to work mainly with the in-image tools and not to mess with text files that much.

But if there’s a Smalltalk system that relies solely on the image file to preserve changes, with no support for other means of change tracking, I’m not aware of it.

I’ve always been curious about this. If you are not writing out the state of the program is there some convenient way to output parts of it? The article describes adding functions and refactoring data types but if you are doing that I’d assume you’d want those changes back in your source files.

The REPL and editor are connected. You have a key command in your editor that sends the form you want to evaluate to the REPL, which is in a separate window or pane.

when i write cl that is precisely what i do. i imagine this is the case with everyone - ie write code into the source file and send new bits to the repl

Right, except that the repl isn't in another pane; a repl prompt is.

Part of the confusion on this point is thinking that the prompt is the repl. It isn't. It's a UI for the repl. The repl is the process that reads, evaluates, and prints, then loops back to do it again.

Nothing about a repl requires there to be a prompt, and not all UIs for repls have them. Interlisp and Smalltalk have relied mainly on promptless UIs for their repls since the 1970s.

When working with Common Lisp, the main UI I use is an Emacs editor buffer and the key bindings that send input directly to the repl (bypassing the prompt). Typing at the prompt is the exception rather than the rule, and I could get along fine if I didn't bother to load the slime-repl contrib that provides it.

Yes, the evaluation of the code needs to be possible from everywhere. From the REPL window, the code editor, the debugger, and the inspector... Any code you select should be possible to evaluate in place.

It is more than four decades old obvious basic idea, and I do not understand why almost everybody ignores it.

> I’d assume you’d want those changes back in your source files

Not an expert, but your changes never 'leave' your source files. You iterate by changing the code in the file, sending it to the image/compiler, and repeat. There's no need for "getting code back into the files".

They always were in my source files. Nearly all of the text I type in a Lisp project is in a source file that is managed by a revision-control system. That's the way I've worked with Lisp systems for more than thirty years.

I send my changes to the repl with the stroke of a key.

Working with a repl doesn't necessarily mean typing text at a prompt. The prompt isn't the repl; it's just a particular UI for the repl.

Smalltalk systems do tend to store edits in the memory image (though they also write them automatically to a changes file and a sources file, so they aren't lost if you blow away your working image). Smalltalkers generally prefer to work in the image and treat text files as a serialization format for exchanging code and data.

But normally there's no question of getting "changes back in your source file" because they were always in your source file, or its equivalent, from the start.

> … is there some convenient way to output parts of it?

The old ways —

"Within each project, a set of changes you make to class descriptions is maintained. … Using a browser view of this set of changes, you can find out what you have been doing. Also, you can use the set of changes to create an external file containing descriptions of the modifications you have made to the system so that you can share your work with other users.

…

The storage of changes in the Smalltalk-80 system takes two forms: an internal form as a set of changes (actually a set of objects describing changes), and an external form as a file on which your actions are logged while you are working (in the form of executable expressions or expressions that can be filed into a system). … All the information stored in the internal change set is also written onto the changes file."

1984 Smalltalk-80 The Interactive Programming Environment page 461

https://rmod-files.lille.inria.fr/FreeBooks/TheInteractivePr...

The somewhat less old way —

https://www.google.com/books/edition/Mastering_ENVY_Develope...

Most Lispers type into source files and send the changes over to the REPL using e.g., SLIME on Emacs. Rebuilding the image from source is always possible.

Even under Smalltalk, which works more like you say, the system tracks and saves source changes. At least that's how it works under Squeak and Pharo.

Very few developers would be foolish enough to use a heap image as the source of all truth. I'm sure it happens, just like the place I worked at whose entire web stack was based on a magic DLL no one knew how to compile, happened. Like the magic DLL, it's widely considered a terrible, terrible idea.

You end up in OPs situation if you type into the REPL - which you're not supposed to.

The workflow is amazing because you write the software in the editor you're used to and have keybindings for interacting with the REPL inside your editor.

The mistake that a lot of "technique x" aficiandos make seems to often be that the technique has to be 100% complete or it's completely worthless.

In this case, there's a claim that python isn't actually REPL-based because you cannot change state while in an exception handler. Yes, that's kinda true, but there are debuggers for python that if you really need to can do a lot of that.

Other notable examples of this are type system maximalists. E.g. Go: no sum types or generics (until recently). Rust: no HKTs. Python: no runtime checks (I actually sometimes think this :)).

> A breakloop is a full-featured repl, complete with all of the tools of the main repl, but it exists inside the dynamic environment of the broken function

Sweeeet.

Well damn.. here I am in my python repl for hours per day, feeling smug about how efficient I am pulling logs and analyzing data. But is there an AWS lisp sdk?

Edit: there is but not officially.

Sounds fun but lisp seems so much less interoperable (out of the box) than python, and more of a departure from the style of everyday engineering. I love my Python repl tools.

Clojure is the ultimate interop language to me.

ClojureScript (shadow-cljs, nbb) for JavaScript

Clojure for JVM

babashka for scripts

libpython-clj for python interop.

ClojureCLR for .NET (beta state IMO)

ClojureDart for Dart/Flutter (early alpha state, one big shipped prod app on iOS/android)

--- Edit:

I don't quite know how to describe it, but having the _exact_ same files in a full-stack web application being read in both the Clojure backend and ClojureScript frontend (API contract) feels pretty neat. As an example having the frontend accurately give the user feedback why his input form isn't going to be accepted without a server round-trip or duplicating code just feels elegant.

I wonder if there is Clojure for the BEAM

Why yes!

https://github.com/clojerl/clojerl

Ok interesting thanks.

What are good resources to learn "proper" repl-driven development in lisp?

not all lisps have the same level of repl-driven development. clojure seems to be good, but it has nothing on common lisp. common lisp has this designed into it

anyway, check out

https://www.youtube.com/user/CBaggers/videos

https://github.com/rabbibotton/clog/blob/main/LEARN.md

edit: https://malisper.me/category/debugging-common-lisp/ (from another poster)

Part of why this [EDIT: to clarify - the debug/break loop process paired with the REPL] works in Smalltalk and Lisp is their dynamic nature, and also to the fact that they are systems as much as languages. In principle, Ruby and Python should be able to have as capable dev environments. With more static languages like C, Java, C#, and others you run into the problem of needing more "skeleton". You can't write:

  void foo() {
    bar();
  }

in those languages and expect the compiler to be thrilled about it. But you could get closer. Java runs on the JVM, C# on the CLR, those provide systems on which the languages (or programs written in those languages) are executed and the kind of access needed to approach the debugger-driven development approach. I haven't done this in a long time but with C# you can do:

  void foo() {
    throw new NotImplementedException();
  }

Now when your program execution hits `foo` you can catch it in the debugger (versus just crashing or otherwise terminating, or needing to set a breakpoint). With VS (and probably others, I did this with VS) you can change this code and recompile it and resume from that point. Now you decide to make the call to `bar`, oops, that method doesn't exist yet so you also have to create `bar`, and you can stub it out the exact same way. So it's not as easy and pleasant as in Lisp or Smalltalk where you can forgo the skeletal framework of the program and jump straight into it, building each function, method, class, etc. only as needed (truly JIT development), but you can get close. I believe Java has some similar facilities with some dev environments. There is more work, but it's not totally impossible.

All that said, I really enjoy the way you can do this in Lisp (and Smalltalk). I've been debating re-solving the Synacor Challenge [0] and recording how I did it in Lisp the first time. I used the debugger in precisely this way, writing a basic version which could read in the binary blob and did minimal parsing, but then entered an ecase (case expression that errors if there is no matching case). For every instruction that I hadn't implemented (none of them at the start) it brought me to the debugger, I modified the program, resumed it and it continued until the next unimplemented instruction. By the time I'd finished, I had the interpreter for the byte code up and running and was immediately getting to work on the rest of the challenge. This also worked as a nice goal-directed approach, implementing the whole interpreter was a "big" (not really) task, how to break it down and which part to do first? Well, interpreting the binary blob acted as my (very complex) test case and gave me immediate feedback (I also added some basic debugger capabilities to the interpreter) on whether I had done it correctly. No need to make the whole thing before testing it, I got to incrementally test each small change.

That approach could also be made to work with C# as I described before, write a basic skeleton and parse the instructions but throw a `NotImplementedException` whenever you get to an instruction you haven't, well, implemented yet. Then using VS change the throw to a method call (or do it locally) that implements the handler for that instruction. Repeat until all of them are filled out. It's not as smooth as the Lisp or Smalltalk cases would be, but it's doable.

[0] https://challenge.synacor.com - programming challenge by the same guy that does Advent of Code.

----------

Also, this has come up on here a few times in the past since it was written (2020). Two past submissions with comments:

https://news.ycombinator.com/item?id=25620256 - Jan 3, 2021 (208 comments)

https://news.ycombinator.com/item?id=25618457 - Jan 3, 2021 (5 comments, 4 + 1 dead, don't know why it's dead)

I do often use this approach in C#, so I started calling it debugger driven development

Seems like I should try Smalltalk

For what it's worth, modern C# can evaluate top-level statements with no other ceremony. The following is a valid standalone C# program:

  Console.WriteLine("Hello");

For REPLs, there are options like (my own) https://github.com/waf/CSharpRepl which stand on top of the Roslyn compiler infrastructure, which is quite extensive and can easily evaluate standalone functions and statements.

It's still nowhere close to the REPLs of lisp and smalltalk, but it's a step in a more flexible direction.

True, I'd forgotten about that. That was in the last major release, right?

What happens if there is a data structure and I make some changes to it that break different parts of the program? Does it warn about them, or do I need to place some tests? Also, is it possible to revert the program state back and forth to try some idea?

Lisp will tell you if it can, by dropping you in a breakloop. A breakloop is a repl that includes all of the normal Lisp repl features plus the full stack and dynamic environment of the broken code (which you can inspect and edit), plus a set of restarts you can select to resume execution, abandon the broken function, or transfer control in some other way defined by the context of the breakloop (which you can customize to your heart's content).

Smalltalk has equivalent features.

Lisp won't catch every possible error you could possibly make, of course, so testing is good defensive programming.

On the other hand, when I'm working in Lisp, testing happens almost by default. It's the majority of what I do: I write experiments in files and hit a key to send them to the repl to test what happens.

Tests generally start as little experiments like this. Some of them are useful over and over. Those graduate from informal experiments to formal tests.

This might be interesting for you: https://malisper.me/category/debugging-common-lisp/

This depends on on the language of course. In Common Lisp, a generics function called REINITIALIZE-INSTANCE is called. It's possibly to implement a method for this that updates the instance based on the new definition.

By default, any slots that have been added will be unbound. Attempting to access them will raise an error which will drop you in the debugger. From there you can then fix the instance and continue running.

I am yet to get truly repl-pilled, currently I just use files for bigger pieces of code, if there's a way to move functions and stuff into a file via SLIME I'm interested

clojure with Calva in VSCode is pretty amazing!

The way I work (and have worked for years) is I just always work in a file buffer and use SLIME keystrokes to send expressions to Lisp for evaluation.

Actually typing text in the repl buffer is very much the exception, not the rule.

tl;dr:

LISP and Smalltalk have the only True REPLs because only in those can you rebuild the interpreter and redefine classes without restarting. Everything else is just so cute with how hard they try.

Its an amusing parody and I appreciate it.

I don't actually think that Lisp and Smalltalk systems are the only possible examples of comprehensively interactive development systems. I'd say FORTH systems qualify (though they lack a lot of creature comforts) and so does Factor. Probably some others I'm not thinking of, as well.

I also don't think it's impossible to build a proper repl-driven development environment with other languages. I just think that the needed work hasn't been done for very many other languages, and for perfectly understandable reasons.

One reason is that it's a lot of extra work to build a proper repl-driven environment. I mean, you have to do all the work that would be needed for a non-repl-driven environment, and then you have a bunch more work to do on top of that.

A second reason is that in order to build such an environment you have to be sufficiently familiar with them to know why you would want one, and then you also need to be the sort of programmer who actually prefers to work that way (or else why would you bother with all that extra work?).

And, finally, you have to have the time, ambition, and other resources needed to actually build the thing.

So it should be no surprise that proper repl-driven environments are the exception rather than the rule, but I still prefer them, so I want more people to know about them, in hopes that some fraction of people will find them preferable, as I do, and the demand for them will grow, and that demand will help ensure that they don't disappear.

They're how I prefer to work. I don't want to lose them.

Right now I'm in this conundrum. Should I setup a REPL env (and build the missing pieces) for web dev (JS/TS) or just adopt Clojure?

Once you've worked that way, the regular way of programming seems archaic and a pita.

> Should I setup a REPL env (and build the missing pieces) for web dev (JS/TS)

check out CLOG - The Common Lisp Omnificent GUI framework for building web apps in common lisp

https://github.com/rabbibotton/clog

How do you use this approach on a team with shared code?

With source files and git. They aren't incompatible with using a REPL to drive your work.

I've been programming in lisp pretty much daily since the 1970s. I've also programmed in probably 50 other languages under as many different development methodologies.

The methodology described in the posting does exist, but it is never used for development; it is, however, very commonly used in three settings: (1) one-shot short programming [1SP], (2) exploratory programming [XP], and (3) interactive debugging [ID].

I just last night did an examples of 1SP: I was trying to work out the equation for a combinatorial probability, and wanted to simulate it so that I could tell what the right answer should be. I wrote the simulation expression right into the REPL -- not even in EMACS (my usual mode of programming is emacs+shell(usually lisp) -- I'm so old-timey that I never even bothered to install SLIME!) It took a couple copy-paste fixes to get it to work, but in a couple minutes, I had my result.

I do XP all the time, but usually on a single function, or small set of them, often when I'm trying to make a complex recursion work. This is like 1SP, but I'm not going to throw away the code; to the contrary, I'm going to copy the final, hopefully working, code into the body of my program. Therefore, I'll often do XP in an emacs+shell(lisp).

ID needs no further explanation; this is well described in the posting, and is, as mentioned, quite a wonderful process. (Sometimes ID and XP are mixed, as suggested in the posting.)

What never even happens is complete software development in lisp "core" -- what's sometimes called "workspace" development. This can be done, but it's a terrible software engineering methodology. (APL, which I've also done quite a lot of, although not so much anymore, has had a REPL since around the same time as Lisp, and has always worked in the workspace parading, and it was always a terrible mess for anything very complex.) [BASIC also had a workspace paradigm and REPL since around the same time as Lisp, but no one did anything big in the early BASICs, so this was fine. They were mostly used by student on early timesharing systems.]

One very important and under-appreciated property of Lisp that really makes interactive programming work so well in Lisp, and not so well in most other languages, is that line turns and white space make no difference at all to Lisp; it's all from the first open paren to the last close, so you can copy-paste whole expressions into the place of variables, if you like (more commonly the other way around, I guess), and there's never any question about it being syntactically correct.

For example, I take this expression:

(foo (bar (baz ...)) (bar (baz ...)))

where, say, (bar (baz ...)) is a heavy computation, so I want to refactor it. All I need to do is pull the expression (bar (baz ...)) out using paren-matching, wrap the thing in a let, and bob's your uncle, as:

(let ((x (bar (baz ...)))) (foo x x))

But the thing to note it that the expression (bar (baz ...)) could be 50 lines long! As I said above, line turns and white space (aside from enough white space to break up obvious things, like the names of separate variables), are mere decoration to Lisp, so you can decorate, or not, all you want.

This facility to not care at all about the lines and spacing, which most other languages care about in one or another way; make interactive programming a real joy.

Reply on news.ycombinator.com