Hacker News Reader

Hacker News

Comment by tosh | Hacker News Reader

Full thread

tosh

REPL-driven development seems like a natural fit with current agent concepts

nb: Bruce also did figwheel, a hot/live reloading tool for Clojurescript

https://figwheel.org

View on HN

I would like to see how to easily get regression tests from REPL driven development.

In general I wonder how to quickly set up mocks and such with chatbot integrations.

> how to easily get regression tests from REPL driven development

Usually it's a matter of renaming "comment" to "deftest", and move some things around. Not sure how/why it would be different when using a LLM compared to doing it manually.

There is often a misunderstanding that “REPL driven development” means typing things into the REPL prompt interactively and accumulating state that’s not easy to reproduce. Writing code down in comment blocks as you go is the solution to that, as you mentioned. With that technique I have no fear of restarting my REPL and getting back to where I was.

That's true. To be fair, if you come from a language like Ruby, JavaScript or Python, a REPL is basically a standalone experience, so I understand why people initially think it's like that, the name is more or less the same :) "Interactive Development" is maybe a better overall term that can lead to people thinking "Oh, what is that?" rather than "I've tried that already, what's the point?"

I like that. So you're saying that the switch between REPL and writing a test is much more seamless in a clojure system. That makes a lot of sense to me.

At a basic level any Lisp-like with an editor integrated REPL lets you put your cursor over the `greet` invocation below and evaluate it, and have the results shown in the manner of your choice (IDE window, inline temporary comment, etc).

This is all in the file as saved to your version control:

  (defn greet [name] 
    (-> (str "Hello, " name "!")
         println))

  (comment
    (greet "worldsayshi"))

It's 90% of the way to being a test already, just from the typical workflow.

But if you're maintaining a bunch of code blocks by comment toggling, why not just turn it into tests from the get go? I suppose there's a somewhat fine line between TDD and REPL given the right tooling for it.

Different purposes. I usually start out with comment blocks, for experimenting/prototyping, once happy, cement whatever assertions I did there, but inside deftests. Not everything is moved to a test though, some things are just "helpers" sort of, and those tend to remain inside comment blocks instead of being converted to deftest or similar.

I use the REPL as lightweight form of testing while I'm in the flow state, which is complementary to TDD. If there's too much code in the comment block, that's a good signal to move it into a formal test like you're saying, or at least move the code into a reusable function so the comment block can be shorter and focused on usage examples instead of implementation details.

Exactly. I start in the repl, but when it becomes a few expressions deep, I transfer it to the editor and turn it into a proper defun. Start 'sending' the defun to the repl and test it out in repl. Rince repeat.

> I start in the repl [...] I transfer it to the editor

That's exactly the kind of misunderstanding parent was talking about :)

Why start somewhere else than your editor, if it's already hooked up to the REPL? When I launch my REPL, I don't think it even accepts stdin, because there is no reason it has to.

There is no hard and fast rule (for me).

It is easy to start creating objects in the repl (say if you are testing an API) and work with those created objects in the repl, one step at a time. You are able to observe the behaviour of the objects every step of the way. This gives you a much better idea when you are writing the functions (usually copy-pasting from repl) in the editor.

I am sure it can be done from the editor itself. (say using the 'comment' block in clojure). It is just matter of preference.

> LLM compared to doing it manually.

When testing against an LLM you probably want to mock its responses for tests. Which is quite doable by copy-pasting its responses. I'm just curious what the "best" workflow is.

If I'd be using Clojure for this, it would look something like this:

1. Write a function that returns a LLM response

2. Play around with this function in a `comment` block, highlighting what I'm testing

3. If I find a response I'd like to "cement"/save and use for testing, Conjure (what I use to connect to the REPL) has a command that is basically "evaluate what's highlighted in the editor with whatever that returns when evaluated", so that basically copy-pastes the return value into the editor

4. Move it into a test somewhere

Not sure there is a better or faster workflow, from deciding "Hmm I want to test against this in the future" to "make test checks this" takes something like 2 minutes with a workflow like this.

I'm no sure, if you mean having the LLM interact with the REPL. REPL requires a lot of discipline from the developer to keep track of its state. LLMs seem to be far worse at this sort of long-term state tracking than most humans. It would likely keep forgetting things, calling undefined functions, or mistaking its own interfaces.

I agree, it’s tricky, but agents are also great at inspecting stuff

right now the advice in the readme makes a lot of sense: use git (or similar) for checkpointing

a bit like climbing with ropes and anchors

> REPL requires a lot of discipline from the developer to keep track of its state

More discipline than what is required when trying to keep track of all of that in your head? Personally I use a REPL because I'm lazy and dumb, and the REPL helps me keep track and makes sure I don't do stupid mistakes. If I wanna know what value a thing has, I just ask the REPL, instead of trying to retrace some logic by reading the code again.

What I don't like about building systems at the REPL is that when I investigate a possibility and it works out, I have to identify the code that's actually needed, clean it up, and turn my ad-hoc tests in the REPL into unit tests. There's a trade-off between quick experimentation and the mess I create in search of immediate feedback.

I think an LLM could help with that. Ideally, I could say, "Please add the widget-muckle-frobnicate function from the current REPL session to the codebase. Fix or flag any shortcuts I took in its implementation, such as hardcoding configuration and naming functions 'foo' or 'bar', and add unit tests that include the examples I tried in the REPL as well as property-based tests to exercise edge cases."

> Please add the widget-muckle-frobnicate function from the current REPL session to the codebase.

I think this is your problem, those two things should be the same, not two different things. When I'm working with a Clojure REPL, I write down code in a .clj file, then evaluate that code via the REPL but without leaving my editor. "Moving" something from a REPL session to the codebase is a matter of saving the current file you're editing.

I wrote more about the exact process I typically use when going from "prototyping with a repl, to having a test case" (https://news.ycombinator.com/item?id=44106445) and it's essentially changing "comment" to "deftest" then saving the file again, now it's a unit test.

I do find value from LLMs in general, but I don't see much value in taking a process that takes a minute or less and trying to make LLMs do it.

Exactly (on the first two paragraphs). I (almost) never type anything into the REPL window itself.

I might type a "5" and evaluate it just to see if the REPL is busy or not, but otherwise, everything the REPL evaluates comes from an emacs source code buffer, sent via CIDER to the REPL. At most, it might be in a "Rich comment" and thus not evaluated next time I load that file, but those tend to be small, interactive tests and not defns.

Unless emacs crashes with the buffer unsaved, there's nothing the REPL has that the source file doesn't.

I usually have a buffer that is kind of like a Jupyter notebook, and it accumulates a lot of stuff that may or may not turn out to be useful in the end. If I'm doing the work to keep things tidy as I go, so it's close to production ready at the end, then it isn't much different from traditional programming. It ends up just being TDD in one file instead of two.

REPL requires less discipline if you combine it with TDD. If there's an "abandon ship, tear down and set up the right state from scratch" red button it helps.

Reply on news.ycombinator.com