Hacker News Reader

Comment by deathmonger5000 | Hacker News Reader

I taught https://github.com/KillianLucas/open-interpreter how to use https://github.com/ferrislucas/promptr

Then I asked it to add a test suite to a rails side project. It created missing factories, corrected a broken test database configuration, and wrote tests for the classes and controllers that I asked it to.

I didn't have to get involved with mundane details. I did have to intervene here and there, but not much. The tests aren't the best in the world, but IMO they're adding value by at least covering the happy path. They're not as good as an experienced person would write.

I did spend a non-trivial amount of time fiddling with the prompts I used to teach OI about Promptr as well as the prompts I used to get it to successfully create the test suite.

The total cost was around $11 using GPT4 turbo.

I think in this case it was a fun experiment. I think in the future, this type of tooling will be ubiquitous.

View on HN

This is pretty cool!

Another use case where the cost of being slightly worse than a human is totally fine.(coming from someone that doesn't write tests lol)

I'd love to learn in more detail how it created those factories, corrected broken test database. It _feels_ that some of these tasks require knowing different parts of the codebase decently well, which from my experience hasn't always been the strong suite for AI assisted coding.

OI fixed the factories and config by attempting to run the tests. The test run would fail because there's no test suite configured, so OI inspected the Gemfile using `cat`. Then it used Promptr with a prompt like "add the rspec gem to Gemfile". Then OI tries again and again - addressing each error as encountered until the test suite was up and running.

In the case of generating unit tests using Promptr, I have an "include" file that I include from every prompt. The "include" file is specific to the project that I'm using Promptr in. It says something like "This is a rails 7 app that serves as an API for an SPA front end. Use rspec for tests. etc. etc."

Somewhere in that "include" file there is a summary of the main entities of the codebase, so that every request has a general understanding of the main concepts that the codebase is dealing with. In the case of the rspec tests that it generated, I included the relevant files in the prompt by including the path to the files in the prompt I give to Promptr.

For example, if a test is for the Book model then I mention book.rb in the prompt. Perhaps Book uses some services in app/services - if that's relevant for the task then I'll include a glob of files using a command line argument - something like `promptr -p prompt.liquid app/services/book*.rb` where prompt.liquid has my prompt mentioning book.rb

You have to know what to include in the prompts and don't be shy about stuffing it full of files. It works until it doesn't, but I've been surprised at well it works in a lot of cases.

What do you mean when you use the word taught for open-interpreter?

Looking at the OI docs wasn't too helpful.

"I did spend a non-trivial amount of time fiddling with the prompts" was it writing prompts?

I am really interested and this seems like a cool use case that I want to explore. Could you share the prompts on a github gist?

Here's the fork of Open Interpreter that I was experimenting with: https://github.com/ferrislucas/open-interpreter/pull/1/files

The system prompt that adds the Promptr CLI tool is here: https://github.com/ferrislucas/open-interpreter/pull/1/files...

I think I have the prompts still, but not on my work machine. I'll look tonight and edit this comment with whatever I can find.

I actually forked OI and baked in a prompt that was something like "Promptr is a CLI etc. etc., give Promptr conceptual instructions to make codebase and configuration changes". I think I put this in the system message that OI uses on every request to the OpenAI API.

Once I had OI using Promptr then I worked on a prompt for OI that was something like "create a test suite for the rails in ~/rails-app - use rspec, use this or that dependency, etc.".

Thanks for your interest! I'll try to add more details later.

Reply on news.ycombinator.com