Hacker News Reader

Comment by gcr | Hacker News Reader

For folks curious what this is, this seems to be a caching optimization for saving time on parallel streams of text. The benchmark is incidental. Most individual users likely have just one un-batchable conversation going at once with llama-cpp, and I think it’s unclear whether this PR improves that case much.

Also note that the demo video is sped up to fit inside GitHub attachment limits. Your observed speed may vary. :)

View on HN

I've been curious about using LLMs for large-scale refactoring. Prompts like

anywhere you find `FooBarBaz(blip, kap)` replace it with `new newThing(blip).bump(kap)`

I don't know how reliable it is, but it seems like if you can easily run this on commodity hardware it could totally replace most IDE refactoring tools, although obviously the IDE refactoring is more reliable, it seems like this could be made simple and flexible, and possibly just as reliable as IDEs.

But also it could enable some interesting things that you could never do with an IDE refactoring tool.

LLMs aren't very good at string operations. They struggle with things like character counts, extracting substrings, and replacements.

Yeah, for this kind of thing I would get the LLM to generate the regular expression.

I think that an LLM could be an amazing interface or translation layer for this sort of thing, but I would argue that the underlying operations of refactoring or something similar should remain very much like a function with discrete inputs and outputs.

I already do things like "write a script to replace `FooBarBaz(blip, kap)` with `new newThing(blip).bump(kap)` in a project folder"

I'm more comfortable with that because I find it usually takes two or three prompts to get it right

e.g. A couple hours ago I prompted it to help me do a diff of two commits ignoring all white space, just to check if there were any other changes. The first response didn't ignore new newlines, the second one was a multiline script, the third response gave me what I actually wanted.

  diff -w <(git show 0bb2c8579efe775de883e0182db48989bfa324f2:"path/to/file"|tr -d '\n') <(git show 6c71efc17497ad7c90b9c7b690075ec031c13c69:"path/to/file"|tr -d '\n')

I believe that the application of multiple streams in parallel is a natural evolution of using a single stream. I've used some local models for help in creative writing, and some of the most productive results I got were from running the same prompt and sequence of interactions dozens and dozens of times. Although in that case, I was personally going through each result line by line, I can certainly imagine fully automated tools that leverage the range of responses to a given prompt.

I always wondered of pipelining/parallelization can help with AutoGPT-like tasks were a supervisor AI delegates subtasks to sub instances.

Speed-wise, on a single stream, I have no need for it to generate text faster than I can read.

However, for scripts that try to use hundreds/thousands of invocations to solve some problem (eg. "write me a whole book"), the parallelism will be great (but obviously the script has to be written with that in mind).

So this will only work on, say, one prompt being run many times at once?

Reply on news.ycombinator.com