Experimental tree-based writing interface for GPT-3 | Hacker News Reader

Experimental tree-based writing interface for GPT-3

github.com

234 pointspyinstallwoes34 comments

The multiverse visualisation was really cool! This whole tool feels like it's designed for a writer from the future.

link for curious: https://generative.ink/meta/block-multiverse/

...from the futures!

I have always been bothered with the multinomial layer that picks the next token. It seems like you should be able to stick a classifier in there to detect if the probabilities align with a satisfactory answer and whether the model is unsure of the response. If that is the case, then it should branch out to a bigger model which would compute that single token that it is unsure about, and from that the rest of the response is obvious for the smaller model.

This way you could combine e.g. a 7B parameter model with a 70B parameter model, and get the quality of the larger model while most of the time you are only running the small model.

Edit: You could also store the full probabilities for each token, and then the classifier could detect if it had gone down a bad path, and then unwind the tokens and pick a different path.

That's reminiscent of a caching scheme, you could probably layer that down to even smaller and up to even larger models for increased performance and accuracy.

Better yet: have many smaller models that can when confused call upon larger models and the larger models could then pick the most appropriate smaller 'expert' model or, alternatively themselves escalate. Sort of a supervisor tree for language models.

I want to say this is where the progress is being made currently. Not only having 'experts per domain' but 'experts per pairing'

Sounds like speculative decoding.

https://huggingface.co/blog/vivien/optimal-lossy-variant-of-...

Contextual loading of a bigger model would incur a lot of latency. In this case it might be a good idea to farm out the request to a non-local model. Could be a good avenue for a hybrid local & remote completion system.

Is there anything experimenting with that?

I would be very surprised if (not at all)OpenAI didn't. I've noticed it with both gpt 3.5 and Gpt4 that if you ask it a difficult question the latency increases a lot, especially at the beginning of the answer. I wouldn't ve surprised if they did exactly what was described.

Right I’ve noticed that too. It’s like it takes a pause and reevals. Definitely feels like a cache or memoization of token paths. Interesting.

I maintain a less featureful but easier to use and less janky Loom implementation as an Obsidian plugin: https://github.com/cosmicoptima/loom

(Why Obsidian? Because then I don't have to write a text editor from scratch, and because one can then combine it with other plugins. Also because I intended for this implementation to work on mobile, but getting the UX right for that is annoying so it isn't supported right now.)

davinci-002 is a good publicly available model to start with. Weaving takes practice if you want something very specific.

How do you use it in your workflow?

i create new notes, put interesting and shiny texts in them, and weave from them for fun and intellectual or aesthetic inspiration

How do you handle the branches, variations? Like the leaf nodes? Do you use it to compare possible paths then when you decide you prune them?

What is the difference?

This is the work of @repligate, I can't recommend them highly enough. A true creative in every sense of the word, a rare spirit with a fundamental grounding and understanding of the spirit of LLMs. I highly, highly recommended pursuing every inch of their writing that interests you.

https://twitter.com/repligate

I don't grasp what this is. Not from reading that page and i'm not alone on that. Is someone able to tell what the practical real-world use cases are?

I was actually in the process of developing something just like this.

Think of not wanting to pollute your main conversation when you come across something you don’t know in the LLMs response, so you create a side chat and then navigate back once you get the info you needed.

I'll shamelessly plug my somewhat related but even less polished take on a text generation studio.

I was fascinated by the idea of constraining an LLMs vocabulary and it ended up as a publication and this: https://github.com/Hellisotherpeople/Constrained-Text-Genera...

SimpleMind does something similar with OPML and it works well across desktop and mobile. Definitely worth checking out on a tablet.

https://simplemind.eu/blog/mapping-your-thoughts-with-chatgp...

Can someone explain the primary use-case for this tool? Is it literally intended for writing a novel? Or just as sort of a plaything for exploring possibilities for generating text?

Wouldn't the limited context window constrain any attempt to write long form text with this model?

How is it limited? If anything it’s unlimited.

There is a 16k context window for GPT3.5. So for example, if you tried to put in a chapter of a novel then you would exceed the context window and the model would "forget" events from the start of the chapter.

Couldn't you train it on the novel?

Does it do or anyone knows some writing tool where the text can be anonymized before it is sent to GPT? I don't want to share sensitive data, like PII etc.

I could just manually specify words to be send to GPT as placeholders, "John Doe" -> "JD". Then on every response the placeholders would be resolved with the original text just for visual purposes "JD" -> "John Doe".

I know it can be made with a little programming and OpenAI API, but maybe no need to reinvent.

This is a good use case for a browser extension or a bookmarklet

This is a tool intended for exploring base language models. The best publicly available base model on the OpenAI API is "davinci-002". Not davinci, but davinci-002.

see also - an experimental tree-based thinking interface for software design:

- https://news.ycombinator.com/item?id=1962051 HN discussion 2010

- https://www.youtube.com/watch?v=f84n5oFoZBc (working video link of presentation)

Could anybody elaborate on what this does?

>Read this[0] for a conceptual explanation of block multiverse interface and demo video

[0] https://generative.ink/meta/block-multiverse/

i (biasedly) thought this was going to be a reddit-like infinitely nested threaded interface to chat with GPT (as opposed to the serial chat-like dialogue based one.

That's what I thought, too. Such an interface would be awesome to explore multiple trains of thought simultaneously and efficiently.

With the ability to seemlessly merge branches together using smart conflict resolution algorithms, this can be an invaluable tool for better decision-making.

Code review: please make method/function names all lowercase.

eg -

  format_openAI_response(...)

Reply on news.ycombinator.com