lachlan_gray

I wonder if eye tracking could be used as a vim motion if it becomes really good on new vr/ar headsets. Vim is already fast, but…

With any luck projects like MLC will help close the gap

https://github.com/mlc-ai/mlc-llm

LMQL helps a lot with this kind of thing. It makes it really easy to swap prompts and models out, and in general it allows you to maintain your prompt workflows in whatever way you maintain the rest of your python code.

I’m expecting there will be more examples soon, but you can check out my tree of thoughts implementation below to see what I mean

https://github.com/LachlanGray/lmql-tree-of-thoughts

You should check out “Neuromancer” by William Gibson. I think most of us are more unimaginative than we know wrt what the future can look like

The obsidian vim bindings can also be modified with the vimrc plugin. It lets you hook into obsidian’s native features with normal mode maps e.g. for pane splitting and navigation.

I’ve also found that GPT4 is fantastic at kabalistic interpretations. I asked it to interpret names and historical events prompting it with the first two paragraphs of this interlude from UNSONG

https://unsongbook.com/interlude-%D7%AA-trump/

Honestly didn’t mean anything by it but I appreciate the heads up!

Hate to be that guy but does anyone have a similar compilation for vim?

I kind of disagree that we will “build” goal directed AI. I think it’s something that gets discovered by accident, like life. Kind of like an accidental chemical replicator near a volcanic vent, we might happen to make a purpose built system that just happens to have other properties we didn’t intend it to, resulting in unbound activity. It’s much more interesting than a machine, we have a great deal of emergent strangeness to look forward to

Reminds me a lot of Asimov’s laws of robotics. It’s like a 2023 incarnation of an allegory from I, Robot

Randomly gained access long after I had forgotten I signed up, maybe 3 or 4 months

I think it started off because humans are humans, and have an easier time talking to something rather than “talking something”. One purpose of RLHF is so that the models tend to work well when you speak to them like that

Maybe useful for mouth gadgets like this thing

https://www.augmental.tech/

I thought it was interesting to see how they are presenting it to a general audience. So far AI search tools feel geared towards “serious questions”, you have to choose to go out of your way to use them. I thought it was interesting to see how they are spinning it to become more of an active technology

Woah thanks for sharing! it works although the total silence is a little unnerving

Upon inspection it turns out yes haha

Was it crystal by chance? I meant to have a look at it but never bothered, if it is maybe I will.

As for the SQLish approach, I’m not sure, it just seems to fit. I think it came from the way that there’s a thing you are requesting, and you have constraints you want applied to it. I think it’s one of those things where the analogy to SQL gives us developers just enough of a toe hold on what we’re doing to produce something but ultimately I think it will start to look less like SQL.

Yeah the semantics are very weird, but I guess “prompt engineering” is weird too, so it makes sense :) .

Everything between “sample” and “from” is basically a script that generates a prompt, which is incrementally fed to the LM.

Each line contained in double quotes will get appended to the prompt, using an f-string syntax, like normal LM templates. So if you have a local python variable “foo”, you can say “how do I make {foo}?” and it will substitute its value into the prompt (not interesting).

But things in square brackets are called “hole variables”, and do the opposite. If you follow up the previous with the line “you make it by [instructions]” , the prompt up to that point is passed to the LM, and the hole in the prompt is filled, and the result is stored in a local variable “instructions” which you can reference later on in the prompt, or in python script.

Any lines in between that don’t have double quotes are interpreted as python. So you can make program logic and LM calls conditional on the result of previous LM calls, or other results of some other process. So for example you could build a critique loop like the critique chain in the LC docs out of an actual while loop, where the while loop breaks when the LM determines the output is acceptable.

The exact same thing is possible with LangChain already, but it would involve creating templates, instantiating chains, etc, which isn’t bad, but adds complexity. In LMQL syntax, you can glance at the program and plainly see what it does using your programming brain… “yeah this while loop breaks when the screenplay is good enough, and the refined version gets returned” whereas I think LC’s abstractions make something simple like this look complex.

The “where” clause is where you specify constraints, which allow you to limit what the value of a hole can be. In this case you could apply a “where” constraint to a hole variable [rating] that forces it to be either “good enough” or “needs improvement”, and nothing else can possibly be sampled from the token distribution. This makes pipelines a lot more efficient by eliminating the need for “correction chains” in a lot of places. Also, once the tokens “ne” or “go” have been generated, LMQL doesn’t have to request any more tokens because the result is already uniquely determined, and it can substitute the rest and move on.

The other thing that I love about LMQL is that everything is async. Last time I tried, maybe two months ago, making a LC chain asynchronous didn’t feel natural. In my use cases, chains were async more often than not and it was kind of annoying.

In fact under the hood, the LMQL query is compiled to a decorated async function. So at the end of the day, you can use any of your queries as simple async functions. If you want to make react Agents, or any other LM abstraction you like, you pretty much just have to stick a few @lmql.query decorated functions inside a class definition and you’re good to go. That’s what I meant by the Tensorflow/keras analogy.

LMQL still isn’t mature and there’s a lot on the roadmap. Prompting is a wild west, and altogether we haven’t even discovered a lot of the problems we will need to solve. I like to think the situation is like how I imagine operating systems and a lot of software in general looked before Bell labs. For now at least, I think of all the options, LMQL is closest to the golden path.

Let me know if you have any more questions, feel free to send an email!

I’m making a similar “chatbot backend” kind of thing right now, meant to be configurable and versatile, while avoiding dependencies. It’ll be compatible with stuff like llama index and pinecone but doesn’t impose it. I’ll have a minimal foundation ready this week, I can send it your way when it’s up if you’d like

LMQL (language model query language) is a different take on prompting, and I find it less restrictive and more intuitive. Langchain is to LMQL what Keras is to Tensorflow

https://lmql.ai/

I’ve been using kindavim, this looks more powerful though, gonna try it out!

https://kindavim.app/

Another beauty of this kind of thinking is that it’s really empowering if you don’t have any computer science background. Coming from somewhere like physics, grasping how computers are actually simple opens very direct paths for getting things done.

I find that I retain the most information when I build myself a “story” of it. A kind of narrative makes it easy to relate from one piece of knowledge to another, and also interject new tidbits as you find them. Things like obsidian lend themselves really nicely

Since when we talk, our tongues tap patterns on the roof of the mouth and the back of the teeth, I wonder if AI processing could infer what words you are shaping from these sensors. Maybe it’s possible to input text by mouthing words silently, but without opening your mouth. Kind of like how it’s possible to eavesdrop from just the sound of keyboard clicks:

https://github.com/ggerganov/kbd-audio

Is the approach similar to what they did for Riffusion? (On front page a while ago)

https://www.riffusion.com/

Nice! I might embed this into my neovim setup. Do you plan on adding support for regenerating responses and editing previous queries? Maybe via a bang operator or something

Location: Toronto

Remote: flexible

Willing to relocate: yes

Technologies: Python, SQL, c++, LLM design patterns and ecosystem

CV: https://lachlangray.blot.im/resume.pdf

Email: lachlan dot gray2000 at gmail

---

I’m a new graduate with a background in physics and 2 years professional experience, interested in fresh problems. Particularly interested in designing/architecting new and capable LLM based software.

Also very keen on technical writing.

True, but as the orchestrator of my hands and limbs, I could not even begin tell you the implementation details of what they are doing to accomplish my high level intentions

It would be really interesting to see how a system like this competes with multimodal transformers. If it's good, systems like GPT-4 and Kosmos-1 won't need to have a built-in vision system.

AI will definitely have NSFW use cases... I think William Gibson's Neuromancer books will prove to have prescient insights into what that will look like. Tech like simstims that can seize your attention and can hold it indefinitely, and highly personal AI-designed fantasies