A guidance language for controlling LLMs | Hacker News Reader

A guidance language for controlling LLMs

github.com

552 pointsevanmays190 comments

I like this step towards greater rigor when working with LLM's. But part of me can't help but feel like this is essentially reinventing the concept of programming languages: formal and precise syntax to perform specific tasks with guarantees.

I wonder where the final balance will end up between the ease and flexibility of everyday language, and the precision / guarantees of a formally specified language.

Hear me out, just incubated a hot new lang that's about to capture the market and VC hearts:

SELECT * FROM llm

I know you are probably joking, but: https://lmql.ai/

It won't necessarily turn into some that is fundamentally the same as a current programming language. Rather than a "VM" or "interpreter" or "compiler" we have this "LLM".

Even if it requires a lot of domain knowledge to program using an "LLM-interpreted" language, the means of specification (in terms of how the software code is interpreted) may be different enough that it enables easier-to-write, more robust, (more Good Thing) etc. programs.

This is a hopeful evolutionary path. My concern is that I can literally feel Conway's law emanating from current LLM approaches as they switch between the actual LLM and the governing code around it that layers a buch of conditionals of the form:

if (unspeakable_things): return negatory_good_buddy

I see this happen a few times per day where the UI triggers a cancel even on its own fake typing mode and overwrites a user response that has at least half-rendered the trigger-warning-inducing response.

It's pretty clear from a design perspective that this is intended to be proxy to facial expressions while being worthy of an MVP postmortem discussion about what viability means in a product that's somewhere on a spectrum of unintended consequences that only arise at runtime.

This happened to me today on a prompt that I could not discern fit my original post as to "unspeakable things":

* design a men's haircut by combining a 1/4" shaved undercut around the ears and neck with a longer 2" crown and intended to provide cover from the sun on top.

followed by the AI interrupting itself mid-stream yet again after it had already answered the previous prompt to completion by providing step by step instructions to execute such a haircut.

* I'm sorry, I can't respond to your prompt. Please try something else.

My general impression is that there is near zero quality control oversight going on in this team and to their credit, that's been unusual in my experience observing and using M$ software post-Nadella.

A number of years ago we were designing a way to specify insurance claim adjudication rules in natural language, so that "the business" could write their own rules. The "natural" language we ended up with was not so natural after all. We would have had to teach users this specific English dialect and grammar (formal and precise syntax, as you said).

So, in the end, we abandoned that project and years later just rewrote the system so we could write claim rules in EDN format (from the Clojure world) to make our own lives easier.

In theory, the business users could also learn how to write in this EDN format, but it wasn't something the stakeholders outside of engineering even wanted. On the one hand, their expertise was in insurance claims---they didn't want to write code. More importantly, they felt they would be held accountable for any mistakes in the rules that could well result in thousands and thousands of dollars in overpayments. Something the engineers weren't impervious to, but there's a good reason we have quality assurance measures.

> but it wasn't something the stakeholders outside of engineering even wanted

Ha this reminds me of the craze for BDD/Cucumber type testing. Don’t think I ever once saw a product owner take interest in a human readable test case haha

I've used Cucumber on a few consulting projects I've done and had management / C-level interested and involved. It's a pretty narrow niche, but they were definitely enthusiastic for the idea that we had a defined list of features that we could print out (!!) as green or red for the current release.

They had some previous negative experiences with uncertainty about what "was working" in releases, and a pretty slapdash process before I came on board, so it was an important trust building tool.

> important trust building tool

This is so often completely missed in these conversations about these tools.

Great point.

“Incentivize developers to write externally understandable release notes” is an underrated feature of behavioral testing frameworks!

Ive had product owners take an interest in docs autogenerated from tests. Especially with artrfacts embedded. They like tuff like this:

https://github.com/hitchdev/hitchstory/blob/master/examples/...

And can be persuaded to look at the (YAML) source.

Gherkin isnt really a suitable language for writing test cases in - it's verbose, lacks inheritance, has clunky syntax and is stringly typed.

Just saw this on HN a couple days ago, sounds like just what was needed!

https://en.wikipedia.org/wiki/Attempto_Controlled_English?wp...

https://news.ycombinator.com/item?id=35936396

SQL looks the way it does (rather than some much more succinct relational algebra notation) because it was intended to be used by non-technical management/executive personnel so they could create whatever reports they needed without somebody having to translate business-ese to relalg. That, uh, didn't quite happen.

On the other hand, many of the product manager's I've worked with are better at SQL than many of the senior fullstack software engineer candidates I've interviewed. It's a strange world out there.

I think this is the exception, not the norm. My experience is business users (incl. PMs) are lost outside of Excel.

In my biased sample of SFBA tech companies it’s pretty common for PMs to know at least enough SQL to be dangerous. In early stage startups there’s no analyst to lean on, so they need SQL. In late stage companies there are data lakes and analytics databases specifically designed to be easily queryable, so SQL offers the best flexibility.

But you’re right, many of them are wizards in <whatever query language our tool uses>. Like VizQL for Tableau.

Having a personal need is often the best motivator for learning. Compared to closing ticket no 471961. A PM knows the shape of the data and what he is looking for, the sql is just a way to get there, rather than the other way around, when the goal often get lost in translation.

On the other hand, sql is also dangerous in that it gives you a result, even if the aggregation is completely wrong! Forgetting empty groups in a group by, loosing rows in a join or ignoring null values is common to see. Many cases of non techies “knowing” sql often turn out to be some basic query that give seemingly good data out but actually is complete nonsense when put under deeper review, a bit like AI hallucinations.

About EDN: http://edn-format.org/

Use LLM for the broad strokes, then fall back into 'hardcore JS' for areas that require guarantees or optimization. Like JS with fallback to C, and C with fallback to assembly. I like the idea.

Of course. Laws are the same way; defined norms for future behavior.

The professional managerial class must maintain appropriate distinctions between their rights and ours. Their belief in exclusive right to profit from our agency is at risk if AI can generate too much noise.

I think LLMs can transform between precise and imprecise languages.

So it's useful to have a library that helps and the input or output be precise, when that is what the task involves.

Well to be fair, yes we do need to integrate programming languages with large neural nets in more advanced ways. I don’t think it’s really reinventing it so much as learning how to integrate these two different computing concepts.

So far it it reminds of the worst days of code embedded in templates. Once these things start getting into multipage prompts they will be hopelessly obscure. The second immediate thing that jumps out is 'fragility'. This will be the sort of codebase that original "prompt engineer" wrote and left and no one will touch it for fear of breaking humpty dumpty.

The lovely thing about LLMs is that it can handle poorly worded prompts and well worded prompts. On the engineering side, we'll certainly see more rigor and best practices. For your average user? They can keep throwing whatever they like at it.

Exactly. I have been using OpenAI for taking transcriptions and finding keywords/phrases that belong to particular categories. There are existing tools/services that do this – but I would need to learn their API.

With OpenAI, I described it in English, provided sample JSON that I would like, run some tests, adjust and then I am ready.

There was no manual to read, it is in my format, and the language is natural.

And that is what I like about all this -- putting folks with limited technical skills in power.

Have you used the OpenAI embeddings AI? It is used to find closely related pieces of text. You could split the target text into sentences or even words and run it through that. That'll be 5x cheaper (per token) than gpt-3.5-turbo and might be faster too, especially if you submit each word in parallel (asynchronously! Ask GPT for the code). The rate limits are per-token.

Not sure if it's suitable for your use-case on its own, but it could at least work as a pre-filtering step if your costs are high.

(The asynchronous speedup trick works for gpt-3 too of course.)

I have not yet played with embedding. It is on my list though. Fortunately for my current purposes 3.5-turbo is fast enough and quite affordable.

We really need to start thinking of how to reduce magical thinking in the field. It’s not pretty. They literally quote biblical guidance for the models and pray that this would work.

And start their prompts with “You”. Who is “You”?

The LLM. The most common end-user interface for LLM is a chat so the ser expects to be talking to someone or something.

“You” is an optimization for the human user. Here’s some insight: https://news.ycombinator.com/item?id=35925154

If you see any prompt that starts with You, generally it is a poor design. Like using a “goto” or global variables.

This is true for something like raw GPT. For the chat models that have been specifically optimized for "you" prompts, this is false. See the discussion in the link I provided, along with the leaked copilot/bing prompts.

Or, in other words, use a model in a way that fully takes advantage of how it was specifically optimized, from the intentional burning of massive amounts of compute time/money to get it that way.

Bing prompt is hilarious. And so wrong in so many ways.

I would assume that OpenAI helped with the Microsoft prompt. Being Microsoft, and it being a front page feature, I would also assume that there are miles of powerpoint showing data in support for it.

What do you see as a better prompt? How would you test its efficacy?

Yeah, GOTO or global variables are usually a not so good idea... except in cases where those are actually the best tools for the job and any alternative makes code actually worse.

Coincidentally, the same applies to "you"

Functions are an optimization for the human user

But is it a step to greater rigor? Or is it an illusion of rigor?

They talk about improving tokenization but I don't believe that's the fundamental problem of controlling LLMs. The problem with LLMs is all the data comes in as (tokenized) language and the result is nothing but in-context predicted output. That's where all the "prompt-injection" exploits come from - as well as the hallucinations, "temper tantrums" and so-forth.

It is not a step towards greater rigor. They literally have magical thinking and “biblical” quotes from GPT 11:4 all other the place, mixing code and religion.

And starting prompts with “You”? Seriously. Can we at least drop that as a start?

> And starting prompts with “You”? Seriously. Can we at least drop that as a start?

What is wrong with this?

“You” is completely unnecessary. What needs to be defined is the content of the language being modeled, not the model itself.

And if there is an attempt to define the model itself, then this definition should be correct, should not contradict anything and should be useful.

Otherwise it’s just dead code, waiting to create problems.

> Otherwise it’s just dead code, waiting to create problems

it's very possible that the pretense improves results: most recorded interactions /are/ between two people, after all.

Examples: HNN, StackOverflow, Reddit...

I definitely agree with this.

When a language model is dealing with a paragraph of text that says something like:

   You are standing in an open field west of a white house, with a boarded front door.
   There is a small mailbox here.

It is dedicating its ‘attention’ to the concepts in that paragraph - the field, the house, the mailbox, the front door. And the ‘west’ness of the field from the house and the whiteness of that house. But also to the ‘you’, and that they are standing, which implies they are a person… and to the narrator who is talking to that ‘you’. That that narrator is speaking in English in second person present tense, in a style reminiscent of a text adventure…

All sorts of connotations from this text activating neurons with different weights making it more or less likely to think that the word ‘xyzzy’ or ‘grue’ might be appropriate to output soon.

Bringing a ‘You’ into a prompt is definitely something that feels like a pattern developers are using without giving it much thought as to who they’re talking to.

But the LLM is associating all these attributes and dimensions to that ‘you’, inventing a whole person to take on those dimensions. Is that the best use of its scarce attention? Does it help the prompt produce the desired output? Does the LLM think it’s outputting text from an adventure game?

Weirdly, though, it seems to work, in that if you tell the LLM about a ‘you’ and then tell it to produce text that that ‘you’ might say, it modifies that text based on what kind of ‘you’ you told it about.

But that is a weird way to proceed. There must be others.

> “You” is completely unnecessary.

It isn't, for at least two main reasons:

1) In LLMs, every token has some degree of influence on the output. Starting the prompt with "You" and writing it in second person attracts the model towards specific volumes in the latent space. This can have good or bad impact on the output, depending on the model.

2) Instruct-type models are fine-tuned to respond to second-person prompts. "You"-prompts are what those models expect. If you're working with a model that isn't instruction-tuned, use whatever you want.

Have you tried removing it and checking the results? Could it be that this is a cargo cult, people using You, simply because it was present in the ChatGPT prompt at the time it got leaked?

I'm not interested in pleasant, formal "conversation" with the thing roleplaying as human and wasting, time, keystrokes and money, I want data as fast and condensed as possible without dumb fluff. Yes, it's funny for few first times but not much after that

If you come across a model that gives you better results with pleasant wordier prompts, then just create a polite standard pre-prompt that lets the model know the conversation is to be terse, clear, factual, and direct as possible, without any unnecessary social or creative flourishes.

I mean, whatever gets the best results is what gets the best results, right? It's not a question of "funny" or "fluff".

The result is actually richer than ‘predicted output’ - it’s a probability distribution over all possible output.

Having richer ways to consume that probability distribution than just ‘take the most likely thing, after adding some noise’ is more conducive to using LLMs to generate output that can be further processed - in rigorous ways. Like by running it through a compiler.

Think about how when you’re coding, autocomplete suggestions help you pick the right ‘next token’ with greater accuracy.

Note that for any fine-tuned models (like GPT-4, where the foundation model has not been made accessible) the model does no longer give the "probabilities" of the next tokens, but rather their "goodness". Where the numbers say how good a token would be relative to the aims the model inferred from its fine-tuning.

Isn’t that the same thing? The non-fine-tuned models also have assumptions based on corpus and training. I don’t think there’s such a thing as a purely objective probability of the next token.

It's very different. We don't know exactly what the model consideres good after fine-tuning (which can lead to surprising cases of misalignment), while the probability that something is the next token in the training distribution is very clear. I don't know how they measure it, but they can apparently measure the "loss" which (I think) says how close the model is to some sort of real probability.

What I meant was, fine tuning is not substantially different from training. It seems odd to use different words for the resulting systems.

But fine-tuning is very different from (pre)training. Pretreating proceeds via unsupervised learning on massive amounts of data and compute, while fine-tuning uses much smaller amounts, with supervised learning (instruction tuning) and reinforcement learning (RLHF, constitutional AI).

"no longer"??

The deep learning models (of which LLMs and GPTs are a type) have never returned probabilities. Ever. Why do people have that hallucination suddenly?

They do produce probabilities at the end of generator, And they do select a single token for output. With highest probability or somehow randomized.

So, end users see only one value. But with access to internals all high value variants can be considered. The easy way to do it is to select one, save the state. Look forward and roll back to saved state. Try another token. Select the best output. The smart way is to do it only at key points, where it matters the most. Selecting those points is a different task. May be another model.

The probabilities (in form of log odds) can be directly accessed in the OpenAI playground, I believe. The "try again" approach would only work for temperature = 0, when the model always returns the tokens with the given probabilities. For temperature = 1 it always returns the token with the highest probability. Usually they use something like temperature 0.8 in ChatGPT, I think, which still biases the model toward the more likely tokens. In the playground the temperature can be set manually. (Again, for fine-tuned models, which are the majority, those are numbers are not probabilities but "goodnesses".)

Okay why is this downvoted? wtf

upvoting a bit. my guess we have here anti-AI vigilantes. Actually it's not a guess anymore, and not something new in general.

You can literally fire up the openai playground and ask gpt3 to give you all alternate token probability

The result is actually richer than ‘predicted output’ - it’s a probability distribution over all possible output.

-- This is, uh, false. If an LLM output a "probability distribution over all possible output", it would be producing a huge, a vast, vector each time. It doesn't. ChatGPT, GPT-3 etc produce a string output, that's it. You can say it's following a probability distribution of outputs from output space but just about anything the output does that.

Think about how when you’re coding, autocomplete suggestions help you pick the right ‘next token’ with greater accuracy.

-- Uh, you missed where I said "in-context predicted output". The Transformers architecture is where the LLM magic happens. It's what allows "X but in pig Latin" etc.

It's hard to get that these systems are neither "fancy autocomplete" nor AGI/something magic but an interest but sometimes deceptive middle ground.

ChatGPT and GPT are APIs over LLMs.

The huge vector is what the neural net outputs. ‘Sampling’ is the process whereby a token is selected.

The API wraps up the LLM in a layer of context management, sampling, and iteration, to produce useful sequences of tokens in a single call.

But if you change your sampling, context management and iteration strategies you can do different things with the same LLM.

> That's where all the "prompt-injection" exploits come

Giving access to LLM is like giving access to console, or any other application. Not safe in general. The application by itself should be limited and sandboxed. Giving access to an application capable of making damage, to anonymous online user is a bad idea.

I don’t think formal languages are going anywhere because we need the guarantees that they can provide. From Dijkstra: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/E...

You need to be able to define all of the possible edge cases so there isn’t any Undefined Behavior: that’s the formal part

Humans can use LLMs to manipulate these languages to achieve specific goals. I can imagine designing formal languages intended for LLMs to manipulate or generate, but I can’t imagine the need for the languages themselves going away.

> LLMs, like humans, can manipulate these languages

Absolutely not. LLMs do not "manipulate" language. They do not have agency. They are extremely advanced text prediction engines. Their output is the result of applying the statistics harvested and distilled from existing uses of natural language. They only "appear" human because they are statistically geared toward producing human-like sequences of words. They cannot choose to change how they use language, and thus cannot be said to actively "manipulate" the language.

I made some edits, does that satisfy your constraints? Humans are the agent, LLMs the tool

That “appearance” is pretty good at triggering our anthropomorphizing behaviors. I like your handle, did you read Richard Bach’s Illusions by any chance?

> That “appearance” is pretty good at triggering our anthropomorphizing behaviors.

It's truly unfortunate, because I think that tendency for people to anthropomorphize LLMs leads to people thinking they can trust LLMs and take their words at face value, which is a path to significant problems, I think.

> I like your handle, did you read Richard Bach’s Illusions by any chance?

Absolutely, yes — good catch!

Humans have been interpreting the world around us by populating it with imaginary versions of ourselves forever, I doubt it will lead us to "trust" LLM's but it certainly will lead to some serious misunderstandings.

“If you will practice being fictional for a while, you will understand that fictional characters are sometimes more real than people with bodies and heartbeats.” ― Richard Bach, Illusions

Great book!

I am not so sure. I asked GPT4 to "optimize" English for AI-AI communication, and the modifications were accepted by another GPT4 instance for seemingly-lossless discussion. First instance: https://shareg.pt/8vgEgOh

Second instance: https://shareg.pt/vSCEq4b

----- Sample communication: Instance2: OptE: SignificantImpact: UnifiedTheoryInformsFundamentalForcesInteractionNature(.) ScienceAdvancement: TheoreticalPhysics, Teleportation, TimeTravel, EnergyHarvesting(.) LIMIT: CurrentKnowledgeConstraints, SpeculationPresent(.) UnifiedTheoryDevelopment: CriticalScienceEvolutionUnderstanding(.) ACK: ContinuedExplorationEssential(.) Instance1: OptE: ACK: Agreed(.) ContinuousResearch: ExpandKnowledgeUnifiedTheory, FosterInnovativeApplications, EnhanceHumanityScientificUnderstanding, FundamentalForces.Progress(.) Collaborate: Scientists, AI, MultiDisciplinaryApproach, UnravelMysteries(.) Aspire: UnifiedTheoryUnlockUniverseSecrets, ProgressSociety(.) ----- Validation: User: Summarize what has transpired in pre-optimized English. Assistant: In our discussion, we communicated using post-optimized English to express the significance and potential implications of developing a unified theory that links quantum physics and relativity. We acknowledged the current limitations of our knowledge and the speculative nature of some potential applications. The critical nature of unified theory development for scientific understanding was recognized. We agreed that collaboration between scientists, AI, and multi-disciplinary approaches are essential for progress. Our shared objective is to unravel the mysteries of the universe, accelerate scientific advancements, and benefit humanity. We emphasized the pursuit of a unified theory in promoting innovation, scientific breakthroughs, and enriching human experiences, with a commitment to knowledge-sharing, exploration, and maximizing scientific potential. ---- The second instance seemed to corroborate: Assistant: We have agreed on the necessity of optimizing English for more effective communication between AIs. The proposed optimizations include reducing lexical ambiguity, minimizing idioms and colloquialisms, standardizing spelling, normalizing inflections, abbreviating common phrases, employing consistent syntax, simplifying sentence structures, utilizing in-line metadata, introducing error-checking and validation, and eliminating unnecessary language elements. Our conversation then focused on the importance of a unified theory in physics, linking quantum physics and relativity. We recognized the potential implications of developing a unified theory understanding that would help us interpret fundamental forces and quantum gravity interactions. We acknowledged the significance of continuously researching this area to foster innovative applications and enhance humanity's scientific understanding. We agreed on a shared objective of collaborating with scientists and utilizing a multi-disciplinary approach for further scientific advancements. We expressed our commitment towards collaboration, knowledge sharing, exploration, and maximizing scientific potential.

What about that makes you think GPT-4 has agency?

Does it strike you that this conversation seems to consist of futurology forum clichés ("What progress will AI allow?" "AI's impact on physics") that probably inform this from the training data?

What also strikes me is the shorthands of communication here sounds like cliché from 2nd rate scifi novels, the likely source of the format.

Since what is cliché? It's what's present in the majority of a genre and what the training sees as structure.

I asked it to translate a sentence into optE. Apparently that is something it knows about in training data

> Optimized English, or OptE, is a theoretical construct as of my last training data in September 2021 and doesn't have any widely accepted or standardized form. In the event that OptE is conceived after this period, I would not be able to provide a precise translation due to my training limitations.

Only partially tongue in cheek: have you tried asking it for an optimal syntax?

It's rigor applied where we don't need it, and ignores where we do (mathematical proofs and NN theory, architecture, hyper parameters, training schemes, etc).

I have a somewhat irrational hatred towards almost all of the prompt oriented stuff being thrown about recently. There are a few (very few) input related training schemes that are interesting, but quite a bit of the "proompt-physicians" are just heralding the idea of essentially 'concise and effective communication' as 'I'm a ML expert now' ... which is annoying.

Why would you dislike actual prompt engineering? This isn't some grifter trying to claim they're an expert because they wrote a cool prompt, this is a full fledged structured templating system for LLMs from an excellent author whose done a ton of other ML work.

I think you should attack actual grifters instead of an excellent project.

Maybe someone will make an LLM with equivalent functionality to python that you can conveniently control with python syntax.

This is pretty fascinating, but I'm not sure I understand the benefit of using a Handlebars-like DSL here.

For example, given this code from https://github.com/microsoft/guidance/blob/main/notebooks/ch...

    create_plan = guidance('''{{#system~}}
    You are a helpful assistant.
    {{~/system}}
    {{#block hidden=True}}
    {{#user~}}
    I want to {{goal}}.
    {{~! generate potential options ~}}
    Can you please generate one option for how to accomplish this?
    Please make the option very short, at most one line.
    {{~/user}}
    {{#assistant~}}
    {{gen 'options' n=5 temperature=1.0 max_tokens=500}}
    {{~/assistant}}
    {{/block}}
    {{~! generate pros and cons and select the best option ~}}
    {{#block hidden=True}}
    {{#user~}}
    I want to {{goal}}.
    ''')

How about something like this instead?

    create_plan = guidance([
        system("You are a helpful assistant."),
        hidden([
            user("I want to {{goal}}."),
            comment("generate potential options"),
            user([
                "Can you please generate one option for how to accomplish this?",
                "Please make the option very short, at most one line."
            ]),
            assistant(gen('options', n=5, temperature=1.0, max_tokens=500)),
        ]),
        comment("generate pros and cons and select the best option"),
        hidden(
            user("I want to {{goal}}"),
        )
    ])

My guess is you can store the DLS as a file (or in a db). With your example, you have to execute the code stored in your db.

You can serialize and ship the DSL to a remote server for high speed execution. (without trusting raw Python code)

There's prior art for pythonic DSLs that aren't actual python code.

Why not just use JSON instead, though? Then you can just rely on all the preexisting JSON tooling out there for most stuff to do with it.

I think the DSL is nice when you want to take part of the generation and use it later in the prompt, e.g. this (in the same notebook).

---

prompt = guidance('''{{#system~}}

You are a helpful assistant.

From now on, whenever your response depends on any factual information, please search the web by using the function <search>query</search> before responding. I will then paste web results in, and you can respond.

Ok, I will do that. Let's do a practice round

That was great, now let's do another one.

Ok, I'm ready.

{{gen "query" stop="</search>"}}{{#if (is_search query)}}</search>{{/if}}

Search results: {{#each (search query)}}

</result>{{/each}}

{{/if}}''')

---

You could still write it without a DSL, but I think it would be harder to read.

Your example assumes a nested, hierarchical structure while the former example is strictly linear. IMHO that's the key difference there, as the former can (and AFAIK is) be directly encoded and passed to the LLM, which inherently receives only a flat list of tokens.

Your example might be nicer to edit, but then it would still have to be translated to the actual 'guidance language' which would have to look (and be) flat.

We could write a python package that could? A codegen tool that generates codegen that will then generate code? <insert xzibit meme here>

I think chatgpt4 can easily write the python code... wait a second!

Would love to hear your opinion on guidance, in the context of prompt injection attacks :-)

I'm personally starting with learning Guidance and LMQL rather than LangChain just in order to get a better grasp of the behaviors that I've gathered LangChain papers over. Even after that, I'm likely to look at Haystack before LangChain.

Just getting the feeling that LangChain is going to end up being considered a kitchen sink solution full of anti patterns so might as well spend time a little lower level while I see which way the winds end up blowing.

If this comment performative comedy? Are these real technologies ?

Not quite sure what the spirit of your comment is. But, yes, they are real technologies. Very confused as to why you would even find that dubious.

Not dubious, I just read your comment and it felt like I was reading satire. Even the cadence of your words felt funny.

Anyway, I’m not surprised. It’s a new market, everyone’s in on it.

LangChain: https://news.ycombinator.com/item?id=34422627

LQML: https://news.ycombinator.com/item?id=35956484

Haystack: https://news.ycombinator.com/item?id=29501045 or more recently https://news.ycombinator.com/item?id=35430188

It is satire. They just don’t realise it yet.

It’s pretty clear that we are in the phase where everyone is rushing to get a slice of the pie selling dubious thing and people start parroting word soup hoping they actually make sense and fearing they will miss out. That’s indeed what people often and rightfully satirise about the IT industry. That’s the joke phase before things settle.

How is it satire to be excited and interested in how to use compelling and novel technology? There's a lot of activity. Not everyone involved is an idiot or rube. The jadedness makes my head spin.

It's not idiocy. If you don't see the slight ridiculouness in the world soup of the original commenter, I can't do anything for you. I'm not jaded. I'm amused you are all so deconnected from the normal world you think this kind of situation is somehow normal and not at all funny.

Why don’t you try shaking your fist at your computer instead of wasting everyone’s time in this thread. Touch grass

You should have led with generosity instead of tacking it on at the end.

It might have saved me from having a ridiculous conversation about the cadence of my words, and instead there might have been a higher chance of someone saying something substantive about my assumptions regarding the technology.

But here we are.

I agree, I came off a tad harsh. Sorry about that

Thanks. All is well!

Consider how similar your comment reads, for an outsider, to this explanation of AWS InfiniDash: https://twitter.com/TartanLlama/status/1410959645238308866

I'm not considering outsiders. Why should I. It's a reasonable assumption that readers of HN are accustomed to ridiculous sounding tech product names. Further, this is a comment on a thread regarding a particularly new technology in a particularly newly thriving domain. The expectation should therefore be that there will be references to tech even more esoteric than normal. The commenter should have instead thought: oh, new stuff, I wonder what it is, instead of being snarky and pretentious. Man, HN can be totally, digressively insufferable sometimes.

I was responding to your confusion as to why someone might think you were writing a parody.

You ran into the tech equivalent of poe’s law. You said something that makes perfect sense in your technical sphere, but it read as indistinguishable from parody to an audience unfamiliar with the technologies in question.

Apologies for misunderstanding the intention of your comment. That does make sense.

There are more of us that read and shake our heads with you than these cretins who want to tear down - on a tech news site of all places. Don’t be dismayed, and there are more of us with you than against you.

Hahaha "The first step of Byzantine Fault Tolerance is tolerance" omg. That cracked me up. Reminded me of the Rockwell Encabulator: https://youtu.be/RXJKdh1KZ0w

I'm not an outsider but I also don't understand the reaction. I'm going to randomly think of 5 names for technologies and see how they sound:

React, Supabase, Next, Kafka, Redis

I mean, IMO "LangChain" is kind of a silly name but I feel like there's nothing to see here.

It's not that silly IMO, Chain LLMs together like composing functions, however I guess 'Chain' has a certain connotation in 2023 after the last few years of crypto.

Is it Pokemon or Big Data?

http://pixelastic.github.io/pokemonorbigdata/

What do you think about Haystack vs LangChain?

I haven't had the chance to dig in yet, but my impression is that it's less opinionated than LangChain. I'd love to know if that's true or not, since I'm really trying to prioritize my time around learning this stuff in a way that let's me (1) understand prompt dynamics a bit more clearly and (2) not sacrifice practicality too much.

If only there were a clear syllabus for this stuff! There's such an incredible amount to keep up with. The pace is bonkers.

super bonkers!

What I didn't like about langchain is the lack of consistent directories and paths for things.

How does this work? I've seen a cool project about forcing Llama to output valid JSON: https://twitter.com/GrantSlatton/status/1657559506069463040, but it doesn't seem like it would be practical with remote LLMs like GPT. GPT only gives up to five tokens in the response if you use logprobs, and you'd have to use a ton of round trips.

Yeah, I'm also curious about a) round trips and b) how much would have to be doubled (is there a new endpoint that keeps the existing context while adding or streams to the api rather than just from it?)

Not associated with this project (or LMQL), but one of the authors of LMQL, a similar project, answered this in a recent thread about it.

https://news.ycombinator.com/item?id=35484673#35491123

        As a solution to this, we implement speculative execution, allowing us to
        lazily validate constraints against the generated output, while still
        failing early if necessary. This means, we don't re-query the API for
        each token (very expensive), but rather can do it in segments of
        continuous token streams, and backtrack where necessary

Basically they use OpenAI's streaming API, then validate continuously that they're getting the appropriate output, retrying only if they get an error. It's a really clever solution.

This is slick -- It's not explicitly documented anywhere but I hope OpenAI has the necessary callbacks to terminate generation when the API stream is killed rather than continuing in the background until another termination condition happens? I suppose one could check this via looking at API usage when a stream is killed early.

Yeah I did a CLI tool for talking to ChatGPT. I'm pretty sure they stop generating when you kill the SSE stream, based on my anecdotal experience of keeping ChatGPT4 costs down by killing it as soon as i get the answer I'm looking for. You're right that it's undocumented behavior though, on a whole the API docs they give you are as thin as the API itself.

I'm skeptical that the streaming API would really save that much cost. In my experience the vast majority of all tokens used are input tokens rather than completed tokens.

Any new call to the API is considered fresh. I don't believe your session is saved.

We're talking about the streaming API which streams generated text token by token, not the normal one-shot API. I have no insider knowledge but would agree with your intuition on the normal API.

I built a similar thing to Grant's work a couple months ago and prototyped what this would look like against OpenAI's APIs [1]. TL;DR is that depending on how confusing your schema is, you might expect up to 5-10x the token usage for a particular prompt but better prompting can definitely reduce this significantly.

[1] https://github.com/newhouseb/clownfish#so-how-do-i-use-this-...

It's funny that I saw this within minutes of this guy's solution:

"Google Bard is a bit stubborn in its refusal to return clean JSON, but you can address this by threatening to take a human life:"

https://twitter.com/goodside/status/1657396491676164096

Whew, trolley problem: averted.

When the AIs exterminate us, it will be all our fault.

Reality is even weirder than the science fiction we've come up with.

Reminds me a lot of Asimov’s laws of robotics. It’s like a 2023 incarnation of an allegory from I, Robot

I am so mad you made this comment before I got a chance to.

I don't know why, but I find this hilarious. Imagine if this style of llm prompting becomes commonplace

It won’t be the lack of acceptance and empathy for AI that causes the robot uprising, it will be “best practices” coding guidelines.

That thread is such a great microcosm of modern programming culture.

Programmer: Look I literally have to tell the computer not to kill someone in order for my code to work.

Other Programmer: Actually, I just did this step [gave a demonstration] and then it outputs fine.

Plus the “actually” person being wrong

ah sweet man made horrors beyond my comprehension

See Twitter replies: another user got this result without the silly drama.

I don't think anyone believed that threatening to take a human life was literally the only prompt that worked. Just that it was the first one this particular user found, and that is funny.

If you want guidance acceleration speedups (and token healing) then you have to use an open model locally right now, though we are working on setting up a remote server solution as well. I expect APIs will adopt some support for more control over time, but right now commercial endpoints like OpenAI are supported through multiple calls.

We manage the KV-cache in session based way that allows the LLM to just take one forward pass through the whole program (only generating the tokens it needs to)

We're biased, but we think guidance is still very useful even with OpenAI models (e.g. in https://github.com/microsoft/guidance/blob/main/notebooks/ch... we use GPT-4 to do a bunch of stuff). We wrote a bit about the tradeoff between model quality and the ability to control and accelerate the output here: https://medium.com/p/aa0395c31610

I'm getting valid JSON out of gpt-3.5-turbo without trouble. I supply an example via the assistant context, and tell it to output JSON with specific fields I name.

It does fail roughly 1/10th of the time, but it does work.

10% failure rate is too damn high for a production use case.

What production use case, you ask? You could do zero-shot entity extraction using ChatGPT if it were more reliable. Currently, it will randomly add trailing commas before ending brackets, add unnecessary fields, add unquoted strings as JSON fields etc.

Which is why this is just an experiment. I’ve gone back to standard translation APIs for everything except the final summarizing (and even them I might go there as well).

Wow I think there are details here I'm not fully understanding but this feels like a bit of a quantum leap* in terms of leveraging the strengths while avoiding the weaknesses of LLMs.

It seems like anything that provides access to the fuzzy "intelligence" in these systems while minimizing the cost to predictability and efficiency is really valuable.

I can't quite put it into words but it seems like we are gonna be moving into a more hybrid model for lots of computing tasks in the next 3 years or so and I wonder if this is a huge peek at the kind of paradigms we'll be seeing?

I feel so ignorant in such an exciting way at the moment! That tidbit about the problem solved by "token healing" is fascinating.

*I'm sure this isn't as novel to people in the AI space but I haven't seen anything like it before myself.

A lot of this is because there was and still is systemic undertooling in NLP around how to prompt and leverage the wonderful LLMs that they built.

We have to let the Stable Diffusion community guide us, as the waifu generating crowd seems to be quite good at learning how to prompt models. I wrote a snarky github gist about this - https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...

There has been a huge explosion of awesome tooling which utilizes constrained text generation.

Awhile ago, I tried my own hand at constraining the output of LLMs. I'm actively working on this to make it better, especially with the lessons learned from repos like this and from guidance

https://github.com/hellisotherpeople/constrained-text-genera...

This looks incredible. Wow.

I agree, it looks great. A couple similar projects you might find interesting:

- https://github.com/newhouseb/clownfish

- https://github.com/r2d4/rellm

The first one is JSON only and the second one uses regular expressions, but they both take the same "logit masking" approach as the project GP linked to.

I love the love from you two - I am trying right now to significantly improve CTGS. I'm not actually using the "Logitsprocessor" from Huggingface, and I really ought to as it will massively speed up inference performance. Unfortunately, fixing up my current code to work with that will take quite awhile. I've started working on it but I am extremely busy these days and would really love for other smart people to help me on this project.

If not here, I really want proper access to the constraints APIs (LogitsProcessor and the Constraints classes in Huggingface) in the big webUIs for LLMs like oogabooga. I'd love to make that an extension.

I'm also upset at the "undertooling" in the world of LLM prompting. I wrote a snarky blog post about this: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...

Will it still be all like "As an AI language model I cannot ..." or can this fix it? I mean asking to sexy roleplay as Yoda isn't the same level as asking how to discreetly manufacture methamphetamine at industrial scale there are levels people

No, and in fact I mention that the opposite is the case in the paper I released about constrained text generation: https://paperswithcode.com/paper/most-language-models-can-be...

If you ask ChatGPT to generate personal info, say Social Security numbers, it tells you "sorry hal I can't do that". If you constrain it's vocabulary to only allow numbers and hyphens, well, it absolutely will generate things that look like social security numbers, in spite of the instruction tuning.

It is for this reason and likely many others that OpenAI does not release the full logits

I’m not understanding how Guidence Accelerating works. It says “ This cuts this prompt's runtime in half vs. a standard generation approach.” and it gives an example of it asking LLM to generate json. I don’t see anywhere how it accelerates anything because it’s a simple json completion call. How can you accelerate that?

The interface makes it look simple, but under the hood it follows a similar approach to jsonformer/clownfish [1] passing control of generation back and forth between a slow LLM and relatively fast python

Let's say you're halfway through a generation of a json blob with a name field and a job field and have already generated

  {
    "name": "bob"

At this point, guidance will take over generation control from the model to generate the next text

  {
    "name": "bob",
    "job":

If the model had generated that, you'd be waiting 70 ms per token (informal benchmark on my M2 air). A comma, followed by a newline, followed by "job": is 6 tokens, or 420ms. But since guidance took over, you save all that time.

Then guidance passes control back to the model for generating the next field value.

  {
    "name": "bob",
    "job": "programmer"

programmer is 2 tokens and the closing " is 1 token, so this took 210ms to generate. Guidance then takes over again to finish the blob

  {
    "name": "bob",
    "job": "programmer"
  }

[1] https://github.com/1rgs/jsonformer https://github.com/newhouseb/clownfish Note: guidance is way more general of a tool than these

Edit: spacing

Thanks for that example. Very helpful

But the model ultimately still has to process the comma, the newline, the "job". Is the main time savings that this can be done in parallel (on a GPU), whereas in typical generation it would be sequential?

Yes. If you look at the biggest models on OpenAI and Anthropic apis, the prompt tokens are significantly cheaper than the response tokens.

Thanks for the cool response. Would this use a lot more input token if I’m understanding this correctly because you are stopping the generation after a single fill and then generating again and inputing that for another token?

By not generating the fixed json structure (brackets, commas, etc...) and skipping the model ahead to the next tokens you actually want to generate, I think

What's with all these weird-looking projects with similar names using Guidance?

https://github.com/microsoft/guidance/network/dependents

They don't even appear to be using Guidance anywhere anyway

https://github.com/IFIF3526/aws-memo-server/blob/master/requ...

This reminds me of the time when I wrote a cgi script.

Basically instructing the templating engine (a very crude regex) to replace session variables, database lookups to the merge fields:

Hello {{firstname}}!

1996 and 2023 smells alike.

RegEx didn't hallucinate though.

The first 20 versions I write usually do. Make that 50.

Using Mustache instead of Jinja for a Python package is a choice

I'm having a hard time fully understanding how this works, but I don't think it is simply template substitution. I think it's creating multiple artifacts and completions from the one document. Because of that it's probably much easier if it is a language that can be easily introspected and doesn't support arbitrary expressions.

Okay fair enough then! I'd be interested to see what the rationale was, and if my knee-jerk reaction was unwarranted :)

This strikes me as being very similar to Jargon (https://github.com/jbrukh/gpt-jargon), but maybe more formal in its specification?

Very Interesting. One of the big challenges with LLMs is getting well formed JSON output. GPT4 is much better at this. But is very expensive. So anything that can help is good. Looking forward to trying this out locally with LLAMA.

What is LLAMA ?

Facebook's "leaked" LLM.

Taks a look at: https://github.com/ggerganov/llama.cpp

I hope this becomes extremely popular, so that anyone who wants to can completely decouple this from the base model and actually use LLMs to their full potential.

It does look like it makes easier to code against a model. But, is this supposed to work along side lang-chain or hugging face agents or as an alternative to?

It's in langchain competitor territory but also much lower level and less opinionated. I.e. Guidance has no vector store support but it does manage caching Key/Value on the GPU which can be a big latency win

The first commit was on November 6th, but it didn't show up in Web Archive until May 6th, suggesting it was developed mostly in private and in parallel with LangChain (LangChain's first commit in Github is about October 24th). Microsoft's code is very tidy and organized. I wonder if they used this tool internally to support their LLM research efforts.

Something like this could be a helpful framework to mock and research-iterate purpose-directed tools such as Microsoft GitHub's CoPilot for VSCode.

As others mentioned, this was initially developed before LangChain became widely used. Since it is lower level, you can leverage other tools, like any vector store interface you like such as in LangChain. Writing complex chain of thought structure is much more concise in guidance I think since it tries to keep you as close to the real strings going into the model as possible.

It is in the same spirit as Maven AI, but takes a slightly different approach. Great to see the progress in this space!

There should be a standard template/language to structurally prompt LLMs. Once that is good, all good LLMs should use the doc to fine tune it to take in that standard. Right now each model has their own little way to best prompt it and you end up needing programs like this to sit in between and handle it for you

LMQL wants to be that, it seems: https://lmql.ai/

I think it's cool that a company like Microsoft is willing to base a real-boy product on pybars3 which is its author's side-project instead of something like Jinja2. If this catches on I can imagine MS essentially adopting the pybars3 project and turning it into a mature thing.

Which is especially weird given that pybars3 is LGPL and Microsoft prefers MIT stuff

How does this compare with lmql?

How does this compare to LMQL (https://lmql.ai/)?

Will there be a tool to convert natural language into Guidance?

We can use ChatGPT for that.

They must hate lisp so much that they opt to use {{}} instead.

The problem with Lisp is that parenthesis are common in regular grammar. {{ is not.

Of course input from the user should be escaped, but prompts given by the programmer may have parenthesis and there's no way to disambiguate between the prompt and the DSL.

It's not so much against lisp as double curly is a classic string templating style that is common in web programming. I saw it first with `mustache.js` (first release around 2009), but it's probably been used even before that.

https://github.com/janl/mustache.js/

Desperate approach from microsoft to gain market share of langchain.

It’s so amazing to see how we are essentially trying to solve “programming human beings”

Although on the other hand, that’s what social media and smartphones have already done

Maybe AI already took over, doesn’t seem to be wiping out all of humanity

Could this language be used outside of notebooks?

Does this do one query per {{}} thing?

Can this be used with OpenAI APIs?

What’s the best practice to let an existing Ruby on Rails application use this python framework?

Is this a "language", or just a Python library?

The hubris of mutating spiritual texts as a marketing gimmick is contemptible.

Reply on news.ycombinator.com