SudoLang: A Powerful Pseudocode Programming Language for LLMs | Hacker News Reader

SudoLang: A Powerful Pseudocode Programming Language for LLMs

medium.com

I've just been using Python "pseudocode" with good success. It's great for getting structured output from GPT3. Works less well with chat-fine tuned models, hopefully openai releases a simple instruct-tuned GPT4 instead of chat-tuned.

For instance, this prompt:

    best_artists = search(year=2014, genre="hip-hop")
    assert(len(best_artists) == 10)
    for i, artist in enumerate(best_artists):
        print(f"{i}. {artist}")

    Stdout:

Yields this output:

    0. Kendrick Lamar
    1. Drake
    2. J. Cole
    3. Logic
    4. Childish Gambino
    5. Joey Bada$$
    6. Chance the Rapper
    7. Schoolboy Q
    8. Vince Staples
    9. Run the Jewels

Note how the `assert` is used to constrain the output. The `search` method is not defined but what it does is obvious so the LLM hallucinates its output.

Why? What is wrong with "10 best hip hop artists of 2014, as a numbered list, only names"

I'm not saying your approach is wrong or anything, but I am trying to understand the value you get from it? I get almost the same output, modulo different ranking in the later ranks. (and that'll happening anyways, ChatGPT is deliberately not deterministic)

10 hip-hop artists is just an example here to demonstrate the general principle. You can do much more complicated stuff with nested loops, conditionals, recursion, etc that are very verbose or difficult to express unambiguously in English.

I believe that, absolutely. I just can't see any examples that exercise this outside of toy examples. (And I have severe doubt LLMs will reliably evaluate those complex examples by themselves. You're better off adding memory and agentic behavior via a formal language wrapper)

Consistency is apparently one current challenge.[1]

[1] https://news.ycombinator.com/item?id=35416637

I might do something like this in SudoLang:

list top 10 hip hop artists from the year 2014 |> sortBy(chart performance, descending)

This example will probably include detailed justifications. I liked the response, but if you just want rank and name, just say so in natural language.

This feels like the future of programming to me. When trying to teach kids to code, a common complaint I've gotten is how rigid existing programming languages are. They mistype a little thing and then get some general error, and it immediately turns a creative and fun process into something that's frustrating for them.

After things evolve a bit. Pseudocode programming languages are going to be a lot of young peoples first programming language. I know something like this integrated into Roblox Studio would instantly hook my 13 year old nephew.

This is an excellent observation. I completely agree, many of my peers when I was a child gave up on programming due to the rigid rules. This should also open up the window of cognition to more creative programmers that do not want to be bound by details such as "missing ;"

So... you use a formal language to create a system that ~handles freeform language, which you then instruct to interpret a semi-formal language?

This might well be useful, but I'd like to see one example of what you can do in SudoLang that you can't do equally well in natural language. Especially given the token cost of the prompt.

To the OP: you're getting a lot of discouraging comments here. But I had the same idea as you and just didn't take the time. I don't know if it's actually a useful idea or not, but I really do feel it has potential to be useful. As I see it, it's about creating a high-level language that is just concise enough to define what you're doing, without requiring more detail than necessary for the logic, and that can be transpired by GPT into any target language one would want. Recognizing that some fixing of the output may be necessary.

So I'm glad you're doing it and I'll keep checking it out to see where you go with it (if indeed you do feel like continuing). Good luck.

You're just seeing people's knee jerk reaction to new ideas here.

What you don't see in these comments is that SudoLang went viral overnight. Tens of thousands of people heard about it and the blog post comments are overwhelmingly positive.

As a programming assistant, SudoLang is already saving me time producing traditional software in JavaScript, but the real unlock is defining unambiguous requirements for complex programs that run directly on the LLM: Chatbots or productivity tools that require NLU - programs with enough complexity that a freeform natural language prompt just doesn't express easily (see the examples folder).

SudoLang meaningfully extends our ability to clearly communicate complex ideas with LLMs, and it's a language that LLMs and programmers and beginners learning to code all intuitively understand.

It opens up a middle ground between natural language and formally structured language and that does in fact open up new possibilities.

The people comparing the syntax of one-liners are missing the point. If you can clearly communicate the idea in natural language, just do that. In fact, that recommendation is right inside the style guide. But you can also do things like declaratively communicate the structure of objects for talking with REST APIs, clearly specify behaviors and requirements, etc.. things that need to be clearly specified and not just invented/improvised/hallucinated by the LLM.

SudoLang's infinitely inferable standard function library with modifiers can be easily and declaratively chained to radically simplify chain-of-thought reasoning: a process that helps LLMs think about tasks more carefully and produce better results.

It's OK that lots of people don't see the value, yet. It often takes time to warm up to new ideas. When I first started using GPT-3 in 2020 nobody believed me when I said it could code. Today, coders who employ AI tools are 2x faster than those who don't (this stat comes from a study on GitHub copilot comparing 95 devs working on the same task).

Thanks for your encouraging remarks.

This feels like inventing horseshoes for cars. The author asks ChatGPT for its advantages over natural language interactions, and the ever agreeable ChatGPT comes up with this:

> Firstly, SudoLang provides a more structured and consistent syntax than free-form natural language interactions. This can make it easier to understand and modify code

> Secondly, SudoLang is designed specifically for interacting with LLMs, which means it can take advantage of their unique capabilities, such as generating code, solving problems, and answering complex questions

> Finally, SudoLang includes features like modifiers and template strings that allow for more precise control over the responses and outputs generated by the LLM

But none of these is an actual advantage.

1. Formal languages' restrictiveness is a disadvantage with an LLM. It artificially limits the LLM to acting like a traditional GPPL. With an LLM, when you need to express something that would break the mold of a formal language, you can simply express it: There's no need to introduce a new version of the language spec or search Stack Overflow.

2. "Designed specifically for" isn't an advantage if the design is poor. LLMs were already designed specifically for interacting with natural language, which already lets the user "take advantage of [LLMs'] unique capabilities," as ChatGPT puts it.

3. Modifiers and template strings aren't useful because, again, LLMs don't have the constraints of formal languages. In fact, as ChatGPT points out in the article, "an LLM (large language model) does not need to be given the specification of SudoLang in order to interpret SudoLang code." In other words, my naive request to ChatGPT to "mail merge" a set of names into some text is as valid as SudoLang's template strings, without having to learn their syntax.

It's an interesting experiment, but I don't think it makes sense beyond that.

To me this looks like an experiment that demonstrates primarily : GPT4 as an assistant would happily translate user’s ideas into a lot of text.

I see the point of this language as it may give you more consistent responses because everyone writes different quality prompts, but I rather just learn to write better prompts you know?

Or just use a real language you know ?

It's an ad, self-promotion.

I wonder, for Chinese speakers or even learners, we could do AI prompting as needed with ChinesePython[0] even though it is not pseudocode but actually a running environment.

0: https://chinesememe.substack.com/i/103754530/chinesepython

Fibonacci program |> transpile(JavaScript):length=very concise const fibonacci = n => (n <= 1 ? n : fibonacci(n - 1) + fibonacci(n - 2));

A concise Fibonacci function in js, don’t explain. const fibonacci = n => n <= 1 ? n : fibonacci(n - 1) + fibonacci(n - 2);

prompt is here https://github.com/paralleldrive/sudolang-llm-support/blob/m...

This seems misguided. Or at least, I can't see the value of it. It just seems like a classic JS community take - shape a thing so it fits with the familiar ecosystem, constrain it with structure and types and such, and keep grinding.

I think the author misses the point by a million miles: part of the attractiveness of LLMs is that you don't have to bother with a formally structured language to achieve results. GPT effortlessly bridges the gap between programming and human languages - making some structured prompt language seems.. pointless. Almost like an artifact of a mind that finds comfort in constraint, at a time when we're given freedom to roam.

Language without formalization is unreliable; in any critical domain, clarity is not optional. Law, maths, code.

The second you start incorporating LLMs in your product chain, you need the abilities to a) efficiently instruct them without wasting tokens and b) interpret their outputs according to the structure required downstream.

When you're just playing, sure, you can talk to it like a toddler would, not like a lawyer/mathematician/programmer. When you actually want to create the kind of software that GPT suddenly enables, you need to come up with an interface between it and, well, other software. I'm not saying SudoLang is the adequate solution to that, but I wouldn't say that it "misses the point by a million miles". That's quite the statement there.

Instead of relying on one big model (and all it's flaws), aren't you better off having separate smaller models? Better for auditing

"Language without formalization is unreliable; "

What makes you believe LLMs will stick to the rules of your formalized language? It's likely. There are only probabilistic guarantees, which kind of obviates the benefits of a formal spec.

If you need formalism, wrap it into a formal language for the formal parts. But assuming that a formal language spec prompt avoids probabilistic outcomes? Yes, that is missing it by a million miles. You're missing the strength of LLMs. (Which is unstructured input->semi structured output)

Unit tests. SudoLang has a port of the Riteway unit testing framework, and programs can be stepped through in debug mode to show intermediate processes. Such debugging sessions could be used to fine train future models on algorithmic thinking.

This stuff is experimental and not 100% yet, but it does mostly work today, just days after SudoLang was specified. And you don't even need to paste the spec: the whole language is inferable by GPT-4.

> Language without formalization is unreliable; in any critical domain, clarity is not optional. Law, maths, code.

That's the purpose of plugins/LangChain. For deterministic tasks, don't use an LLM.

Whatever you use for process orchestration should be able to make use of the LLM where it's a good fit and something else where it's not.

> you need the abilities to a) efficiently instruct them without wasting tokens

Plain old brevity. There's no need for semicolons and parentheses and the level of punctiliousness that a formal language imposes.

> b) interpret their outputs according to the structure required downstream.

You can include downstream structure in your prompt. Still no need to design a language for it. LLMs understand imperatives like "structure your output like this."

You're both right

a picture speaks a thousand words; a formula speaks a thousand pictures

Electromagnetism is described with formulas. And yet, once you have the formulas, we use natural language to give meaning to the constituent semantic elements in the context of our reality. Plug the appliance into the 120V outlet. How much of what we do is just organizing pre-built components?

Coding has a technical debt problem, but that's caused by human conflict of interests / principle agent problem. We'll have the Maxwell's Equations of CRUD apps soon enough and the Tesla/Edisons with the applications will follow shortly after that.

Reply on news.ycombinator.com