Hacker News Reader

Comment by codegladiator | Hacker News Reader

> less common DSLs like liquidsoap’s stream management DSL

seems to be on github since 2008 so definitely in the training data. i am not talking about less or more common. either "your dsl" would need to look something like someone elses dsl (at this point is it your dsl?) or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt.

> LLMs respond well to clear, simple structure

and what a "clear simple structure" for a dsl is also quite not mentioned. clear and simple would be quite subjective based on the domain, the article says let the llm go in a loop trying to figure out the dsl for you.

> checkpoints that have more structure than natural language

if llm is at any point in the structured generation part then either you have a deterministic validator/compiler or you are back to reading/reviewing it manually, what can you trust ?

View on HN

> or you need some way to get your dsls examples in the training data for the llm, or feed it in the prompt.

I'm really not sure this is true. Recently for work I've been making changes in a system written in tcl in the early 2000s, including a custom dsl that has never escaped this company. The LLMs can write it fine. It wrote it almost acceptably on first sight, an hour or two of targeted test cases to extract a one page functional spec of the subset we needed and that was it. They are quite good at finding and fitting to patterns, go figure.

> The LLMs can write it fine. It wrote it almost acceptably on first sight

you are probably talking of some coding harness which looked up the existing code base and then made it and not something like first prompt to llm "write xyz testcase in my systems company dsl in tcl from early 2000".

> LLMs can write it fine

sure, coding by examples is fine (it goes back to "system prompt describing the language"). but the claim we are arguing is reliability. did the llm generate 100% test case or code after reading your existing codebase, likely not as you mention it "almost acceptably on first sight".

i would probably be in denial if was suggesting llms not good at finding and fitting patterns.

Well yes of course I was using a harness lol. We're talking about the application of tools to engineering problems. Getting an LLM to cold start on a completely novel to it language without prompting is a different problem that I don't have.

The whole point is that a dsl syntax is constrained enough to fit in the context window and not rely on training data. LLMs are damn near built for this exact use case.

For open source llms you can give your grammar to the llm to help it decide what valid next tokens it can generate.

Re: less common, I was just saying it doesn’t take millions off examples like PlanetUML.

> what can you trust

I wasn’t clear enough here — you’re responding to DSLs as an interface from non-deterministic LLMs to deterministic external systems.

What I meant was using DSLs as intermediate checkpoints in multi-LLM processing. If you just flow natural language through 5 LLM calls, the last one may be getting something very different from what it’s prompt is designed for.

But if you make the DSL a contract for handoff, results are much more stable.

Perfect and deterministic? No, of course not. Just an improvement and mitigation. But it’s served me well.

its served me okay.

> Perfect and deterministic? No, of course not. Just an improvement and mitigation.

exactly, not reliable, as the article tries to portray

Interesting … do you see reliable == infallible? I don’t. But certainly all software is imperfect, and LLMs doubly so.

for me reliable = deterministically repeatable. if a llm has been able to do a task successfully and i ask it to do the same task again, i want the reliability that i get the same outcome (be if failure or success)

like if someone says "is this car reliable", i dont expect an infallible car. but if the cars third gear is 'you know sometimes it doesnt work', i wouldnt take that car out of city.

This seems too strong to me. I agree with reliable as "repeatable", it's the determinism which seems excess to purpose.

If I have a great meal every time I go to a restaurant, that chef is reliably good. I do not require that the tomatoes on the salad be of identical number or placement, or cut along the precise same bias. Nor would I count the sesame seeds on the bun.

Are LLMs reliable by that standard? Sometimes yes, for some tasks. The envelope of tasks for which they're reliable is continuing to grow.

Reply on news.ycombinator.com