Hacker News Reader

Comment by villgax | Hacker News Reader

Should be the other way around, LLM should check against language spec to see compliance

I'm not sure what you mean.

It seems to me less of an issue that LLMs produce code that aren't compliant with language specs, but rather that LLMs produce code that do not meet the stated requirements, or that they break from earlier stated requirements as new requirements are requested.

And of course, code not meeting requirements is not a problem native to LLMs, but to all code production agents everywhere (including humans).

> or that they break from earlier stated requirements as new requirements are requested.

This nails one of the biggest issues that we're trying to help Mito users with -- code that meets __changing__ requirements. ie: the input data column headers change.

The best approach we have right now is giving users flexibility to parametrize their script (ie: tell the script the right column header or reference columns by position instead of name)

It turns out that teams that have moved their data to Snowflake tend to avoid these issues primarily because the schema of their data changes much less frequently.

That's the problem with humans as well, since we don't exactly memorise the spec, we just check & update basis errors & looking up references. But since this is an LLM we are talking about, it should be able to infer the same once given the spec of a language without needing a compiler or atleast being able to check with one via plugin or api call.

This will be very interesting.

LLM/ChatGPT is based on stochastic approach to NLP while the alternative e.g. Typed Feature Grammar (TFG) is based on deterministic approach [1]. Apparently Cuelang is based on the latter and since it's a constraint based language you can use it against the prompt (input) and also for checking the output as well [2],[3].

What it means is that LLM/ChatGPT and TFG/Cuelang can be used in concert to produce or generate codes by well crafted prompt based on the language spec and similarly also to debug based on the spec. This will probably change and enhance the discipline of software engineering as we know it for the better.

[1]Feature (linguistics):

https://en.wikipedia.org/wiki/Feature_(linguistics)

[2]The CUE Data Constraint Language:

https://cuelang.org/

https://github.com/cue-lang/cue

[3]Show HN: GPT-JSON – Structured and typehinted GPT responses in Python:

https://news.ycombinator.com/item?id=35821748

You know, I wonder if Python allows you to check code shape and if valid, eval them (yes this could be dangerous)

I wonder if Clojure can do it... I think the spec can check if function-calls and data given have valid types and reasonable unit/quantity each... Hmm.

I guess you could build an AST from the Python code and eval if it's successful. No gaurentee the code will run, of course (you could always get an error).

In general, it feels it's halting problem hard to determine if any general purpose programming language program is executable or safe... but maybe if we reduce the size of the spec under consideration?

That's an interesting idea. When converting an Excel workbook to Python, checking your work as you go is the hardest thing for us to convince people to do. (See here: https://blog.trymito.io/automating-spreadsheets-with-python-...). So tools to make that double checking process easier are really interesting to us.

I've personally never seen a Mito users with a detailed enough spec of their report that an LLM would be able to use it to check compliance -- but maybe if we built the functionality users would create them...

Reply on news.ycombinator.com