Comment by intended | Hacker News Reader

Comment by intended | Hacker News Reader

I may not articulate this perfectly, but I cannot be the only person frustrated with the state of affairs.

New features which have some random amount of error do not make me happier. They make me more afraid.

I have been grilled on single commas being missing in documents - for good reason! It was a financial model.

For me - LLMs are

- English majors - Decompression systems

They cannot conform to my expectation of code

- their output is inherently randomly distributed - they are being pigeon holed into doing logic / Thinking.

Any use case that - exposes an LLM to infinite interactions (open to users at internet scales)

Simply represents a unsolvable edge case issue to me.

Production solutions need to figure out - a way to verify the output is correct (fast, scaled, cheap!), - figure out use cases that accept uncertainty gracefully.

From what I am seeing, there is a bit of both happening.

1) Get the LLM output close enough to 99% correct, 2) send it to a human as a “co-pilot” to verify it is correct.

It is fatiguing to track every new variation on getting somewhat closer to some figure between 89 to 99.9999%.

I love the tech. I love what it actually achieves. I use it regularly to make my life better. This is not an attack on the tech.

My life would be MUCH easier, if someone could tell me their bot or solution is going to screw up massively 89% of the time, and that expert users can identify the 11% cases 100% of the time.

Also that my experts arent going to suddenly see their workload expand by infinity% if this tool goes live.

Code generation should be relatively straightforward with language model. Start with a problem description and let the LLM generate the code. It should be fairly simple to ask for a proof of correctness for the generated code, then verify that proof (for instance by creating property-based tests), and finally ask for a human-readable summary of the proof. Pass this to human code review and you have a pretty good code generation machine. Failure at any step can simply lead to another iteration.

The thing is: SE doesn't work like this anymore. People aren't used to formalized requirements and take 99% just for granted. (Which web developer really takes network errors into account nowadays?).

With "AI" we might have the tools we need to really solve the software crisis.

This is likely why Coding assistants are talked about in glowing terms. “35% increased coder productivity with co-pilot.”

This still hits the “expert required” problem. Use an LLM to code in a language or domain you have 0 experience in.

Contrast your experience with that of any explainer video where an expert in a domain uses an LLM to make an MVP.

Experts will get something up in the span of a video. They will avoid innumerable mis-steps and rabbit holes because they know code that gets to the result effectively on sight.

Non experts wont even know what part of the output is wrong.

——-

Generalize this problem away from code. Think of financial models and another version of the verification problem.

Your LLM works, it feeds signal into a financial model that predicts the future value of a firm.

The model output is dramatically out of sync with market estimates.

If an analyst showed this result, they would be told to fix their excel or tweak the assumptions. Thats the typical “verification” loop.

Let’s say hyou tune the model to get to 99.9%. I still have to run sampling /EVALS / QC all the time to make sure my output is within tolerances.

I still need to know that I can expect X% accuracy, that we have a plan for the 1-X% cases, and that this isn’t going to overwhelm the people with domain expertise in my team.

Reply on news.ycombinator.com