Hacker News Reader

My own attempt at "chain-of-code with a Prolog DSL": https://news.ycombinator.com/item?id=45937480. Similarly to CodeAct the idea there is to turn natural language task descriptions into small programs. Some program steps are directly executed, some are handed over to an LLM. I haven't run any benchmarks yet, but there should be some classes of tasks where such an approach is more reliable than a "traditional" LLM/tool-calling loop.

Prolog seemed like a natural choice for this (at least to me :-), since it's a relatively simple language that makes it easy to build meta-interpreters and allows for a fairly concise task/workflow representations.

Nice, I do like the direction. A prolog dialect does seem like a natural choice if we must pick only one kind of intermediate representation, but ideally there could be multiple. For example, I saw your "legal reasoning" example.. did you know about https://catala-lang.org/ ? I think I'd like to see an LLM experiment that only outputs formal specifications, but still supports multiple targets (say prolog, z3, storm, prism, alloy and what have you). After you can output these things you can use them in chain-of-code.

Anyway the basic point being.. it is no wonder LLM reasoning abilities suck when we have no decent intermediate representation for "thinking" in terms of set/probability primitives. And it is no wonder LLMs suck at larger code-gen tasks when we have no decent intermediate representation for "thinking" in terms of abstract specifications. The obsession with natural-language inputs/intermediates has been a surprise to me. LLMs are compilers, and we need to walk with various spec -> spec compilers first so that we can run with spec -> code compilers

Thank you, https://catala-lang.org/ looks very interesting. I've experimented a lot with LLMs producing formal representations of facts and rules. What I've observed is that the resulting systems usually lose a lot of the original generalization capabilities offered by the current generation of LLMs (Finetuning may help in this case, but is often impractical due to missing training data). Together with the usual closed world assumption in e.g. Prolog, this leads to imho overly restrictive applications. So the approach I am taking is to allow the LLM to generate Prolog code that may contain predicates which are interpreted by an LLM.

So one could e.g. have

is_a(dog, animal). is_a(Item, Category) :- @("This predicate should be true if 'Item' is in the category 'Category'").

In this example, evaluation of the is_a predicate would first try to apply the first rule and if that fails fallback on to the second rule branch which goes into the LLM. That way the system as a whole does not always fail, if the formal knowledge representation is incomplete.

I've also been thinking about the Spec->Spec compilation use case. So the original Spec could be turned into something like:

spec :- setup_env, create_scaffold, add_datamodel,...

I am honestly not sure where such an approach might ultimately be most valuable. "Anything-tools" like LLMs make it surprisingly hard to focus on an individual use case.

> While an error rate of 1-in-1,000 seems low, [...], on a task that requires successful execution of thousands of steps in a row, such a system results in inevitable failure.

This is also why (edit: non-LIDAR) FSD cars are an illusion.

Waymo has driven a hundred million miles and is far safer than human drivers.

Yes because they use non-AI techniques (LIDAR) to prevent them from bumping into things.

I should have said non-LIDAR in my comment, yes.

I think you're still in the edit window, FYI. (At least for a few more minutes.)

LIDAR doesn’t stop them bumping into anything. LIDAR is a sensor, it doesn’t recognise anything or make decisions about steering, acceleration, or braking.

You need more than the sensor, obviously. The point is that you don't need any AI to make a system this way that is substantially safer than a system based only on camera feeds and AI.

Do you think Waymo using LIDAR means that Waymo aren’t using AI? Waymo are using AI.

That's not what I'm saying at all. Waymo uses LIDAR to ensure safety. They use AI for most of the rest.

> > While an error rate of 1-in-1,000 seems low, [...], on a task that requires successful execution of thousands of steps in a row, such a system results in inevitable failure.

> This is also why (edit: non-LIDAR) FSD cars are an illusion.

In this scenario, Waymo’s AI is executing thousands of steps in a row. The fact that it uses LIDAR for sensing doesn’t change that. It’s still AI driving you around no matter what its eyes are made of.

Waymo is a counterexample to the point you were making and their use of LIDAR doesn’t change that.

No because safety is guaranteed by the LIDAR and navigation is done by GPS+classical algorithms. Mistakes made by the AI can be overcome by those two non-AI approaches + reiterating the AI-based steps.

LIDAR cannot possibly guarantee safety. It is a sensor.

That's like saying an algorithm cannot guarantee safety because it is not an actuator.

An algorithm controls the actuator. The sensor does not control the algorithms or the actuators.

Look, it is a hell of a lot safer to use an input that does not hallucinate objects than an input that does.

FSD isn't, and never was, a sensor problem. It's an AI problem. Always was. Always will be.

Humans drive around with two mid-tier cameras on a pivot mount. Which means that any sufficiently advanced AI can do the same.

When a FSD car gets into an avoidable collision, you dump the blackbox data, and what do you see? You see that the cameras had it. All the information the car needed to avoid a collision was right there in the visual stream. The car had every bit of information it needed to make the right call, and didn't make the right call.

You can acknowledge that, and focus on building better AIs. Or you can neglect AI altogether, and have a car with 6 LIDARs drag a pedestrian, because it had all the sensor coverage but zero object permanence.

> Humans drive around with two mid-tier cameras on a pivot mount. Which means that any sufficiently advanced AI can do the same.

Yes, if we can get their error rate below 0.000001%. Until we get there, sensors + old school computer-vision provide safety.

What makes you think that "sensors + old school computer-vision" gives you an error rate better than "completely fucked"?

In simple terms, the LIDAR sensor will allow you to do "If object at X, don't go to X". But obviously, you need more than that. Old school Kalman filters for object tracking etc.

Raw dog LIDAR and "old school kalman filters" don't give you anywhere near good enough performance.

Want to know how poor performance looks like in practice? Like Tesla phantom braking but ten times worse. And if you dial it down to avoid false positives, then it stops exerting any control over AI, and you're back to getting your AI to work well.

> Or you can neglect AI altogether, and have a car with 6 LIDARs drag a pedestrian, because it had all the sensor coverage but zero object permanence.

False dichotomy, much?

No, it's not false dichotomy. It's Cruise.

I am annoyed to no end by all the LIDAR wankery - while in practice, LIDAR systems don't provide much of an advantage over camera only systems, and consistently hit the same limitations on the AI side of things.

Nonetheless, there is no shortage of people who, for some reason, think that LIDAR is somehow a silver bullet that solves literally everything.

So why do you think the only reliable FSD car out there is built around an expensive LIDAR system?

LIDAR may not solve everything, but the point is that it allows for greater safety margins. All the non-safety-critical parts can be done with AI, yes.

Reliable?

It's interesting that this is a level of tech reductionism that is really common right now and that it's not more openly challenged by engineers. Tough intractable problem? AI. How close are we? Soon! How soon? I don't know, I don't work on that problem.

Of course FSD is solvable with advanced AI and the same applies to all other problems but we don't yet have this level of AI and we don't know how far away we are from reaching it.

Companies that bet on assistive AI solutions (i.e. more sensors to plug AI gaps) will win and have the best chance at eventually reaching the level of AI where additional sensors are no longer needed.

Companies that go all-in on perfect AI have a very, very high chance of failure, not because they're not smart enough, or not driven enough, or are capital constrained, but because they dont fully understand the scope of the problem.

Also worth noting they are heavily incentivised to pump the AI bubble for existential reasons, and so their AI progress forecasts are not trustworthy.

"Reductionism" is right. If you could just always "plug the gaps with more sensors", then the car with 900 cameras and 400 LIDARs would have reached L4 autonomy back in year 2010.

It doesn't work like that. No amount of sensors can salvage piss poor driving AI. The gains from more and better sensors bottom out pretty fast. You completely saturate your ability to sink and fuse sensor data long before your driving actually gets good.

Of course, we struggle to get humans to low error rates on large number of steps in sequence too, to the point where we devote vast amount of resources to teaching discipline, using checklists, doing audits and reviews to coax reliability out of an unreliable process.

So nobody should be surprised that this also applies to LLMs.

The issue is when people assumes that a zero failure rate, or even close to zero, is necessary for utility, even though we don't need that from humans for humans to be useful for complex tasks.

For a whole lot of tasks, the acceptable error rate boils down to how costly it is to work around, and that is a function of the error rate, consequence of an error that slips past, and the cost of a "reliable enough" detector to let us mitigate to whatever extent is cost effective by using one or more detection steps.

For a lot of uses, voting or putting the AI in a loop, produces a good enough results cheap enough. For some it will require models with lower error rates first.

For some applications, sure, maybe solvers will be part of that, or in the mix. As will a lot of other tools. E.g. Claude likes to try to bisect when I ask it to fix a parser problem, and Claude is really bad at doing sensible bisection, so I had it write a dumb little bisection tool instead, and told it steps to solve this type of problem that includes using that tool. So when we can have planning steps output "microsteps" that we can automate with more deterministic tools, then we absolutely should.

Heck, the models themselves "likes" to write tools to automate if you give them long lists of tedious little tasks to do, to the point it's effort to make them not do it even when they have to write the tools themselves.

For a comprehensive rebuttal to this point of view, you may be interested in the works of W. Edwards Deming.

“No one knows the cost of a defective product - don't tell me you do. You know the cost of replacing it, but not the cost of a dissatisfied customer.” -Deming

No, I would not, as this argument is entirely irrelevant and doesn't address what I said.

> The issue is when people assumes that a zero failure rate, or even close to zero, is necessary for utility, even though we don't need that from humans for humans to be useful for complex tasks.

This argument doesn't carry because it is beside the point. Human vs. LLM utility parity isn't a sensible stop-goal for improvement. New technology isn't adopted for its legacy parity. Nor are there any specific technical barriers around human parity.

Fewer mistakes than humans, by definition, delivers unique value. People also want to spin up LLMs to handle tasks at scale in ways humans never could, where human level mistakes would be unacceptable.

So we very much do need LLMs (or whatever we call them tomorrow) to operate with lower error bars than humans. It is a reasonable demand. Lots of applications are waiting.

Given that demand, the value of avoiding any mistake, and the many people working on it, error rates will keep falling indefinitely.

> This argument doesn't carry because it is beside the point. Human vs. LLM utility parity isn't a sensible stop-goal for improvement. New technology isn't adopted for its legacy parity. Nor are there any specific technical barriers around human parity.

This is just utter nonsense. New technology is sometimes adopted because it is better, but just as often adopted even when the quality is strictly worse if it is cheaper.

But apart from that you appear to arguing against a point I never made, so it's not clear to me what the point of your response is.

> Fewer mistakes than humans, by definition, delivers unique value.

Yes, but that is entirely irrelevant to the argument I made.

> Given that demand, the value of avoiding any mistake, and the many people working on it, error rates will keep falling indefinitely.

And this is also entirely irrelevant to the point I made, and not something I've ever argued against.

> when the quality is strictly worse if it is cheaper

True. I stand corrected.

> we struggle to get humans to low error rates on large number of steps in sequence too

Who said anything about AI vs humans? The contest in this context would be AI vs classical deterministic code, algorithms, solvers

> how costly it is to work around .. a function of the error rate, consequence of an error that slips past, the cost of a "reliable enough" detector.. produces a good enough results cheap enough.

I mean, you're right, but only sort of. Someone can use this same argument to justify the assertion that bogosort is really the pinnacle of engineering excellence. How would you respond?

> Who said anything about AI vs humans?

I did, because it is a relevant comparison.

> The contest in this context would be AI vs classical deterministic code, algorithms, solvers

No, it is not. In cases where we know how to solve things that way, we probably should, on the assumption that if they can deliver good enough results they are likely cheaper.

Those are not the things we generally are trying to use LLMs for.

> I mean, you're right, but only sort of. Someone can use this same argument to justify the assertion that bogosort is really the pinnacle of engineering excellence. How would you respond?

That it is an obivously specious argument, because we have clearly lower cost sort algorithms, and so no, you can't use this same argument to justify that assertion.