Hacker News Reader

Comment by d_silin | Hacker News Reader

I understand the frustration, but factually it is not true.

Engines are reliable to about 1 anomaly per million flight hours or so, current flight software is more reliable, on order of 1 fault per billion hours. In-flight engine shutdowns are fairly common, while major software anomalies are much rarer.

I used LLMs for coding and troubleshooting, and while they can definitely "hit" and "miss", they don't only "miss".

View on HN

I was actually comparing aviation HW+SW vs. consumer software...and making the point that an old C++ invoices processing application, while being way less reliable than aviation HW or SW, is still orders of magnitude more reliable than LLMs. The LLMs don't always miss, true...but they miss far too often for the "hit" part to be relevant at all.

They miss but can self correct, this is the paradigm shift. You need a harness to unlock the potential and the harness is usually very buildable by LLMs, too.

Hm, that is a a lot of generic talk - but very little concrete data and examples.

Concrete examples are in your code just as they're in my employer's which I'm not at the liberty to share - but every little bit counts, starting from the simplest lints, typechecks, tests and going to more esoteric methods like model checkers. You're trying to get the probability of miss down with the initial context; then you want to minimize the probability of not catching a miss, then you want to maximize the probability of the model being able to fix a miss itself. Due to the multiplicative nature of the process the effect is that the pipeline rapidly jumps from 'doesn't work' to 'works well most of the time' and that is perceived as a step function by outsiders. Concrete examples are all over the place, they're just being laughed at (yesterday's post about 100% coverage was spot on even if it was an ad).

Reply on news.ycombinator.com