We should probably cut this off, but let me try to explain what I had in mind, which might be none sense, but was certainly not well explained. Too briefly, presumably we can agree that it’s even hard for experts to check and fix code created by other humans effectively. Now put this in a dangerous domain. I originally said aviation, but (confusingly) switched to medicine for reasons not worth going into, but we can replace “program” with “treatment regimen” and “expert programmer” with “specialist physician”. The fail I was reaching for is where what is hallucinated by the LLM looks really convincing (has great grammar), and is close enough to correct that it isn’t glaringly wrong - and these models are good at doing those two things! - but by virtue of the fact that they can’t reason, they aren’t actually “thinking things through” - which is why, of course, you need the programmer/specialist. But the hallucination is so good (yet wrong, per unreasoned) that users (patients) are taking the word of the model. You think that can’t happen, but we do it all the time with results from “Dr Google”. A friend who is a doctor says that patients regularly come in with inch think binders of stuff they printed off the web that he has to talk them down from. We’re headed for a near future where instead of being talked down from stuff you merely printed off, you’ll have to be talked down from an extremely convincing wrong treatment hallucinated by Dr HAL. Anyway…never mind.