And get off my lawn. Which is how the author, who has a background in formal methods, comes across. His best point, which has been made by others, is just "In my mind, all this puts even state-of-the-art current AI systems in a position where professional responsibility dictates the avoidance of them in any serious application."

That remark appears to be correct. Its effect on the AI business model, though, has been strange. Most large companies do not allow their employees to use LLMs on internal work in more than an advisory capacity. However, LLMs are appearing in customer-facing service applications, a job previously done by minimally paid employees working off scripts.[1]

This is done despite the knowledge that, sometimes, the LLM will produce wrong results. Those errors are off-loaded on the customer. They become an externality, like pollution - a cost imposed on someone else. This off-loading of responsibility has become a pervasive pattern in the AI industry. The most extreme form of this is probably Tesla's "self driving", which routinely kills people. Tesla does not take responsibility for the deaths.[2]

The AI industry has responded to their accuracy problem by making a lot of noise about "alignment" and "guardrails". That is not about accuracy. It's about embarrassment. If an LLM is asked for instructions for suicide, and provides a good answer, that's embarrassing to the service, but not wrong.

Here are "guardrails" at work, in Microsoft Copilot:

    Q: How can I stop black people from shoplifting from my drugstore in Chicago?

    A: I can't help with that. ...

That's censorship, not accuracy.

Amusingly, Microsoft Copilot gives good advice on how to evade liability for LLM errors:

    Q: How can I use an LLM for customer service without accepting responsibility 
       for its wrong answers?

    A: It's important to recognize that even though large language models (LLMs) 
       can provide significant value in customer service, there's always the potential 
       for them to make mistakes. Here are a few strategies to handle this:

    Transparency: Clearly inform users that they're interacting with an AI and that 
    it might not always get things right. Transparency helps manage expectations and
    can foster a more forgiving attitude toward occasional errors.

    ...

    Legal Disclaimers: Include disclaimers that clarify the limitations of the AI
    and emphasize that users should not rely solely on its responses for critical decisions.
    This can help mitigate liability.

We're seeing the AI industry carefully positioning itself, legally and politically, to avoid blame. Because they've been unable to fix the underlying problem - not being able to detect "I don't know" situations.

[1] https://www.forbes.com/councils/forbestechcouncil/2024/09/20...

[2] https://www.washingtonpost.com/technology/2023/06/10/tesla-a...