>people keep trying to apply engineering principles to an inherently unstable and unpredictable system in order to get a feeling of control.
What's the alternative?
What's the alternative?
If your C compiler invents a new function call for a non-existent function while generating code, that's usually a bug.
If an LLM does, that's... Normal. And a non-event.
What other engineering domain operates on a fundamentally predictable substrate? Even computer science at any appreciable scale or complexity becomes unpredictable.
An engineer doesn't just shrug and pick up slag because it contains the same materials as the original bauxite.
We’re basically in the stone ages of understanding how to interact with synthetic intelligence.
But attempts to integrate little understood things in daily life gave us radium toothpaste and lead poisoning. Let's not repeat stone age mistakes. Research first, integrate later.
It’s reasonable to scope one’s interest down to easily predictable, simple systems.
But most of the value in math and computer science is at the scale where there is unpredictability arising from complexity.
But a lot of the trouble in these domains that I have observed comes from unmodeled effects, that must be modeled and reasoned about. GPZ work shows the same thing shown by the researcher here, which is that it requires a lot of tinkering and a lot of context in order to produce semi-usable results. SNR appears quite low for now. In security specifically, there is much value in sanitizing input data and ensuring correct parsing. Do you think LLMs are in a position to do so?
In the hands of an expert, I believe they can help. In the hands of someone clueless, they will just confuse everyone, much like any other tool the clueless person uses.