I always get the feeling that fundamentally our software should be built on a foundation of sound logic and reasoning. That doesn't mean that we cannot use LLMs to build that software, but it does mean that in the end every line of code must be validated to make sure there's no issues injected by the LLM tools that inherently lack logic and reasoning, or at least such validation must be on par with human authored code + review. Because of this, the validation cannot be done by an LLM, as it would just compound the problem.
Unless we get a drastic change in the level of error detection and self-validation that can be done by an LLM, this remains a problem for the foreseeable future.
How is it then that people build tooling where the LLM validates the code they write? Or claim 2x speedups for code written by LLMs? Is there some kind of false positive/negative tradeoff I'm missing that allows people to extract robust software from an inherently not-robust generation process?
I'm not talking about search and documentation, where I'm already seeing a lot of benefit from LLMs today, because between the LLM output and the code is me, sanity checking and filtering everything. What I'm asking about is the: "LLM take the wheel!" type engineering.