The sanding down you refer to is what generates those tests and documentation.
Are you suggesting that LLM's can't test for people who use screen readers? Keyboard only users? Slow network requests?
You're acting like the issues an app faces are so bespoke to the actual app itself (and have absolutely no relation to existing problems in this space) that an LLM couldn't possibly cover it. And it's just patently wrong.
If you disagree with that, I think the onus is on you to show me that an LLM could simulate the full context in which a user interfaces with software. That's a ridiculous claim.
Feel free to show literally any evidence for this claim.
lol And you made the claim, not me. The proof is on YOU.
The status quo is that this capability does not exist. Whoever makes a claim contradicting the status quo has the burden of proof. I can't prove a negative.
And even with your logic, I did not make the original claim, it was made by simon.
Your statement now also makes little sense. For any nontrivial software project, the usage patterns and interactions with other systems are complex enough that the code itself does not contain enough context to understand how it is used, or what the invariants are.
There may be very simple codebases where an LLM can actually give you "thorough documentation" or "robust tests", but those are rare.
Its not rare. I've built 2 dozen line-of-business apps in it last handful of years that were glorified CRUD apps. Every environment I've been in has had a mix of the 2.
And even then, that's at odds with your absolute above. On top of being in a field that changes daily.
I wasn't really going for an exact, formal statement, but I can give you a formal interpretation of what I said above, if you want to be pedantic.
In general, you can't expect an LLM to produce thorough documentation or robust tests for nontrivial software, because the use of those software (i.e. how their interfaces are expected to behave) contains assumptions from the context in which they are used, and that information will not be encoded in the source.
If the above was somehow ambiguous, this should be clear and uncontroversial.
That is in fact what I did and if you meant otherwise, then yes I agree that currently there are plenty of cases in which those tools fall short and will never replace a human.
I don't think it's feasible to fully simulate the full depth of actual usage, given that (especially in the case of screen readers and the like) there's a great deal of combinatorial depth and context to the problem. Which screen readers, on which operating systems, and which users thereof?