This reminded me of a conversation I recently had at work. Someone was turning over a project to me so he was explaining the context and history of the technical design decisions they've gone through. One stage of the project was basically linting through someone else's source code, evaluating possible code branches.

"So, we want to know all the possible states this could go through. We gave that task to OpenAI with the following prompt."

He shows me a sample prompt they would've sent to OpenAI. It started with the premise "You are a senior Ruby developer with ten years experience. Read through this source file and give me a list of all the code paths which involves global variables".

"Really?" I said, amused. I knew ahead of time that this project they were turning over is LLM-heavy. I have misgivings about LLMs but I try to keep an open mind so it was really impressive that it was used at this stage. "How well did it work?"

"It didn't. Instead we parsed it to a syntax tree and just performed static code analysis."

(Usual disclaimer that details of the anecdote were changed/obfuscated but the spirit of the interaction has been preserved.)