Okay here's something LLMs can't. They can't solve problems that are longer than ~10 pages of math. They also can't maintain codebases without supervision. It's because they are have no memory and use various tricks to supplant that fact.
My opinion of claims like "LLMs need memory to manage codebases" has also hit the dumpster bin a while ago.
Why would knowing how to make a maintainable change to a codebase require any more "memory" than knowing how to play an optimal chess move? The codebase is the memory. A sufficiently capable LLM can ingest it, figure out what changes to make, and make them.
Were that the case LLMs would’ve been phenomenal code monkeys from the get go. They were not. They still are not.
2/ SWE are doing way more than "writing code", and since those areas are less "computational verifiable" (what is a good architecture that will stand in 5y?) and more "connected to real world" (what are the requirements?), LLMs struggle with them