Hacker News Reader

> Models are very good at generating locally coherent code and very bad at maintaining global architectural integrity across iterations.

Is there any published work on this? Or just more of an observed accusation?

View on HN

No single paper nails that exact claim. SWE-bench Princeton does show that models struggle significantly with real-world issues requiring changes across multiple files and functions which points in that direction. But the local vs global framing is mostly practitioner-observed, not a formally tested hypothesis yet. Fair point, I should have hedged it. https://arxiv.org/abs/2310.06770

Reply on news.ycombinator.com