Trusteando

Code is not solved seems to reflect the idea that using AI badly is a bad idea, and that to use it well you have to be good at making software.

The real tension is in the human AI interface and there are many unknowns. Can a software engineer with weak design skill use AI to produce good code. Will the future make those requisites less important. Will productivity increase with AI stagnate even for the best. Can a new way to interface humans and AI break that wall. Will software be created in a new way using dynamic libraries that AI agents prepare to cover a large scope of problems. Nobody knows yet. The article is strong on what is closed, accountability, ownership, NFRs, slop, and silent on what is open, which is where the argument actually is.

Reading isn't the right comparison. Human memory is lossy and fades while LLM encoding is durable with no degradation. The valuable content that the author provides: the content, style, selection of topics, and more is encoded, written into the LLM weights, and they obtain profit from them (now directly, via ads). No one can compete with that kind of copying and pasting from copyright-protected material. And the scale is what hurts authors most: flooding the market with millions of copies on demand, without paying for it.

Mathematics can give for example theoretical lower and upper bounds for complexity of algorithms. And that information is very valuable for tradeoffs. For example we know that many algorithms have optimal average complexity or asymptotic behavior so they can be a good tradeoff in certain scenarios. Also knowing that some problems don't have a perfect solution allows you to accept essential tradeoffs

One question: LLM might answer correctly many logical and math problems, but I don't see any guarantee that when the LLM context receives new information the answer could get worst. Security in LLM answer is not a monotone increasing function of context size.

A cynical could say that AI could not interpolate the Dedekind cut but it could extrapolate to create a lot of money but just using an imaginary extra point.

The ironic part is that generating a joke like this requires jumping across three distant fields: real analysis, machine learning limits, and VC market cynicism. For an AI to discover that specific overlap through statistical search, the combinatorial space is absurdly huge. Yet a human brain connects them in a fraction of a second. This tiny joke is a micro-proof of the macro-argument: human conceptual leaps routinely bypass exponential search spaces.

Just to add that the Dedekind cut example seems to stand beyond any RL policy used in chess or go, AlphaZero or AlphaProof. In the classical RL there is an state-action space. If the solution requires jumping to a totally difference action space (that must be created) the local policy stalls. Dedekind cut is an example of a out-of-distribution state-space generation.

I think the author did not mention that the way humans construct the knowledge ladder requires low complexity because each step gives power to the next, but the AI with the exponential exploration can give biggers steps but then it stagnates because the next step can be beyond the exponential exploration complexity. In chess a good strategy can be the best tool, in the game of go our experience suggest the same, but it could happen that mathematical thinking requires a type of policy that could be beyond the current ideas. I can not fathom a LLM could conceive from scratch concepts like the real numbers by Dedekind.

For example he could tell you about how the LLM destroyed the main database (source of data for him) and so you should prompt the LLMs for how to avoid losing all your data. Real experiences help but not painless.

IMHO, I think that it could be better if the question about how to learn programming in the age of LLMs were asked to someone who is learning now by using LLMs. Someone who learned programming thirty years ago can perhaps give you only one side of the coin, whereas someone learning today from scratch using LLMs could give you good advice on what the real difficulties are and where the main drawbacks lie. Combining both views would give a better idea of the landscape.

Design a test to verify that the harness keeps the rider on the horse. Parameterize it by context size.