Hacker News Reader

Comment by nradov | Hacker News Reader

The fact that GPT3 works at all for coding indicates that our programming languages are too low level and force a lot of redundancy (low entropy). From a programmer productivity optimization perspective it should be impossible to reliably predict the next statement. Of course there might be trade offs. Some of that redundancy might be helping maintenance programmers to understand the code.

View on HN

I think this is kind of true, but also kind of not true. Programming, like all writing, is the physical manifestation of a much more complex mental process. By the time that I know what I want to write, the hardwork is done. In that way, you can think of co-pilot as a way of increasing the WPM of an average coder. But the WPM isn't the bit that matters. In fact almost the onlt thing that matters are hte bits you won't predict.

Indeed, in the limit of maximal abstraction, i.e. semantic compression, code becomes unreadable by humans in practice. We can see this in code golf competitions.

Let me rephrase:

> The fact that GPT3 works at all for English indicates that English is too low level and forces a lot of redundancy (low entropy).

I don't think the goal is to compress information/language and maximize "surprise".

That would only follow if we were trying to optimize code for brevity, and I have no clue why that would be your top priority.

I have indeed seen codebases where it seems like the programmer was being charged per source code byte. Full of single letter methods and such - it takes a large confusion of ideas to motivate such a philosophy.

Not at all. Brevity (or verbosity) is largely orthogonal to level of entropy or redundancy. In principle it ought to be possible to code at a higher level of abstraction while still using understandable names and control flow constructs.

I’m really not sure about that. By definition, high entropy means high information density (information per character). So with the same amount of information you would have less characters.

Think in terms of information density per symbol on the abstract syntax tree, not per character in the source code.

I write code directly as gziped text. Saves lots of key strokes and you can use off the shelf compilers by piping through zcat first

>From a programmer productivity optimization perspective it should be impossible to reliably predict the next statement

Why? 99.9% of programming being done is composition of trivial logical propositions, in some semantic context. The things we implement are trivial, unless you're thinking about symbolic proofs etc

I think that’s precisely the problem the parent commenter is describing.

Reply on news.ycombinator.com