Hacker News Reader

Comment by kqr | Hacker News Reader

> it is sad to me just how much people are trying to automate away programming and delegating it to a black box

I take it you're not using a compiler to generate machine code, then?

Scratch that, I guess you're not using a modern microprocessor to generate microcode from a higher-level instruction set either?

Wait, real programm^Wartists use a magnetised needle and a steady hand.

Programming has always been about finding the next black box that is both powerful and flexible. You might be happy with the level of abstraction you have settled on, but it's just as arbitrary as any other level.

Even the Apollo spacecraft programmers at MIT had a black box: they offloaded the weaving of core rope memory to other people. Programming is not necessarily about manually doing the repetitive stuff. In some sense, I'd argue that's antithetical to programming -- even if it makes you feel artistic!

View on HN

The thing is, all these stacks are built by people and verified against specifications. When they failed to perform the way they should, we fixed them.

Plus, all the parts are deterministic in this stack. Their behavior is fixed for a given input, and all the parts are interpretable, readable, verifiable and observable.

LLMs are none of that. They are stochastic probability machines, which are nondeterministic. We can't guarantee their output's correctness, and we can't fix them to guarantee correct output. They are built on tons of (unethically sourced) data, which has no correctness and quality guarantees.

Some people will love LLMs, and/or see programming as a task/burden they have to complete. Some of us love programming for the sake of it, and earn money by doing it that way, too.

So putting LLMs to the same bucket with a deterministic, task specific programming tool is both wrong, and disservice to both.

I'm also strongly against LLMs, not because of the tech, but because of how they are trained and how their shortcomings are hid and they're put forward as the "oh the savior of the woeful masses, and the silver bullet of all thy problems", and it's neither of them.

LLMs are just glorified tech demos which shows what stochastic parrots can pose as accomplishing when you feed the whole world to them.

I would argue thats an LLM spec --> Generate probabilistic output with a degree of confidence on the output nearing p(1). IMHO End users are supposed to not take the output of these machines as is but rather iterate on top and finish their task in lesser time.

> We can't guarantee their output's correctness

We can (for programming, at least): run the output thru theorem prover, ensure that proof is constructive, the Curry-Howard correspondence guarantees that you can turn the output into a correct program. It doesn't guarantee that formal properties of the program correspond to the informal problem statement. But even people occasionally make such errors (a provably correct program doesn't do what we wanted it to do).

> and we can't fix them to guarantee correct output

Same thing with other systems capable of programming. That is people.

You just can't make a system that guarantees correct transformation from an informal problem statement into a formally correct implementation. "Informal" implies that there's wiggle room for interpretation.

No, it doesn't mean that current LLMs are ready to replace programmers, it also doesn't mean that ML models of 2030s will not be able to.

This doesn’t mean the LLM is deterministic. It means it can spoof the right answer give enough time.

Indeterminism is not always bad. A probabilistic Turing machine is more powerful than a Turing machine, for example (BPP complexity class is a superset of P).

Yeah, didn't mean to imply something was bad or good here, just that it's not a purely deterministic thing.

>> We can (for programming, at least): run the output thru theorem prover, ensure that proof is constructive, the Curry-Howard correspondence guarantees that you can turn the output into a correct program. It doesn't guarantee that formal properties of the program correspond to the informal problem statement. But even people occasionally make such errors (a provably correct program doesn't do what we wanted it to do).

That sounds very ambitious. Automated theorem provers are real sticklers for complete specifications in a formal language and can't parse natural language at all, but when you generate code with an LLM all you have in terms of a specification is a natural language prompt (that's your "informal problem statement"). In that case what exactly is the prover going to prove? Not the natural language prompt it can't parse!

The best you can do if you start with a natural language specification, like an LLM prompt, is to verify that the generated program compiles, i.e. that it is correct syntactically. As to semantic correctness, there, you're on your own.

Edit: I'm not really sure whether you're talking about syntactic or semantic correctness after all. Which one do you mean?

>> You just can't make a system that guarantees correct transformation from an informal problem statement into a formally correct implementation. "Informal" implies that there's wiggle room for interpretation.

Note that in program synthesis we usually make a distinction between complete and incomplete specifications ("problem statements") not formal and informal. An incomplete specification may still be given in a formal language. And, for the record, yes, you can make a system that guarantees that an output program is formally consistent with an incomplete specification. There exist systems like that already. You can find a bit about this online if you search for "inductive program synthesis" but the subject is spread over a wide literature spanning many fields so it's not easy to get a clear idea about it. But, in general, it works and there are approaches that give you strong theoretical guarantees of semantic correctness.

> Which one do you mean?

Ah, I said "theorem prover", I should have said "proof verifier". What I meant is something like DeepMind's AlphaProof with an additional step of generating a formal specification from a natural language description of the problem. In this way we get a semantically correct program wrt the formal specification. But with current generation of LLMs we probably won't get anything for non-trivial problems (the LLM won't be able to generate a valid proof).

> Note that in program synthesis we usually make a distinction between complete and incomplete specifications

Program synthesis begins after you can coherently express an idea of what you want to do. And getting to this point might involve a ton of reasoning that will not go into a program synthesis pipeline. That's what I mean when I say an "informal problem statement": some brain dumps of half-baked ideas that doesn't even constitute an incomplete specification because they are self-contradictory (but you haven't noticed it yet).

LLMs can help here by trying to generate some specification based on a brain dump.

>> What I meant is something like DeepMind's AlphaProof with an additional step of generating a formal specification from a natural language description of the problem.

That's even more ambitious. From DeepMind's post on AlphaProof:

First, the problems were manually translated into formal mathematical language for our systems to understand.

https://deepmind.google/discover/blog/ai-solves-imo-problems...

DeepMind had to resort to this manual translation because LLMs are not reliable enough, and natural language is not precise enough, to declare a complete specification of a formal statement, like a program or a mathematical problem (as in AlphaProof) at least not easily.

I think you point that out in the rest of your comment but you say "the LLM won't be able to generate a valid proof" where I think you meant to say "a valid specification". Did I misunderstand?

>> Program synthesis begins after you can coherently express an idea of what you want to do.

That's not exactly right. There are two kinds of program synthesis. Deductive program synthesis is when you have a complete specification in a formal language and you basically translate it to another language, just like with a compiler. That's when you "coherently express an idea of what to do". Inductive program synthesis is when you have an incomplete specification, consisting of examples of program behaviour, usually in the form of example pairs of the inputs and outputs of the target program, but sometimes program traces (like debug logs), abstract syntax trees, program schemas (a kind of rich program template) etc.

Input-output examples are the simplest case. Today, if you can express your problem in terms of input-output examples there are approaches that can synthesize a program that is consistent with the examples. You don't even need to know how to write that program yourself.

> where I think you meant to say "a valid specification". Did I misunderstand?

What do you mean when you say "a valid specification"? There are known algorithms to check validity of a proof. How do you check that specification is valid? People are inspecting it and agree that "yes, it seems to be expressing what was intended to be expressed in the natural language", or, "no, this turn of phrase needs to be understood in a different way" or some such. Today there's no other system that can handle this kind of a task besides humans (who are fallible) and LLMs (that are much more fallible).

That is deciding that specification is valid cannot be done without human involvement. I left that part out and focused on what we can mechanistically check (that is validity of a proof).

So, no, I didn't mean "a valid specification". And, yes, I don't think that today's LLMs would be good at producing specifications that would be deemed valid by a consensus of experts.

> Today, if you can express your problem in terms of input-output examples there are approaches that can synthesize a program that is consistent with the examples

In a limited domain with agreed-upon rules of generalization? Sure. In general? No way. The problem of generalizing from a limited number of examples with no additional restrictions is ill-defined.

And the problem "generalize as an expert would do" is in the domain of AI.

Curry–Howard correspondence = direct relationship between computer programs and mathematical proofs

> "oh the savior of the woeful masses, and the silver bullet of all thy problems"

Who said that? Everyone I've talked to warns about their shortcomings (including their creators) and even the platform where I use them has a warning plastered right under the input box saying "ChatGPT can make mistakes. Check important info."

Yep. LLM’s don’t guarantee any interesting mathematical properties. That’s up to you.

This why you should write good tests, review the code, and don’t approve anything you don’t understand.

It’s not a reason to reject pretty good autocomplete, though.

>> I take it you're not using a compiler to generate machine code, then?

The dismissive glibness of your comment makes me wonder if it's worth it trying to point out the obvious error in the analogy you're making. Compilers translate, LLMs generate. They are two completely different things.

When you write a program in a high-level language and pass it to a compiler, the compiler translates your program to machine code, yes. But when you prompt an LLM to generate code, what are you translating? You can pretend that you are "translating natural language to code" but LLMs are not translators, they're generators, and what you're really doing is providing a prefix for the generated string. You can generate strings form an LLM with an empty prefix; but try asking a compiler to compile an empty program.

>> Even the Apollo spacecraft programmers at MIT had a black box: they offloaded the weaving of core rope memory to other people.

You're referring to core rope memory:

https://en.wikipedia.org/wiki/Core_rope_memory

There is no "black box" here. Programmers created the program and handed it over to others to code it up. That's like hiring someone to type your code for you at a keyboard, following your instructions to do so. You have to stretch things very far to see this as anything like compilation.

Also, really, compilers are not black boxes. Just because most people treat them as a scary unknowable thing doesn't mean that's what they are. LLms are "black boxes" because no matter how much we peer at their weights, arrays of numerical values, there's nothing we can ... er ... glean from them. They're incomprehensible to humans. Not so the code of a compiler. Even raw binary is comprehensible, with some experience.

I recently had used an LLM to convert a lot of Python to Rust. It got it 99% right, and it took me a short while to fix the compile-time errors, and carefully check the tests weren't broken (as I trusted the code worked when the tests passed).

Is that "compiling" or "translating"? Lots of people use language to C "compilers".

Compilation is generally deterministic with strict semantics.

LLMs are great, but they are the opposite of a compiler (in a good way)

Translating Python to Rust is translation, prompting an LLM for new Python or Rust code is generation.

Both of them are black boxes though by the author's own definition.

Which definition do you mean?

Exactly

I get what you're trying to say, but I don't entirely agree. Raising levels of abstraction is generally a good thing. But up until now, those have mostly been deterministic. We can be mostly confident that the compiler will generate correct machine code based on correct source code. We can be mostly confident that the magnetised needle does the right thing.

I don't think this is true for LLMs. Their output is not deterministic (up for discussion). Their weights and the sources thereof are mostly unknown to us. We cannot really be confident that an LLM will produce correct output based on correct input.

I agree with you but I want to try to define the language better.

It's not that LLMs aren't deterministic, because neither are many compilers.

It's also not that LLMs produce incorrect output, because compilers do that to, sometimes.

But when a compiler produces the wrong output, it's because either (1) there's a logic error in my code, or (2) there's a logic error in the compiler†, and I can drill down and figure out what's going on (or enlist someone to help me) to fix the problem.

Let's say I tell an LLM to write a algorithm, and it produces broken code. Why didn't my prompt work? How do I fix it? Can anyone ever actually know? And what did I learn from the experience?

---

† Or I guess there could be a hardware bug. Whatever. I'm going to blame the compiler because it needs to produce bytes that work on my silicon regardless of whether the silicon makes sense.

Compilers are deterministic

This is in general only true for either trivial toy compilers or ones which have gone to lengths to have reproducible builds. GCC for instance uses a randomised branch prediction model in some circumstances.

Ok, but my understanding is that that they are mostly deterministic. And that there are initiatives like Reproducible Builds (https://reproducible-builds.org) that try to move even more in that direction.

But what does "mostly" mean? You can compile the same code twice and literally get two different binaries. The bits don't match.

Sure, those collections of bits tend to do exactly the same thing when executed, but that's is in some sense a subjective evaluation.

---

Szundi said in a sibling comment that I was "completely [missing] the point on purpose" by bringing up compiler determinism. I think that's fair, but it's also why I opened my post by saying "I agree [with the parent], but I want to try to define the language better." Most compilers in use today are literally not deterministic, but they are deterministic in a different sense, which is useful as a comparison point to LLMs. Well, which sense? What is the fundamental quality that makes a compiler more predictable?

I'd like to try to find the correct words, because I don't think we have them yet.

I'm not an compiler expert, not by far. But my understanding is that if you compile the same code on the same machine for the same target, you'll get the same bits. Only minor things like timestamps that are sometimes introduced might differ. In this sense, maybe they are not deterministic. But I think it's fair to classify them as "determinstic" compared to LLMs.

Arguing with compilers about LLM determinism is not really adequate as an analogy, completely misses the point on purpose

I’d say it’s not only determinism, but also the social contract that’s missing.

When I’m calling ‘getFirstChar’ from a library, me and the author have a good understanding of what the function does based on a shared context of common solutions in the domain we’re working in.

When you ask ChatGPT to write a function that does the same, your social contract is between you and untold billions of documents that you hope the algorithm weights correctly according to your prompt (we should probably avoid programming by hope).

You could probably get around this by training on your codebase as the corpus, but until we answer all the questions about what that entails it remains, well, questionable.

> we should probably avoid programming by hope

I use Cursor at work, which is basically VSCode + LLM for code generation. It's a guess and check, basically. Plenty of people look up StackOverflow answers to their problem, then verify that the answer does what they want. (Some people don't verify but those people are probably not good programmers I guess.) Well, sometimes I get the LLM to complete something, then verify the code is completed is what I would have written (and correct it if not). This saves time/typing for me in the long run even if I have to correct it at times. And I don't see anything wrong with this. I'm not programming by hope, I'm just saving time.

This increases the time you spend proofing other’s work (tedious) versus time you spend developing a solution in code (fun). Also, if the LLM output is correct 95% of the time, one tends to get more sloppy with the checking, as it will feel unnecessary most of the time.

> This increases the time you spend proofing other’s work (tedious) versus time you spend developing a solution in code (fun).

I find that I don't use it as much for generating code as I do for automating tedious operations. For example, moving a bunch of repeating-yourself into a function, then converting the repeating blocks into function calls. The LLM's really good at doing that quickly without requiring me to perform dozens of copy-paste operations, or a bunch of multi-cursor-fu.

Also, I don't use it to generate large blocks of code or complicated logic.

Just what I was thinking about lately, what if LLMs are not 95% precise, but 99,95%. After like 50-100 checks you find nothing, and you just dump the whole project to be implemented - and there come the bugs.

However ... your colleagues just do the same.

We'll see how this unfolds. As for now the industry seems to be a bit stuck at this level. Big models too expensive to train for marginal gains, smaller are getting better but doesn't help this. Until some one new idea comes in how LLMs should work, we won't see the 99.95% anyway.

one idea is obvious: multi-model approach. it partially done today for safety checks. the same can be done for correctness. one model produces result, different model only checks the correctness. optionally several results, second model checks correctness and selects the best. this is more expensive, but should give better final output. not sure, this may have been already done.

Yeah, I’m more worried about the middle ground that would make software quality (even) worse than it is today.

> We can be mostly confident that the compiler will generate correct machine code based on correct source code.

Recently got email about gcc 14.2, they fixed some bugs in it. Can we trust it now, these could be the last bugs. But before that it was probably a bad idea to trust. No, even compiler's output requires extensive testing. Usually it's done at once, just final result of coding and compilation.

> Their output is not deterministic

yes.

> Their weights and the sources thereof are mostly unknown to us

Some of them are known. Does it make you feel better. There are too many weights, so you are not able to track its 'thinking' anyway. There are some tools which sort of show something. Still doesn't help much.

> We cannot really be confident that an LLM will produce correct output based on correct input

No, we can't. But it's so useful when it works. I'm using it regularly for small utilities and fun pictures. Even though it can give outright wrong answers for relatively simple math questions. With explanations and full confidence.

For the average programmer the infinite layers of abstraction, libraries and middleware isn't deterministic either. The LLMs actually honest to god being probabilistic estimators doesn't change anything about what they produce or how they see their own stuff.

> We cannot really be confident that an LLM will produce correct output based on correct input.

There are 2 things at play here, one is LLM with human in the loop, in which it's just a tool for programmers to do the same thing they have been doing, and the other is LLM as black box automaton. For the former, it's not a problem that the tool is undeterministic, we are double checking the results and add our manual labour anyway. The fact that a tool can fail sometimes is an unsurprising fact of engineering.

I think the criticism in this chain of comment applies more to the latter, but even it always has values to non-tech people, just like how no-code approaches are, however shitty it looks to us software enfineers.

I don't know. Programming with an LLM turns every line of code into legacy code that you have to maintain and debug and don't fully grok because you didn't write it yourself.

If it's in your PR then you wrote it, no one should be approving code they do not understand whether that's from AI or googling. Nothing changes there.

What if the we can't do abstraction anymore. I mean we certainly can but we will loose ability to configure tiny details of system.

So other path forward could very well be LLMs as they can save lot of time with writing boilerplate code

So what if the output is stochastic? LLM's have self consistency, so you can repeat the inference several times and pick the most frequent output.

Most frequent output does not imply correctness, LLMs often are confidently wrong.

They can't even perform basic arithmetic (which is not surprising since they operate at the syntactic level, oblivious to any semantic rules), yet people seem to think offloading more complex tasks with strict correctness requirements is a good idea. Boggles the mind tbh.

And you can pay in time and/or $$ for the privilege of having to do do this extra unnecessary work.

Saying a compiler, and an "AI" hallucination model are even remotely the same is a pretty huge stretch.

They aren't saying they're the same, I'm not sure how you got that interpretation. It's very clear they're highlighting the hypocrisy that arises from claiming to be against automating away aspects of programming while relying on tools that do exactly that for you - only being Ok with it as long as they aren't called "AI".

The crux of why this is a bad analogy is that everyone talking about "automating" things with LLMs is misusing the word "automation". A machine can automate a repetitive manual task. A computer can automate the operation of machinery. A machine instruction set is an abstraction on top of circuitry that can automate the labor of extrapolating the logic physically executed by that circuitry into human-comprehensible routines. In the same way, a programming language implementation (e.g. a compiler) can somewhat automate programming (in the sense that it uses higher levels of abstraction to describe the same thing, saving labor while keeping determinism). What do these things have in common? We can reliably make them approach deterministic behavior. In the case of compilers, completely and reliably and transparently so. Just because you haven't bothered to read what a compiler is doing doesn't mean someone can't verify what it's doing. Physical machines are less reliable, but we have reliable ways to test them, reliable error margins, reliable failure modes, reliable variance. When you are on a stack of abstractions like a programming language on top of a compiler on top of transistors on top of a machine, an error at the top of that stack can have a lot of implications. A tool that probabilistically generates code is not automation. We have no guarantees about how and when it will get things wrong, how much this will happen, and what kinds of things it will get wrong. We have no way to audit their results that will generalize to every problem upstream of them. We have no way to reliably measure improvement in consistency, let alone improve that margin of error reliably. The entire idea that this is an automation at all is nonsense.

They’re extremely different. But you can certainly use them together. Thus, new tools:

https://ai.meta.com/research/publications/meta-large-languag...

how do you figure? people were making directly analogous arguments about compilers back in the day. (not trying to argue that they are 'the same', but their is definitely a spectrum of code generation methods, with widely varying genres of guarantees, suiting a widely varying range of use cases)

Well, AI models don't have to hallucinate.

All current LLMs do though.

True, but LLMs do.

I get the point that they are in different magnitudes of unknown but the analogy is still pretty good when it comes to the median programmer, who has no idea what goes on within either one. And if you argue that compilers are ultimately deterministic, that same argument technically holds for an LLM as well.

The biggest difference to me is that we have humans that claim they can explain why compilers work the way they do. But I might as well trust someone who says the same about LLMs, because honestly I have no way to verify if they speak the truth. So I am already offloading a lot of burden of proof about the systems I work on to others. And why does this ”other” need to be a human.

This is like saying “I don’t understand how airplanes fly, so I’ll happily board an airplane designed by an LLM. The reality is determined by how much I know about it.”

No, the other way around. I am saying it is not a smart take to say ”a safe airplane cannot be built if LLMs were used in the process in any way, because reasons”. The safety of the airplane (or more generally the outcome of any venture) can be measured in other ways than leaning on some rule that you cannot use an LLM for help at any stage because they are not always correct

Thank you for saying this. It's always baffled me that people will decry ChangeX as unnatural and wrong when it happens in their lifetime, but happily build their lives upon NearlyIdenticalChangeY so long as it came before them.

“I've come up with a set of rules that describe our reactions to technologies:

1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.

2. Anything that's invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.

3. Anything invented after you're thirty-five is against the natural order of things.”

― Douglas Adams, The Salmon of Doubt: Hitchhiking the Galaxy One Last Time

This is so accurate I'm offended.

We think of ourselves more intelligent than the generation before us, and more wise than the ones after.

Makes you wonder if that sort of dynamic between generations is ever going to be something that can be overcome. Maybe if humanity cures aging.

I don't think that this is a fair comparison because at some point the nature of the craft actually does change.

To give an analogy, a carpenter might be happy with hand tools, happy with machine tools, happy with plywood, and happy with MDF. For routine jobs they may be happy to buy pre-fabbed cabinets.

But for them to employ an apprentice (AI in this example) and outsource work to them - suddenly they are no longer really acting as a carpenter, but a kind of project manager.

edit: I agree that LLMs in their current state don't really fundamentally change the game - the point I am trying to make is that it's completely understandable that everyone has their own "stop" point. Otherwise, we'd all live in IKEA mansions.

Running state of the art LLMs for programming is nowhere near project management. At least in my experience, all LLMs are really good at is dumping plausible tokens quickly. They can't think, design, or handle tradeoffs intelligently.

They help me with the keyboard work, not any of the actual programming.

An apprentice is another person performing the same kind of work as the carpenter. That's fundamentally different from using an LLM, which is not a person and does not function like a person.

Whether you think LLMs are spectacularly worthwhile or odious and destructive, it's crucial not to classify them as being a person instead of a software tool.

yep, I would call this the anthropomorphisation of llms. undesirable, just as any other kind of anthropomorphisation is.

This is a very poor analogy. It's not a matter of abstractions, it's a matter of getting someone or something else to do the work, while you mostly watch and fix any errors you're able to catch.

This is a qualitatively different kind of abstraction. All other abstractions still require the programmer to express the solution in a formal language, while LLMs are allowing the user to express the solution in natural language. It's no longer programming, but much more like talking to a programmer as a manager.

You can learn how compilers work and understand how they do what they do. Nobody understands what’s in those billions of parameters, and no one ever will.

why are there so many parameters in the first place? and was it humans who generated so many? seems like a very big job for a human to do, or even a team of humans to do.

disclaimer: I know next to nothing about llms. and I'm not that interested to learn about them. just asking casually.

> why are there so many parameters in the first place?

Because parsing and writing human language in a natural way is extremely complex.

> and was it humans who generated so many?

No, it is generated using an algorithm that tries to predict the next word in human written text using the words that comes before it. It ingests basically all the text on the internet to do this, without that much text the LLM performs horribly.

They're not manually generated or anything, it's just a setting. Too few and the model doesn't have enough flexibility to capture complex patterns. Too many and the model can just memorize the data you train it on rather than capturing the patterns driving it.

There is a difference between those abstraction levels: with LLMs you are moving into the realm of management.

That was precisely the goal of many high level languages.

…which all failed for that reason.

The question is about the abstraction being understandable and predictable. All the examples you have follow that, LLMs throw that out of the window.

>> Scratch that, I guess you're not using a modern microprocessor to generate microcode from a higher-level instruction set either?

Hell, I design gate level logic -> map it to instructions -> use them in C for the very LLMs and can fully understand[0] every aspect of it (if it doesn't behave as expected, that is a bug) but I cannot fathom or predict how the LLMs behave when i use them even though I know their architecture and implementation.

[0] Admittedly I treat the tools I use during the process, like cad tools, compiler, as black boxes, however I know that if I want to or the need arises, I can debug/understand them.

there's a strict contract between the programmer and the compiler (the language spec)

Which would be relevant if either side respected it.

In practice, compilers frequently have bugs and programmers even more frequently make use of "what the compiler actually does" rather than adhering to the language specification -- to the point where the de facto spec for many languages is "what the canonical implementation does".

And the specifications change over time.

The frequency with which compiler implementations functionally diverge from language specifications is dwarfed, by many orders of magnitude, by the frequency with which LLMs generate provably nonsensical code in response to a prompt.

To wit, a compiler diverging from the specification is so relatively rare that people will get angry about it and demand that it be fixed, while an LLM spewing creative nonsense is so accepted and par for the course that complaining about that fact is met with a shrug and "well, what did you expect?"

LLMs don’t have a canonical output that could serve as a specification. And if they had, we wouldn’t consider that a satisfactory specification at all.

I guess we are currently in the special situation, that we as human programmers can understand the output of a coding LLM. That's because programming languages are designed to be human readable. And we had an incentive to learn those languages.

I imagine that machine learning powered coding will evolve to an even blacker box, than it is today: It will transform requirements to CPU instructions (or GPU instructions, netlists, ...). Why bother to follow those indirections, that are just convenience layers for those weak carbon units (urgh)?

Simultaneously, automation will likely lead to fewer skilled programmers in the future, because there will be fewer incentives to become one.

Together those effects could lead to a situation where we are condemned to just watch.

So what do you do when the LLM creates a bug in a multi-million CPU instruction program it generated and you can't prompt it to fix the bug?

Yeah, good question. You could ask the same question for today's or near future coding LLMs used by non programmers.

>> delegating it to a black box

> I take it you're not using a compiler to generate machine code, then?

An LLM is much, much closer to a "black box" than a compiler ...

In fact, an LLM is pretty much a black box even to the people who created it.

Reminds me long ago of working with a guy that did everything in C when we were rewriting things in Perl . Yes, his stuff was faster. Yes, it was also buggier, harder to debug, and took 3x as long to write for similar levels of functionality (it wasn't speed dependent code by any stretch.)

> black box

Actually the opacity of the abstraction layer is the core of the issue. First we note that opacity is a measure in both the inner-workings of the 'box' and orthogonally (in context of LLMs) a measure of deterministic outcome.

Programming, it is asserted by some of us, is exactly the act of instructing a deterministic 'black' box.

Peer-coding with an LLM is the act of cajoling a mechanism to hopefully consistently produce the input to a sensible "blackbox". It is not programming, it is getting help. Now if the help was 100% reliable, we could discuss programming the helper.

The other day I had a vision of the future AI whisperer in the corporate setting. They wear capes of varied colors and possibly sport a wand. "It's an art you see".

By that light, the artist that is painting using store-bought pigments instead of hand made, or brushing paints instead of reallocating molecules, or even better, atoms, is also using a black box.

I think OP has a point, and is about guiding the design, then overall structure and dynamic of the code. Nobody expects to write in Assembler or not using libraries, but making a concious decision on the design.

I have met few programmers, but many coders. For them coding is a job, and generally they don’t care about overall architecture, algorithm efficiency, and code elegance is limited to syntax-coloring-themes in their editor. I respect their existence, but generally they are building on top off somobody else’s effort.

The transformations you're referring to are fully deterministic and guaranteed to be correct.

LLMs provide statistically probable answers with no guarantee of correctness. It takes more time to review LLM code than it takes to write it correctly from scratch.

We can argue the same thing for artists. Who wrote the algorithms for your favorite Photo/Image editor? Who created the image formats and standards, infrastructures for you to be able to push binary files to millions of people?

I think it would be a mistake to classify an LLM as a level of abstraction. You cannot use it as a stable black-box foundation to build on top of.

Your argument is a false dichotomy and thus a logical fallacy. Not all steps forward in abstraction or code generation are necessary good steps and have to be considered on their own merits. If LLMs are indeed superior, then you should be able to articulate their merits without condescending to fallacious attacks.

HN commenter: Samuel, why don't you use an LLM to write this play you are working on?

Beckett: What?

HN commenter: Well it's just like when you decided to work in French instead of English. Your art was no less because of it. Now you can use an LLM instead of French. It will be so much quicker.

While the base of your argument is true, it’s also a bit dishonest. LLMs are significantly different than any of these other abstractions because they can’t be reasoned about or meaningfully examined/debugged. They’re also the first of these advances which anyone has claimed would eliminate the need for programmers at all. I don’t believe the C compiler was meant to do my whole job for me.

Cobol and other early high-level languages were designed with the intention of allowing businesspeople to write their own programs so programmers wouldn't be needed. Some people really believed that!

SQL too.

I'd really like to have everything written in Rust, not C. Rust does a lot of verification, verification that is very hard to understand. I'd like to be able to specify a function with a bunch of invariants about the inputs and outputs and have a computer come up with some memory-safe code that satisfies all those invariants and is very optimized, and also have a list of alternative algorithms (maybe you discard this invariant and you can make it O(nLog(n)) instead of O(n^2), maybe you can make it linear in memory and constant in time or vice versa...)

Maybe you can't examine what the LLM is doing, but as things get more advanced we can generate code to do things, and also have it generate executable formal proofs that the code works as advertised.

I agree with the second part of your argument, regarding the assertion that LLMs may eventually replace programmers.

However, I don't understand your claim that an LLM acting as a programming assistant "...can’t be reasoned about or meaningfully examined/debugged."

I type something, and Copilot or whatever generates code which I can then examine directly, and choose to accept or reject. That seems much easier to reason about than what's happening inside a compiler, for example.

Current LLMs, we do not know that we cannot have future LLMs which can be almost formal. Think mathematics written in English and LaTeX

If using an LLM meant carefully crafting a complex, precise, formal prompt that specified only one possible output, I might be interested. But then I wonder if the prompt would be very much shorter.

Thinking about it, this depends on which differences we consider aspects of the output program, and which ones we consider trivial differences that don't count. If you say "build an RPG about dragons with a party of magic using heroes" and the LLM spits one out, you reached a level of abstraction where many choices relating to taste and feeling and atmosphere (and gameplay too) are waved aside as trivial details. You might extend the prompt to add a few more, but the whole point of creating a program this way is not to care about most of the details of the resulting experience. Those can be allowed to be generic and bland, right? Unless you care about leaving your personal touch on, say, all of them.

Things constructivist mathematics cannot do:

1) Prove Addition of natural numbers.

2) Prove two real numbers are equal.

RNNs are only TC with infinite precision and unlimited resources, once you have finite precision they are very limited.

LLMs do not have recursion at all and can't even emulate finite automations. In fact soft attention can only emulate TC_0

Feed forward networks are effectively DAGs and with soft attention, DAGs built with AND,OR, NOT, and threshold circuits.

One of the state of the art inference in code methods is bi-abduction, probably best described here.

https://fbinfer.com/docs/separation-logic-and-bi-abduction/

But this localization makes it computationally possible, and has limits.

The qualification and frame problems, combined with the very limited computational power of transformers is another lens.

LLMs being formalized doesn't solve the problem. Fine tuning and RAG can help with domain specificity, but hallucinations are a fundamental feature of LLMs, not a bug.

Either a use case accepts the LLM failure mode (competent, confident, and inevitably wrong) or another model must be found.

Gödel showed us the limits of formalization, unless we find he was wrong, that won't change.

Thanks for your insightful comment. I'll read the links later.

I had just assumed that RNNs were TC, didn't think of limitation put on by bounded precision since I assumed that any bounded precision could be compensated by growing memory module.

So, after your comment, I went literature searching and I found this: https://papers.nips.cc/paper_files/paper/2021/hash/ef452c63f...

I haven't read it yet. But if it is true, then RNN would seem to be TC

Note from the open review of that paper.

> As discussed in the paper and pointed out by the reviewer, the growing memory module is non-differentiable, and so it cannot be trained directly by SGD. We acknowledge this observation.

Two stack FSA/RNN are interesting, but as of now, not usable in practice.

I don't buy that you're actually examining compiled programs. Very few people do. Theoretically you could, but the whole point of the compiler is to find optimizations that you wouldn't think of yourself.

The point of an optimizing compiler is to find optimizations which, crucially, are semantics-preserving. This is the contract that we have with compilers, is the reason that we trust them to transform our code, and is the reason why people get up in arms every time some C compiler starts leveraging undefined behavior in new and exciting ways.

We have no such contract with LLMs. The comparison to compilers is highly mistaken, and feels like how the cryptocurrency folks used to compare cryptocurrency to gestures vaguely "the internet" in an attempt to appropriate legitimacy.

No, the point of the compiler is to translate code into machine instruction.

Yes, it can optimize things for you, but that is not its purpose.

Okay I'll revise my statement

A big feature of compilers is to find optimizations you wouldn't think of. I tried to make the point that compiled output is typically not read by humans

> I don't buy that you're actually examining compiled programs. Very few people do

I take it you don't write C, C++, or any language at that level? It is very common to examine compiled programs to ensure the compiler made critical optimizations. I have done that many times, there are plenty of tools to help you do that.

I think you’re assuming your reference is the correct one. I can’t reason about the assembly language that the compiler spits out, the microcode in the CPU kernel or any of the electronics on the motherboard. That anyone can or not doesn’t change things in my opinion. It’s an arbitrary distinction to say _this_ abstraction is uniquely different in this very specific way.

Compilers are deterministic.

LLMs are not.

>Compilers are deterministic

They seem that way, until you're tasked with getting a repeatable, idempotent build out of a non-trivial build system.

LLMs are deterministic if you force a seed or disable sampling. They however do not guarantee that small input changes will cause small output changes.

> LLMs are deterministic if you force a seed or disable sampling

Not with todays GPU's, you would need to run it locally with special GPU settings or run it on your CPU to ensure it is deterministic.

So? Compilers compile and LLMs do not. Compilers use linkers and LLMs do not. Arbitrary distinctions don’t means “this time it’s different”.

So as the OP said, all the parts are deterministic in this stack. Their behavior is fixed for a given input, and all the parts are interpretable, readable, verifiable and observable.

This is entirely different from LLMs which are opaque even to their designers, and have unpredictable flaws and hallucinations, they are probability machines based on what data they have been exposed to, which means they are not a reliable way to generate programs.

Maybe one day we'll fix this, but the current generation is not very useful for programming because of this.

This is completely wrong.

Compilers are complex programs fraught with bugs. Modern microprocessors are hideously complex devices fraught with bugs. But at least we understand them in principle and practice.

LLMs are nonsense generators, you need a second device that can recognize correct programs to use them effectively. Only humans can do that all-important second part.

> Programming has always been about finding the next black box that is both powerful and flexible.

That's the opposite of programming. Programming is the art and science of developing reliable algorithms. You can treat programs as black boxes only after you're sure that they work correctly. Otherwise you're just engaged in a kind of cargo cult.

That the scary thing about the LLM fad: so many people seem so willing to abdicate their responsibility to actually think.

Um those black boxes have determinism.

Not at all. See the cries for reproducible builds, exploits like Spectre etc.

You are precisely right but practically wrong here

I 100% agree with you. Whenever I see artists buy their brushes I cringe. Real artists don't draw anything until they've grown a tree and raised horses to obtain the raw materials (wood and horse hair) to make their first brush.

Using a bought brush to paint and generating a painting via a prompt are basically identical.

Nailed it.

Reply on news.ycombinator.com