Does programming have a clear reward function? A vague description from a business person is not it. By the time someone (a programmer?) has written a reward function that is clear enough, how would that function look compared to a program?
Also look out for optimization the clever way.
Sounds like this work would involve asking questions to collaborators, guess some missing answers, write specs and repeat. Not that far ahead of the current sota of AI...
While writing simple functions may be mechanistic, being a developer is not.
'guess some missing answers' is why Waterfall, or any big upfront design has failed.
People aren't simply loading pig iron into rail cars like Taylor assumed.
The assumption of perfect central design with perfect knowledge and perfect execution simply doesn't work for systems which are for more like an organism than a machine.
Next gen LLMs, with their encyclopedic knowledge about the world, won't have that problem. They'll get the design correct on their first attempt because they're already familiar with the common pitfalls.
Of course we shouldn't expect LLMs to be a magic bullet that can program anything. But if your frame of reference is "visual programming" where the goal is to turn poorly thought out requirements into a reasonably sensible state machine then we should expect LLMs to get very good at that compared to regular people.
I keep looking for discoveries that show any movement there. But LLMs are still basically pattern matching and finding.
They can do impressive things, but they actually have no concept of what the 'right thing' even is, it is statistic not philosophy.
Why do you think they would need a more defined spec than a human?
There's nothing about AI that makes such operations intrinsically impossible, but they require much more than just the ability to generate working code.
Anything you don't define, is literally undefined behavior the same as in a compiler. The human will do something, and maybe you like it and maybe you don't.
A perfect spec is just another way to dedcribe a formal language, ie any programming language.
If you don't care what you get, then say little and say it ambiguously and pull the slot machine lever.
If you care what you get then you don't necessarily have to say a lot but you have to remove ambiguity, and then what you have is a spec, and if it's rigorous enough, it's a program, regardless what language and syntax is used to express it.
There is a difference between a human and an ai, and it is more than a difference of degrree, but filling in gaps with something that fits is not very significant. That can be done perfectly mechanistically.
In general, implementations can be vastly more complicated than even a complicated spec (e.g. by having to deal with real-world network failures, etc.), whereas a spec needs only to describe the expected behavior.
In this context, this is actually super useful, since defining the problem (writing a spec) is usually easier than solving the problem (writing an implementation); it's not just translating (compiling), and the engineer is now thinking at a higher level of abstraction (what do I want it to do vs. how do I do it).
However I agree that's the hard part. I can write a spec for finding the optimal solution to some combinatorial problem - where the naive code is trivial - a simple recursive function for example - but such a function would use near infinite time and memory.
In terms of the ML programme really being a compiler - isn't that in the end true - the ML model is a computer programme taking a spec as input and generating code as output. Sounds like a compiler to me.
I think the point of the AK post is to say the challenge is in the judging of solutions - not the bit in the middle.
So to take the writing software problem - if we had already sorted the computer programme validation problem there wouldn't be any bugs right now - irrespective of how the code was generated.
You haven't disproved this idea, merely re-stated the default obvious intuition that everyone is expected to have before being presented with this idea.
Their point is correct that defining a spec rigorously enough IS the actual engineering work.
A c or go program is nothing else but a spec which the compiler impliments.
There are infinite ways to impliment a given c expression in assembly, and doing that is engineering and requires a human to do it, but only once. The compiler doesn't invent how to do it every time the way a human would, the compiler author picked a way and now the compiler does that every time.
And it gets more complex where there isn't just one way to do things but several and the compiler actually chooses from many methods best fit in different contexts, but all of that logic is also written by some engineer one time.
But now that IS what happens, the compiler does it.
A software engineer no longer writes in assembly, they write in c or go or whatever.
I say I want a function that accepts a couple arguments and returns a result of a math formula, and it just happens. I have no idea how the machine actually impliments it, I just wrote a line of algebra in a particular formal style. It could have come right out of a pure math textbook and the valid c function definition syntax could just as well be pseudocode to describe a pure math idea.
If you tell an ai, or a human programmer for that matter, what you want in a rigorous enough format that all questions are answered, such that it doesn't matter what language the programmer uses or how the programmer impliments it, then you my friend have written the program, and are the programmer. The ai, or the human who translated that into some other language were indeed just the compiler.
It doesn't matter that there are multiple ways to impliment the idea.
It's true that one programmer writes a very inefficient loop that walks an entire array once for every element in the array, while another comes up with some more sophisticated index or vector or math trick approach, but that's not the definition of anything.
There are both simple and sophisticated compilers. You can already right now feed the the same c code into different compilers and get results that all work, but one is 100x faster than another, one uses 100x less ram than another, etc.
If you give a high level imprecise directive to an ai, you are not programming. If you give a high level precise directive to an ai, you are programming.
The language doesn't matter. What matters is what you express.
The same is true for math. There is a clear reward function when the goal is well-specified, e.g., "we need a sequence of mathematical statements that prove this other important mathematical statement is true."
>when the problem being solving is well-specified
I’d suggest that the Python code for your example prompt with reasonable defaults is not actually that far from the prompt itself in terms of the time necessary to write it.
However, add tricky details like how you want to handle connection pooling, differing retry strategies, short circuiting based on one of the results, business logic in the data combination step, and suddenly you’ve got a whole design doc in your prompt and you need a senior engineer with good written comms skills to get it to work.
I feel we're looking at the need for a measure of the computational complexity of problem specifications -- something like Kolmogorov complexity, i.e., minimum number of bits required, but for specifying instead of solving problems.
Ah, got it. I was just trying to keep my comment short!
Remember all those attempts to transform UML into code back in the day? This sounds sorta like that. I’m not a total genai naysayer but definitely in the “cautiously curious” camp.
I think generative AI is already a "really good autocomplete" and will get better in that respect, I can even see it generating good starting points, but I don't think in its current form it will replace the act of programming.
There’s the task of writing syntax, which is the mechanical overhead of the task of telling the computer what to do. People should focus on the latter (too much code is a symptom of insufficient automation or abstraction). Thankfully lots of people have CS degrees, not “syntax studies” degrees, right?
I feel like this is a very different type of programming, even if in some cases it would wind up being the same thing.
the reason why we spend time programming is because the problems in question are not easily defined, let alone the solutions.
Here I have assumed something about the set that f lives in. I am taking for granted that a probability measure can be defined. In theory, perhaps there are difficulties involving the various weird infinities that show up in computing, related to undecideability and incompleteness and such. But at a practical level, if we assume some concrete representation of the program then we can just define that it is smaller than some given bound, and ditto for a number of computational steps with a particular model of machine (even if fairly abstract, like some lambda calculus thing), so realistically we might be able to not worry about it.
Also, since our input and output sets are bounded (say, so many 64-bit doubles in, so many out), that also gives you a finite set of functions in principle; just think of the size of the (impossibly large) lookup table you'd need to represent it.
If you’re the most junior level, sure.
Anything above that, things get fuzzy, requirements change, biz goals shift.
I don’t really see this current wave of AI giving us anything much better than incremental improvement over copilot.
A small example of what I mean:
These systems are statistically based, so there’s no probability. Because of that, I wouldn’t even gain anything from having it write my tests since tests are easily built wrong in subtle ways.
I’d need to verify the test by reviewing it and, imo, writing the test would be less time than coaxing a correct one, reviewing, re-coaxing, repeat.
1) Addition of the natural numbers 2) equality of two real numbers
When you restrict your tools to perceptron based feed forward networks with high parallelism and no real access to 'common knowledge', the solution set is very restricted.
Basically what Gödel proved that destroyed Russel's plans for the Mathmatica Principia applies here.
Programmers can decide what is sufficient if not perfect in models.
The interesting kind of programming is the kind where I'm figuring out what I'm building as part of the process.
Maybe AI will soon be superhuman in all the situations where we know exactly what we want (win the game), but not in the areas we don't. I find that kind of cool.
I've found when porting that the answers to these are sometimes not universal for a codebase, but rather you are best served considering case-by-case inside the code.
Although I suppose an AI agent could be created that holds a conversation with you and presents the options and acts accordingly.
Also the reasoning is something business is dissonant about. The majority of planning and execution teams stick to processes. I see way more potential automating these than all parts in app production.
Business is going to have a hard time, when they believe, they alone can orchestrate some AI consoles.
Every decade it's 'we don't need programmers anymore'. Then it turns out specifying the problem needs programmers. Then it turns out the auto-coder can only reach a certain level of complexity. Then you've got real programmers modifying over-complicared code. Then everyone realizes they've wasted millions and it would have been quicker and cheaper to get the programmers to write the code in the first place.
The same will almost certainly happen with AI generated code for the next decade or two, just at a slightly higher level of program complexity.
I literally refuted this in my comment…
That being said, some kind of “no-code” is not necessarily a bad idea, as long as you treat it as just an abstraction for people who actually are programmers, like C versus assembly, or high level languages vs C.
In fact I worked for a train manufacturer that had a cool “no code” tool to program automated train control software with automated theorem proving built in, and it was much more efficient than there former Ada implementation especially when you factor the hiring difficulties in.
Certainly "compiled" is one reward (although a blank file fits that...) Another is test cases, input and output. This doesn't work on a software-wide scale but function-wide it can work.
In the future I think we'll see more of this test-driven development. Where developers formally define the requirements and expectations of a system and then an LLM (combined with other tools) generates the implementation. So instead of making the implementation, you just declaratively say what the implementation should do (and shouldn't).
It also seems less unique to code. You could also have a chat bot write an encyclopedia and see if the encyclopedias sold well. Chat bots could edit Wikipedia and see if their edits stuck as a reward function (seems ethically pretty questionable or at least in need of ethical analysis, but it is possible).
The maybe-easy to evaluate reward function is an interesting aspect of code (which isn’t to say it is the only interesting aspect, for sure!)
Well, to give an example: the complexity class NP is all about problems that have quick and simple verification, but finding solutions for many problems is still famously hard.
So there are at least some domains where this model would be a step forward.
And in any case, there are plenty of problems in NP that are not NP hard, too.
Yes, approximation is also an important aspect of many practical problems.
There's also lots of problems where you can easily specify one direction of processing, but it's hard to figure out how to undo that transformation. So you can get plenty of training data.
No, running it as a linear program is still slow.
I'm talking about small n=50 taking tens of minutes for a trivial linear program. Obviously the actual linear program is much bigger and scales quadratically in size, but still. N=50 is nothing.
But here again: solutions to your problem are easy to verify, so it might be interesting to let an AI have a go at solving it.
In other words, to what extent does Agile's ubiquity prove our competence in turning product goals into de facto reward functions?
There are objective criteria like 'compiles correctly' and 'passes self-designed tests' and 'is interpreted as correct by another LLM instance' which go a lot further than criteria that could be defined for most kinds of verbal questions.
If you have those many well written tests, you can pass them to a constraint solver today and get your program. No LLM needed.
Or even run your tests instead of the program.
One very strong LLM could generate billions of tests alongside the working code and then train another smaller model, or feed it into the next iteration of training same the strong model. Strong LLMs do exist for that purpose, Nemotron 320B and Llama 3 450B.
It would be interesting if a dataset like that would be created like that, and then released as open source. Many LLMs proprietary or not, could incorporate the dataset in their training, and have on the internet hundreds of LLMs suddenly become much better at coding, all of them at once.
What should the colleagues of the business person review before deciding that the system is fit for purpose? Or what should they review when the system fails? Should they go back over the transcript of the conversation with the LLM?
1) The business person made a mistake in their conversation/specification.
In this case the LLM will have generated code and tests that match the mistake. So all the tests will pass. The best way to catch this before it gets to production is to have someone else review the specification. But the problem is that the specification is a long trial-and-error conversation in which later parts may contradict earlier parts. Good luck reviewing that.
2) The LLM made a mistake.
The LLM may have made the mistake because of a hallucination which it cannot correct because in trying to correct it the same hallucination invalidates the correction. At this point someone has to debug the system. But we got rid of all the programmers.
That an LLM or a human is where the code comes from, doesn't make much difference.
Though it does kinda sound like you're assuming all LLMs must develop with Waterfall? That they can't e.g. use Agile? (Or am I reading too much into that?)
How do they do this? They can't trust the tests because the tests were also developed by the LLM which is working from incorrect information it received in a chat with the business person.
Mediocrely.
Sometimes the current process works, other times the planes fall out of the sky, or updates causes millions of computers to blue screen on startup at the same time.
LLMs in particular, and AI in general, doesn't need to beat humans at the same tasks.
Once the system can both test and update the spec etc to fix errors in the spec and build the program and ensure the result is satisfactory, we have AGI. If you argue an AGI could do it, then yeah it could as it can replace humans at everything, the argument was for an AI that isn't yet AGI.
Excel sheets are not fuzzy and underspecified.
> It's OK if a human employee is in the loop and has to intervenes sometimes
I've never worked on software where this was OK. In many cases it would have been disastrous. Most of the time a human employee could not fix the problem without understanding the software.
But we're not anywhere close to maximally automated. Today (many? most?) office workers do manual data entry and processing work that requires very little thinking. Even automating just 30% of their daily work is a huge win.