Kickstarting AI for Code: Introducing IBM’s Project CodeNet
research.ibm.com
research.ibm.com
1) it’s new code that works well in 90-95% of the cases, but fails in unexpected ways and cannot be debugged, you have to generate a new piece of code that works in the above failure but has its own: works 90-95% times and fails in mysterious ways
OR
2) it looks like code, it reads like code but humans can’t figure out what it does
OR
3) it is a collection of blocks of code put in order automatically to do a certain job, that works in 90-95% of cases and…
OR
4) ??
- if this is something like automated theorem provers, it's something like (2). It's correct, provably correct even, but it's a bunch of nonsense that provides no insight to humans.
- if this is something like natural language query generators, it's something like (1), but with a much lower success rate (closer to 50% IME).
I don't even need to specify that AI-generated code is interesting, but we aren't even remotely close to being there.
First and most importantly, programming computers is a very precise endeavor. The logic needs to be exactly correct, not only statistically correct. 'Close enough' won't cut it, not even remotely, and not even for relatively unimportant software.
Second, the general problem is undecidable. This isn't a roadblock per se, because we are reasonably good at other undecidable problems (viz. garbage collection), but it means simple algorithmical approaches won't work.
Third, software is in a weird place because it requires working at different abstraction levels simultaneously. Often, top-level specifications are fuzzy and incomplete, but some parts require absolute precision and we need to "drop down" to a lower level of abstraction. Humans are able to make the process work (kinda) using lots of common sense, something machines are currently very bad at. If you require the operator of the 'AI' to fill in the details, you just invented a very complicated compiler.
Finally, rarely if ever present-day software is made once and never changed: the output needs to be inspectable and maintainable, other software might need to call into it, etc. If the pipeline is more complicated than the software itself, I might as well be writing the code myself.
I can see some minor, specific tasks being increasingly done with AI, and I can see tools making more and more use of AI technology, but AI generated code isn't even on the horizon.
Eventually you could also have another separate AI that learns to generate the test suite itself from instructions given to it by a human (or even another AI!).
Most program implementations would not be perfect, but as we know, software written by humans certainly isn’t perfect either.
IBM is an awful company to deal with. We were interested in getting an IBM data provider for .Net Core to connect to DB2 database. Figuring out the cost was insanely hard and nobody could answer which version we needed. I was then provided with a trial version, which did not work for version of .Net Core I was using. I had to dig deep and spend days figuring out why it wasn't working. I was put in touch with IBM engineer and figured out the issue before he did. After they provided the correct version, the setup was extremely annoying. Placing license files in a folder. Then, I had issues with errors and IBM deleted a bunch of forum threads and I was presented with many dead links.
Many questions on IBM forum were answered with "I sent you a PM", followed by people screaming "why can't you just openly share the solution".
Documentation was awful to non-existent.
They just want to nickle and dime everyone.
I decided IBM doesn't deserve any money, and I am using good ole OleDb to connect to DB2.
IBM is a great old brand, but all things pass, and it's way overdue for them. New things grow where old things die, and I'm ager to see what follows them.
I can see it now, IBM low code / no code platform for migrating COBOL on the IBM Mainframe to really bad javascript on the IBM Cloud.
Now, what could this be used for: 1. Since there are functional and non-functional codes for each problem that is precisely defined, data could be used for AI model to learn how to debug and modify the code to make it "correct" - the test data will be critical here for model's learning to be reinforced and driven with. 2. Improve the performance of the code - similar to above, since we know for each solution what the performance was, for a given problem, learn the ways to improve performance. 3. similar to 2 above, improve memory 4. code similarity, since we can compare the underlying graph as well, which are in the metadata for many samples, and AST generators are provided too.. 5. Code translation since solutions for the same problem are in polyglot of languages, code translation is another critical usecase.
understand the terms of the license under which your contributions will be governed and applied, your ability to know where this is being deployed, your rights to be paid for skilled work.. (?)
No seriously, AI will take years to understand the context.
Coding is more about capturing real world knowledge into a functional program. It involves so much learning about how to translate processes, workflow and regulations into a binary that it seems naive to have an AI code generation tool we even can't understand.
How would that effect the job market for human software engineers?