> The model can unquestionably code. [https://news.ycombinator.com/item?id=38205052]
Seems pretty clear most of the fantastic results from before was overfitting. Like every other case. It's amazing to me how these models can be caught red handed being overfitted again and again and again, and people don't get the memo.
The data is out there to objectively assess these tools, but it’s not good for business.
Generative AI is only as good as the dataset that you give it, so for problems that exist in a heavily contrived and parameterized space (like leetcode-style problems) it works really well. But give it a novel problem with intertwining libraries and external dependencies along with custom type and structure definitions, it's going to fall flat.
Humans do something AI can't, and that's draw from experience to apply a solution on a novel problem. This is why I'm not terribly worried about AI coming for engineer jobs anywhere in the near future. I use ChatGPT all the time to write me small functions, generate regular expressions, etc. Basically all the drudgery.
I'd argue we've hit peak-AI at this point because from this point on, all datasets are going to be colored by AI-generated results. Generative-AI is now on a trajectory were it'll simply regress to the mean of knowledge.
I don't know about the rest of your comment, albeit I consider myself more in the non-Chomsky camp. This emergence thing seems a little more than an elaborate hoax to me.
Well, maybe they've just trained GPT-4 to wiggle my balls, when I ask it to analyse a poem that I wrote 15 years ago.
The local llm scene, regardless of this debate, is nevertheless the hottest topic IMHO atm.
Several finetunes have repeatedly blown my mind even in the past 2 weeks. On a Raspberry Pi.
AI can interface. Natural Language Processing. This is the first time in history humaity has such knowledge and technology.
It's basically C-3PO.
Maybe I shouldn't, though I justify it because it allows me to focus on the bespoke bits of whatever I'm doing...
Feels a bit double edged to me still.
Even when I do need some, I use swagger or copier templates or something similar.
If you’re using ChatGPT for scaffolding, I feel you’ve fucked up?
And the above applies to a lot of prod code as well.
Now keep in mind that it had never seen this problem, and it was written in a very terse "competition solving" style with zero comments, one-letter identifiers, and generic function names like "day1" and "parse".
It figured it all out. It worked out that it was for solving a maze -- even though the word "maze" was not used anywhere in the code!
It worked out that a constant "(-1,0)" in one isolated bit of code was "orienting the character west", even though to figure that out it had to trace the logic through 4 or 5 layers of indirection. It connected a single 'w' character in the parser to a vector somewhere else!
Etc...
PS: It wrote a more useful, more accurate, more coherent, and more grammatically correct comment than I had seen in any codebase I had worked on in something like two years.
Allegedly.