Hacker News Reader

Comment by mjburgess | Hacker News Reader

In my experience, in many cases, the so-called "first draft" (of code, content, etc.) is more just helpful for a mental warm-up. It isn't actually suitable.

The time it takes to write/code/etc. isnt in the typing, that's trivial. It's in modelling the problem (, audience) and solving it.

ChatGPT isnt in the game of modelling problems and solving them, it possesses only superficial semantic characteristics of the questions posed

That its answers satisfy the requirements would require ChatGPT to ask you questions (and much else besides)

View on HN

> ChatGPT isnt in the game of modelling problems and solving them, it possesses only superficial semantic characteristics of the questions posed

You can give it the datamodel and a semantic description and it will do a pretty good job of reprojecting your data.

People talk about chatgpt as being a fixed thing, it isn't. ChatGPT is the Apple II of LLMs. 3-4 moore units from now every issue that people think is a "gotcha" for chatgpt will have evaporated.

In your analogy to the Apple II, is there a fundamental problem people claimed for early computers, which didn’t just boil down to “they need to be faster”?

There are fundamental limitations to what an LLM can achieve.

You missed the ever-powerful ellipsis in what he said:

Today "..." and then it will be solved.

It's the most powerful operator in the whole of applied computer science.

True, I mean, just look at all of the examples of technologies that were projected to revolutionize computing and machine learning "..." eventually, here:

https://en.wikipedia.org/wiki/AI_winter

Just because they haven't happened to yet doesn't mean they never will! I've heard very good things about expert systems and IBM's Watson for Oncology.

The Harvard Business Review has a great article on all of the eventual "..." AGIs here:

https://hbr.org/2022/03/why-ai-failed-to-live-up-to-its-pote...

Yeah, ChatGPT/copilot kinda is that ellipsis.

Integrate it with wolfram alpha and Google, tack on a bullshit detector and spank dispenser conetwork and it looks like limitations of LLMs are easily overcome by not running anywhere near them.

Some of the problems with ChatGPT are not mere “gotcha”s but rather problems inherent to how LLMs work and their philosophy. The biggest issue I’ve seen some other scientists [1] who don’t have a stake in OpenAI bring up is that: “LLMs make sh*t up”.

Maybe LLMs are not the final answer. For now, it looks like they are not.

[1] Yann LeCun: https://twitter.com/ylecun/status/1617254676428111872?s=46&t...

By the time the semantic description is sufficiently exact, you may as well have written the code, though. [1]

[1] https://i.pinimg.com/originals/e3/2f/22/e32f2209b4c3d6b32bbc...

Still waiting for those autonomous cars.

They are already here. We can haggle over error rate and incidents per million miles or what have you but that is more an argument about safety and suitability.

Those must be some big gotchas, because self-driving cars have not replaced a job here, yet.

Large language models show more promise for replacing non-programming office workers.

'Replaced all forms of relevant labor, or threaten to' isnt really what I am addressing or care to address.

We have recognizably autonomous vehicles - that is the statement I am addressing and would ask you to, as well.

The similar thought I had yesterday was that ChatGPT is pretty good at generating sample code that doesn't otherwise exist. It can be really useful if you treat it like that.

Of course I've also had it call functions and use constants which don't exist, and fail to correct itself even after repeated prompting. So even that only goes so far.

Imagine if ChatGPT (or any other competitor) had access to a compiler and a set of tests for the program, a la HackerRank.

You then give it the prompt and a lot of test cases to pass in order to consider the task complete.

I can see this concept being implemented and working before the end of the year.

Google / Deepmind has already done this last year. They published the paper at the end of 2022 I think. (searches) yep: https://www.deepmind.com/blog/competitive-programming-with-a...

It relies on test availability of good test cases to verify the code functionality: they generate many, many possible programs, throw out all the ones that aren't syntactically correct, compile the ones that are & then test them. This works reasonably well for programming competitions & the code generated will probably improve over time.

The issue for "real-world" problems is that you can't verify the code without the test cases & writing good tests that ensure the code does what you want can be as much work as writing the code itself in the first place - the tests form a kind of mathematical co-domain to the code after all & only a small subset of problems are so simply defined that you can pin them down with just one or two tests.

You can get it to ask more questions…

This

Reply on news.ycombinator.com