Hacker News Reader

Comment by FiberBundle | Hacker News Reader

I think it must have seen the solution somewhere already on the web. I find it extremely hard to believe that such a general purpose chatbot would just be able to solve programming problems. Deepmind had a paper [1] on solving programming problems a couple of months ago and they had to apply quite specialized heuristics in order to solve these problems. Obviously ChatGPT does nothing of the sorts and it just seems extremely unrealistic that it would be capable of outperforming previous work like that.

[1] https://news.ycombinator.com/item?id=30179549

View on HN

I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.”

I don’t believe the AI is generalized by any means. But I think it’s more than a Eliza-style parlour trick.

It does really well with facts but misses subjectivity and nuance. Like I ask it to write stories for Seinfeld or Raymond and it misses the tone and style entirely. But it gets all the names and other factual details right. I find it works incredibly well when you want things that are objective, like code or rhymes or comparisons of things.

What does any of this have to do with solving programming problems?

> I’ve had this skepticism until I started asking the bot to do things like “create a limerick using some words that have never existed before and then explain the definition of those words.”

Sounds interesting! Mind sharing the results?

I’m not saying they’re good. Just that they’re (mostly) following the rules and aren’t just thematically random.

https://ibb.co/RpSxcV8 https://ibb.co/883GpHY

That first one is pretty good.

Perhaps they aren't good, but they aren't that bad either. Probably better than I'd manage...

> things that are objective, like code or rhymes

rappers disagree

Oh my yes, for sure. I mean the literary rules of a rhyme or a limerick. Rules that artists can, do, and should break for effect.

Though it may look amazing it is not really thinking. It is simply capable of using a placeholder word. For HN’s sake AI does not understand anything, it’s just leveraging patterns, patterns that are sometimes complicated for us and that does make a great tool. But it’s just a tool for now.

We are all philosophical zombies. Let’s not single out the software. ;)

AlphaZero and Stockfish are not really thinking either when it comes to chess.

I just don't understand how a thinking human can not see that this is basically irrelevant.

Take a look at the sibling. Maybe be less than understanding but it’s a lot more than a placeholder.

It's really quite amazing. Inspired by your example, I tried to make it create something which nobody could have thought about before:

Write a deep poem about life, death, and the website Rotten Tomatoes

The result:

Life is a fleeting flame

A dance in the endless game

We come and we go

Leaving our mark, but never to know

What the future holds or who's to blame

Death is the great unknown

A mystery, never to be shown

But Rotten Tomatoes, a website so bold

Attempts to rate our stories untold

As if life and death can be bought and sold.

But in the end, what does it matter?

The ratings, the reviews, the bitter chatter

For in the grand scheme of things

We are but fleeting moments, the faintest of flings

And Rotten Tomatoes, a mere blip on the radar.

The top leaderboard spots for AoC day 2 were taken by people who passed the response directly into GTP3. https://twitter.com/max_sixty/status/1598924237947154433

The AoC challenges this early aren't difficult, but they have several steps and are significantly more challenging than something you would be able to find as a Stackoverflow answer.

This was the first solution, submitted very quickly after the problem was published.

For one problem. For the rest of the problems it has been very challenging to get the AI to write the correct solution. Still an impressive result that with specification, testing, and feedback the AI can come up with the correct result in the end.

Does it have access to the web? For one of my questions it answered:

"Unfortunately, I am unable to provide a detailed description of the education system in Poland and its changes over the last 30 years because I have limited access to information and cannot browse the internet."

but I have no idea if it's not lying :)

Some posts yesterday showed that network is usually disabled, but with the right prompt, you can enable network and someone got it to like a Twitter post.

I think the “liking Twitter post” part was just a coincidental joke.

Correct. He also claimed the chatbot signed in as Grime's Twitter account to do the action, which is obviously implausible. If you look in the replies you'll see him clarify it was a joke.

Thanks for the correction, guess I did not read that post close enough.

I would like to see how to achieve that. I tried asking it to translate the first paragraph of a site, but it provided more of an interpretation or summary of the article, rather than the actual text. When asked to copy the text from the website, it said that it couldn't. Additionally, when asked to provide a summary of a website that didn't exist before (created it myself yesterday), it gave a summary that was completely fictional, based on its interpretation of the URL.

You can check this by taking some time to create some specific weird puzzle that is unlikely to have been made in the format you come up with, then see if it can solve it. If you don't write it anywhere then it is being solved for the first time. Just make sure it is a pretty unusual puzzle.

These first few advent problems are extremely trivial. Solvable <1 min with experience programmers. And are at the level of someone with cs 101 knowledge.

Personally I don't see it being difficult for the AI to solve these trivial problems at all.

AlphaCode is solving much harder problems than these first few days of AoC

Yeah it can't solve novel questions. The AI couldn't do much to solve this one for example:

https://codeforces.com/contest/1672/problem/D

ChatGPT is just a very good copy/paste, not a logical problem solver(yet).

To be fair, humans also have great problems solving novel problems. Monkey see, monkey do!

Have you tried describing the algorithm?

https://imgur.com/a/7da1vFj

I used 'z' instead of 'a' to avoid any possible issues with the article 'a'. I think I messed up the assignment of z[l] at z[r] being after its updated, not sure.

But it created the input format, described it and the program ran the first time once i fixed the indents (code formatting is broken for some reason). If I run against the input at the contest page I get NO NO NO NO YES.

Isn't the expected output YES YES NO YES NO?

Yes according to the puzzle, but the operation as described isn’t clear (to me anyway) about which value should be used for the final assignment. It just says to use the value from the first element, but it’s unclear if it’s before or after that first element is replaced by the first operation.

I see, your description to GPT does not match the problem statement.

> It just says to use the value from the first element, but it’s unclear if it’s before or after that first element is replaced by the first operation.

Is the following python code unclear to you?

a = 0 b = 1 a, b = b, a

or alternatively:

a = [1,2,3,4,5] a[0:3] = a[1:3] + [a[0]]

There's nothing in the statement that indicates the assignments should be done one element at a time (if so, the order of the assignments would need to be specified). It's an atomic operation that circularly shifts the values in the array in the range l...r

I didn’t spend any time at all on it, I was curious to see if the results would be improved by describing the algorithm in prose rather than mathematical notation.

Reply on news.ycombinator.com