Comment by thomasfromcdnjs | Hacker News Reader

Comment by thomasfromcdnjs | Hacker News Reader

Parent Full thread

thomasfromcdnjs

I'm a bit more open minded to what "reason" or our own understanding of our own "reason" might be.

I gave GPT-4 a bunch of code that had two regex functions to parse a given custom url. I also gave it in the same prompt an example of a url of what I wanted it to parse, and asked it to explain the code and tell me the answer.

It explained the code perfectly in moments and also gave me the correct answer.

Here was it's response -> https://pastebin.com/raw/CP9iZh7S

I would use this as an interview question, asking a junior dev to "reason" through the code, explain their thinking and without running it, tell me what they might think the answer would be. If I had to give them feedback for that answer, I would have commended their ability to "reason" about the problem. ¯\_(ツ)_/¯

(extra: The GPT-4 response knew to not bother with the second regex because it should only run the second if the first regex had a third part match, amazing...)

(side note: I can't wait until I can generate live documentation of every file in my code base)

I'm not surprised it can write that regex and explain it, because that's a pretty simple regex and regexes generally are very well documented. Here's something it can't do:

I was writing a python script that took a list of tuples with x elements. Each element in the tuple represented a value in a dataset, so the first dataset would be the list of elements at position 0 in each tuple, the second dataset would be the list of elements in positions 1 etc. I wanted to extract each dataset into a list, calculate the derivate of each element in the list, and then reassemble the original list of tuples with the derivates in place of the original values.

I tried to get ChatGPT to give me a one liner to do this for _hours_. It would generally get step one and two correct, splitting the list of tuples of x length into x lists, and then calculating the derivative of each element in the list, but inevitably at the last step it would do something screwy. Most commonly it would reassembly a list of tuples, except the elements of each tuple would be from the indexes in the _second_ list. So I would get a list of all the derivates in position one in every dataset. Even if I explained what was wrong with the output, it would apologize and then blithely give me the exact same answer! Why did it do this? Because it doesn't actually understand the prompt, and the answer to my question is not in its dataset in a way that can be statistically derived.

3.5 or 4?

And did you also try the problem with Co-Pilot?

I've tried it with both, and it makes the same error. Copilot doesn't even get close. This is just one example of many. Beyond a certain degree of complexity, for problems that aren't documented in some tutorial or answer somewhere in its training dataset, ChatGPT is not going to be able to help you. I've run up hard against its limits many of times, and I feel like most of the examples people provide are actually a lot simpler than they believe them to be.

I'd be curious to take a stab on your problem. I could arguably say it is simpler as it is just purely documented mathematics.

And/or you could use a prompt injection strategy to make sure it does have the available context to solve your problem.

e.g. a vector search for some text book snippet that gets injected in

BTW have you asked it about " davidjl"? Maybe ask it about SolidGoldMagikarp too.

Reply on news.ycombinator.com