Hacker News Reader

It seems to me that much of both the hype and the hate around ChatGPT is a gut reaction from people who haven't tried to incorporate it in their workflow for at least a month, understand its limitations and strengths, and only then comment on its ineptitude or world changing abilities.

The fact that it can instantly reply with a broad consensus of the ideas it has been fed is precisely what I need from it. When I ask it to write a function that performs a binary search in Rust with a given set of constraints, I'm not looking into original research or "truth", I want precisely the boring boilerplate I'm too lazy to write. For that use-case, it's invaluable and awe inspiring.

It's limits are laughable too, try as I might I could not make it count the numbers in a string of '9's that fit in a 128bit integer. It hallucinated fictional names of tools that I should use, such as "cobol_to_rust".

It seems clear to me that both camps have profound insights and LLM are both invaluable and wholly insufficient to reach any form of substantial "Artificial Intelligence".

People do a terrible job of binary search to the extent that it is towards the back of my toolbox. You can get eaten alive by off-by-one errors, mishandled integer overflows, etc. Human-written binary searches work most of the time but often fail for special inputs. I wouldn’t trust a machine that learned by reading 1000s of other people’s binary searches any more than I’d trust one I found off StackOverflow unless it came with a detailed explanation of the corner cases…. It is one to look up in an algorithms book, a recent algorithms book.

I know that you are reacting to particular comment but Chomsky's article is not denying usefulness of ML in specific domains.

I asked ChatGPT to assume "word" is a number, ignore numeric digits (ie 0-9) and assume the characters in "word" are digits. Also assume "word" contains the largest possible digit. What would it conclude the base to be.

Well...ChatGPT has been struggling. It generally struggles with anything subtle, no matter how simple. If it can't regurgitate based on what's slurped up from the internet it struggles.

Do note, my question is simply meant to assume i am using a-w as my digits and "word" is a number formed via a positional number system. It is something a human with requisite knowledge will veer towards, ChatGPT doesn't.

To your point, it is indeed an awe-inspiring tech for the thing it is good at, ie replacing the current crop of search engines.

To be fair, the way inputs are tokenised does make per character operations much harder than you would expect. If OpenAI had instead made one character one token, then it would be worse in most senses (slower and less history) but would do much better at that kind of task.

>> I asked ChatGPT to assume "word" is a number, ignore numeric digits (ie 0-9) and assume the characters in "word" are digits. Also assume "word" contains the largest possible digit. What would it conclude the base to be.

For your intended use-case (as explained in your comment) shouldn't you be phrasing the second part of this specification as "treat the characters in "word" as if they are digits"?

The way you worded it introduces some ambiguity. The ambiguity can be resolved by a process of elimination - I had to think of what could possibly happen if I "assumed the characters in 'word' are digits" and realise that it doesn't make sense, so it must mean you want to use the characters as digits.

ChatGPT and LLMs in general really can't be expected to deal with that kind of ambiguity, not least because it's not the entire story (you also need another little logical jump, from "treat characters as digits" to "treat characters as denoting their positional values in the alphabet" or some such).

I'm saying, you're asking ChatGPT to take many reasoning steps, when everyone is having a big debate about whether ChatGPT and LLMs can reason _at all_. To make any heads or tails of ChatGPT's abilities, one needs to start with simple examples, and then get to the more complex stuff.

On the other hand, I expect that if you clarified your question more, or perhaps stated it just the right way, there's a good chance you'd get a correct answer. After all, the idea of counting with characters is very common and it's hard to believe ChatGPT hasn't seen a few thousands of examples of it during training.

> The fact that ChatGPT cannot tell if anything is real or correct or not is not a small thing that one can hand-wave away and say, "Well, maybe tomorrow." The inability to discern truth is built into the LLM method. This is not true of people. Certainly not intelligent ones.

To be fair, ChatGPT is not a pure predictive LLM. It also undergoes RLHF, a reinfocement learning step which is much more similar to human learning, and allows it to develop a more human-like character, and seems to give it a pretty good grasp on truth(telling), ethics and context, certainly compared to the behavior of a raw predictive LLM with no prompt engineering.

I recommend read this blog post for a more nuanced understanding of LLMs: https://astralcodexten.substack.com/p/janus-simulators

I am way, way, way far from an expert on AI. Not even particularly interested in it. But is it really fair to say these models just give you the next likeliest word based on training? If that were true shouldn't you be able to replicate what they can do with a really advanced Markov chain?

> The fact that ChatGPT cannot tell if anything is real or correct or not is not a small thing that one can hand-wave away

I don't disagree with your premise. But this applies equally to human intelligence. There's no way to ever tell if the reality you experience is real or not. This is a fundamental fact of existence that we hand wave away so that we can continue to interact with and learn about the world. My favorite thought experiment on this is Boltzmann Brains.

https://en.m.wikipedia.org/wiki/Boltzmann_brain