Hacker News Reader

Comment by tomohelix | Hacker News Reader

An AI can theoretically ingest 1M token of data, analyze, summarize, indexing, and storing the condensed information in a hard drive. Then later on retrieve it based on the metadata for specific situations that needs it. At millions of token, it is basically a superhuman that can learn new things on the fly at the speed unimaginable to any organic based life.

We are so close to something amazing, and scary.

View on HN

Do we yet have a single example of a transformer based AI, i.e. LLM, learning something new that we didn’t teach it in the training data?

Maybe we do.

I’m not sure how to define it, but new should be a discovery or insight or even relationship that is not explicitly taught in the training data.

If we don’t then it suggests without humans in the loop that the super intelligence is not so close?

GPT can learn things "in context" E.g. you can teach it something by chatting with it, but it will eventually forget it after its context length is exceeded. It cannot continuously learn and remember like biological organisms since its weights are frozen.

So we still have catastrophic forgetting, just on a slightly longer scale.

Except it cannot conceptualize reasonably. It is utterly incapable of symbolic thinking in a general sense.

It’s incapable of thinking. But what it’s surprisingly good at is symbolic reasoning similar to human common sense.

Most people forget that there is no memory or thought independent of whatever the output it generates.

But when you ask it to follow a chain of thought and generate it in the output, the eventual conclusions it lands on is scarily human.

People asking ChatGPT for word counts and sighing at how wrong it is. Yeah of course, it has no counter variables to hold numbers and increment them. It has no memory.

But when you ask it to generate a numbered list of all the words in a passage and then output the word count, it gets it right every time. Because you basically gave it a memory by encoding the counter in the generated output.

> But when you ask it to follow a chain of thought and generate it in the output, the eventual conclusions it lands on is scarily human

I have tried this so many times for novel problems and no matter what I do, it eventually recognizes what we're trying to solve and determines it's unsolvable. Since nothing new comes out I can only attribute the times it does seem to do symbolic manipulation as regurgitation.

No reason why Toolformer+GPT couldn't use a symbolic reasoning program. Symbolic reasoning programs are much better than humans.

Probably bugs will happen in the interface between human language and the symbolic reasoning. But that happens with humans too.

GPT-4 has toolformer capabilities as emergent properties, so using it in combination with toolformer is really unnecessary.

This is true too. Even chatGPT can learn some tool use in-context. See https://twitter.com/minosvasilias/status/1627076214639976449

GPT-4 writes some wicked complicated (but correct!) SQL if given a schema and a relevant task:

https://gist.github.com/int19h/428cea1d87dfc389b99de1b79727f...

What I found really amazing about this particular experiment is that the schema I gave it didn't contain any information that could be used to query for things like distances between places, and yet it came up with the idea of using settlements' culture as a proxy to determine "border fiefs", completely unprompted (and yes, it actually is a very effective proxy for this particular case!).

I wonder now what it would do with Prolog - or maybe Datalog for simplicity? - although that might depend on how much of it there is in the training data. Do you know if anyone tried it yet?

That's wild.

I work in this field and have tried many things. Sure.. GPT + something else can do symbolic reasoning almost on par with a middle schooler (it gets things wrongs often, not in the way people do either, but just confidently incorrect). However, GPT + something is not a transformer model, which is what the original comment was about.

There are plenty of examples of "new" things a LLM can do.

A good example is all those toy examples of "Program a whatever in the style of Shakespeare and David Bowie's love child". This isn't a thing that it has seen in training data.

From my limited understanding of how LLMs work, I believe this behavior is enabled by embeddings. The model maps its ~50,000 token vocabulary into a lower-dimensional vector space. Each dimension in the vector adds some sort of meaning (or at least association) to the word.

I saw an example in a Numberphile video where they were able to take the vector of the word "prince", subtract the vector of the word "man", add the vector of the word "woman", and the resulting vector was closest to the word "princess". So in theory there could be a "gender" dimension and a "position of authority" dimension in that vector (or the model might be making other, stranger connections between words that we don't understand).

I think the same thing is happening in your example. The model identifies and produces output that keeps it in the general region in the vector space for the "Shakespearean" and "Bowiean" dimensions while still satisfying other requirements.

Not saying it is not fascinating but, isn't this program something 50% in the style of Shakespeare and 50% in the style of David Bowie?

I don't think that's really "new". That's combining two existing styles that the LLM has seen in its training data, and the creative idea to combine those styles has been supplied by the operating human.

It's phenomenally impressive, and may be a stepping stone to models that can come up with new ideas, but I don't think we're there yet.

LLMs seem to be able to capture the idea of Shakespeare and Bowie's styles, and intuit a combination of the two, but when I start asking it questions about what it thinks about the process I don't get the impression of any understanding. It can magic up text from prompts but it doesn't understand what it's doing.

Reply on news.ycombinator.com