Hacker News Reader

Parent Full thread

richrichie

> the more data you feed into an LLM, the smarter it should be in the response

This is not obvious though.

View on HN

It’s in theory. The more information you have, the better the decision in theory.

It's quality not quantity.

You need to have accurate, properly reasoned information for better decisions.

It’s a good thing that SFDC and Slack are both well known for being a repository of high quality data.

/sarc

Not quite. There are bounds on capacity of learning machines.

https://en.wikipedia.org/wiki/Vapnik%E2%80%93Chervonenkis_di...

Anyone got a TL;DR on this?

Think about an AI with a 1 bit model. If you feed that AI data that can't possibly be classified into less than 2 bits, it can't get it precisely right, no matter how much data you train it on, or what the 1 bit of the model represents.

For any given size of system, there will be a ceiling on what it can learn to classify or predict with precision.

I used "system" rather than "model" there for a reason:

Memory in any form, such as context and RAG or API access to anything that can store and retrieve data affects the maximum - a turing machine can be implemented with a very small model + a loop if there's access to an external memory to act like the tape. But if the "tape" is limited, there will be some limitation on what the total system can precisely classify.

Reply on news.ycombinator.com