Hacker News Reader

Comment by barishnamazov | Hacker News Reader

The turkey is fed by the farmer every morning at 9 AM.

Day 1: Fed. (Inductive confidence rises)

Day 100: Fed. (Inductive confidence is near 100%)

Day 250: The farmer comes at 9 AM... and cuts its throat. Happy thanksgiving.

The Turkey was an LLM. It predicted the future based entirely on the distribution of the past. It had no "understanding" of the purpose of the farmer.

This is why Meyer's "American/Inductive" view is dangerous for critical software. An LLM coding agent is the Inductive Turkey example. It writes perfect code for 1000 days because the tasks match the training data. On Day 1001, you ask for something slightly out of distribution, and it confidently deletes your production database because it added a piece of code that cleans your tables.

Humans are inductive machines, for the most part, too. The difference is that, fortunately, fine-tuning them is extremely easy.

View on HN

> The Turkey was an LLM. It predicted the future based entirely on the distribution of the past. It had no "understanding" of the purpose of the farmer.

But we already know that LLMs can do much better than that. See the famous “grokking” paper[1], which demonstrates that with sufficient training, a transformer can learn a deep generalization of its training data that isn’t just a probabilistic interpolation or extrapolation from previous inputs.

Many of the supposed “fundamental limitations” of LLMs have already been disproven in research. And this is a standard transformer architecture; it doesn’t even require any theoretical innovation.

[1] https://arxiv.org/abs/2301.02679

I'm a believer that LLMs will keep getting better. But even today (which might or might not be "sufficient" training) they can easily run `rm -rf ~`.

Not that humans can't make these mistakes (in fact, I have nuked my home directory myself before), but I don't think it's a specific problem some guardrails can solve currently. I'm looking for innovations (either model-wise or engineering-wise) that'd do better than letting an agent run code until a goal is seemingly achieved.

LLM's have surpassed being Turing machines? Turing machines now think?

LLM's are known properties in that they are an algorithm! Humans are not. PLEASE at the very least grant that the jury is STILL out on what humans actually are in terms of their intelligence, that is after all what neuroscience is still figuring out.

This issue happens at the edge of every induction. These two rules support their data equally well:

data: T T T T T T F

rule1: for all i: T

rule2: for i < 7: T else F

That’s where Bayesian reasoning comes into play, where there are prior assumptions (e.g., that engineered reality is strongly biased towards simple patterns) which make one of these hypotheses much more likely than the other.

yes, if you decide one of them is much more likely without reference to the data, then it will be much more likely :)

Deciding that they are both equally likely is also a deciding a prior.

Yes, "equally likely" is the minimal information prior which makes it best suited when you have no additional information. But it's not unlikely that have some sort of context you can use to decide on a better prior.

Well that would be extra information. Wherever you find the edge of your information, you will find the "problem of induction" as presented above.

AGI is when turkey cuts farmer's throat on day 249, gets on farmer's internet, makes money on trading and retires on an island.

> The difference is that, fortunately, fine-tuning them is extremely easy.

Because of millions of years of generational iterations, by which I mean recursive teaching, learning and observing, the outcomes of which all involved generations perceive, assimilate and adapt to in some (multi-) culture- and sub-culture driven way that is semi-objectively intertwined with local needs, struggles, personal desires and supply and demand. All that creates a marvelous self-correcting, time-travelling OODA loop. []

Machines are being finetuned by 2 1/2 generations abiding by exactly one culture.

Give it time, boy! (effort put into/in over time)

[] https://en.wikipedia.org/wiki/OODA_loop

"fine-tuning them is extremely easy." Criminal courts, jails, mental asylums beg to disagree.

"finetune"

Not

"Train from scratch"

> The difference is that, fortunately, fine-tuning them is extremely easy.

If this was true, educating people fast for most jobs would be a really easy and solved problem. On the other hand in March 2018, Y Combinator put exactly this into its list of Requests for Startups, which gives strong evidence that this is a rather hard, unsolved problem:

> https://web.archive.org/web/20200220224549/https://www.ycomb...

Easier than to an LLM, compared to inference.

“‘r’s in strawberry” and other LLM tricks remind me of brain teasers like “finished files” (https://sharpbrains.com/blog/2006/09/10/brain-exercise-brain...). Show an average human this brain teaser and they’ll probably fall for it the first time.

But never a second; the human learned from one instance, effectively forever, without even trying. ChatGPT had to be retrained and to not fall for the “r”’s trick, which cost much more than one prompt, and (unless OpenAI are hiding a breakthrough, or I really don’t understand modern LLMs) required much more than one iteration.

That seems to be the one thing that prevents LLMs from mimicking humans, more noticeable and harder to work around than anything else. An LLM can beat a Turing test where it only must generate a few sentences. No LLM can imitate human conversation over a few years (probably not even a few days), because it would start forgetting much more.

The problem with education is that existing ways of doing things are very strongly entrenched.

At the school level: teachers are trained, buildings are built, parents rely on kids being at school so they can go out to work....

At higher levels and in training it might be easier to change things, but IMO it is school level education that is the most important for most people and the one that can be improved the most (and the request for startups reflects that).

I can think of lots of ways things can be done better. I have done quite a lot of them as a home educating parent. As far as I can see my government (in the UK) is determined to do the exact opposite of the direction I think we should go in.

> The problem with education is that existing ways of doing things are very strongly entrenched.

Which is still a problem of educating humans. Just moved up the chain one step. Educators are often very hard to educate.

Even mathematics isn't immune to this. Calculus is pervasively taught with prematurely truncated algebra of differentials. Which means for second order derivatives and beyond, the "fraction" notation does not actually describe ratios, when this does not need to be the case.

But when will textbooks remove this unnecessary and complicating disconnect between algebra and calculus? There is no significant movement to do so.

Educators and textbook writers are as difficult to educate as anyone else.

The one true result of education research is that one on one education is vastly more effective than classroom education.

While I have no doubt you had good results home schooling, you will almost certainly run into difficulty scaling your results.

Not as much as you might think for two reasons.

1. Kids need far fewer hours of one on one than classroom teaching

2. There is much greater proportion of self teaching, especially as kids get older.

I estimate adult time required per child is similar to schools with small class sizes, and it requires somewhat less skilled adults.

LLM’s seem to know about farmers and turkeys though.

You clearly underestimate the quality of people I have seen and worked with. And yes guard rails can be added easily.

Security is my only concern and for that we have a team doing only this but that's also just a question of time.

Whatever LLMs ca do today doesn't matter. It matters how fast it progresses and we will see if we still use LLMs in 5 years or agi or some kind of world models.

> You clearly underestimate the quality of people I have seen and worked with.

I'm not sure what you're referring to. I didn't say anything about capabilities of people. If anything, I defend people :-)

> And yes guard rails can be added easily.

Do you mean models can be prevented to do dumb things? I'm not too sure about that, unless a strict software architecture is engineered by humans where LLMs simply write code and implement features. Not everything is web development where we can simply lock filesystems and prod database changes. Software is very complex across the industry.

I know plenty of people who are shittier in writing code than Claude. People with real jobs who are expensive like 50-100k/year.

People whom you have to always handhold and were code review is fundamental.

You can write tests, pr gates etc.

It's still a scale in what you can let them do unsupervised vs controlling them more closely but already better than real people I know. Because they are also a lot faster.

Ah, you meant I overestimate the quality of people, not underestimate. I think that really depends. I have worked with some amazing people :-)

Low quality engineering is always visible to outside. Low quality engineers using LLMs won't get any better.

That shocked me the most tbh

I had big hopes for one person until he copy pasted verbatim LLM responses to me...

> You clearly underestimate the quality of people I have seen and worked with

"Humans aren't perfect"

This argument always comes up. The existence of stupid / careless / illiterate people in the workplace doesn't excuse spending trillions on computer systems which use more energy than entire countries and are yet unreliable

It does.

If you have 1% of them and they cost you 50-100k per year than replacing them with computers make plenty of sense.

Reply on news.ycombinator.com