Hacker News Reader

> At some point it won't be necessary to train an LLM on the entirety of Reddit plus everything else ever written because there are obvious limits to statistical models like this and, as a counter point, that's not how humans learn. You may have read hundres of books in your life, maybe even thousands. You haven't read a million. You don't need to.

I agree but I think it may be privileging the human intelligence mechanism a bit too much. These LLMs are polymaths that can spit out content at a super human rate. It can generate poetry and literature similarly to code and answers about physics and car repair. It’s very rare for a human to be able to do that especially these days.

So I agree they’re transitional but only in the sense that our brains are transitional from the basal ganglia to the neocortex. In that sense I think LLMs will probably be a part of a future GAI brain with other things tracked on, but it’s not clear it will necessarily evolve to work like a human’s brain does.

I think the actual reason people can't do it is that we avoid situations with high risk and no apparent reward. And we aren't sufficiently supportive of other people doing surprising things (so there's no reward for trying). I.e. it's a modern culture problem, not a human brain problem.

> These LLMs are polymaths that can spit out content at a super human rate.

Do you mean in theory or currently? Because currently, LLMs make simple errors (eg [1]) and are more capable of spitting out, well, nonsense. I think it's safe to say we're a long way from LLMs producing anything creatively good.

I'll put it this way: you won't be getting The Godfather from LLMs anytime soon but you can probably get an industrial film with generic music that tells you how to safely handle solvents, maybe.

Computers are generally good at doing math but LLMs generally aren't [2] and that really demonstrates the weaknesses in this statistical approach. ChatGPT (as one example) doesn't understand what numbers are or how to multiply them. It relies seeing similar answers to derive a likely answer so it often gets the first and large digits of the answer correct but not the middle. You can't keep scaling the input data to have it see every possible math question. That's just not practical.

Now multiplying two large numbers is a solvable problem. Counting Rs in strawberry is a solvable problem. But statistical LLMs are going to have a massive long tail of these problems. It's really going to take the next generational change to make progress.

[1]: https://www.inc.com/kit-eaton/how-many-rs-in-strawberry-this...

[2]: https://www.reachcapital.com/2024/07/16/why-llms-are-bad-at-...

Both the "count the Rs in strawberry" and the "multiply two large numbers" things have been solved for over a year now by the tool usage pattern: give an LLM the ability to delegate to a code execution environment for things it's inherently bad at and train it how to identify when to use that option.

I think the point is that playing whack a mole is an effective practical strategy to shore up individual weaknesses (or even classes of weaknesses) but that doesn’t get you to general reasoning unless you think that intelligence evolved this way. Given the adaptability of intelligence across the animal kingdom to novel environments never seen before, I don’t think that can be anything other than a short term strategy for AGI.

Sure, LLMs won't ever get to general reasoning (for pick your definition of "reasoning") unassisted.

I think that adding different forms of assistance remains the most interesting pattern right now.

(I did an earlier attempt at this with a "ok, longer conversation" ... and then did a "well, what if I just asked it directly?")

https://chatgpt.com/share/670bfdbd-8624-8011-bc31-2ba66eab3e...

I didn't realize that it had come that far with the delegating of those problems to the code writing and executing part of itself.

I think we’re in agreement. It’s going to take next generation architecture to address the flaws where the LLM often can’t even correct its mistake when it’s pointed out as with the strawberry example.

I still think transformers and LLMs will likely remain as some component within that next gen architecture vs something completely alien.

> You may have read hundres of books in your life, maybe even thousands. You haven't read a million. You don't need to.

Sometimes online forums are the only place where you can find solutions for niche situations and edge cases. Tricks which would have been very difficult to figure out on your own. LLMs can train on the official documentation of tools l/libraries but they can't experiment and figure out solutions to weird problems which are unfortunately very common in tech industry. If people stop sharing such solutions with others, it might become a big problem.

"LLMs can train on the official documentation of tools l/libraries but they can't experiment and figure out solutions to weird problems"

LLMs train on way more than just the official documentation: they train on the code itself, the unit tests for that code (which, for well written projects, cover all sorts of undocumented edge-based) and - for popular projects - thousands of examples of that library being used (and unit tested) "in the wild".

This is why LLMs are so effective at helping figure out edge-cases for widely used libraries.

The best coding LLMs are also trained on additional custom examples written by humans who were paid to build proprietary training data for those LLMs.

I suspect they are increasingly trained on artificially created examples which have been validated (to a certain extent) through executing that code before adding it to the training data. That's a unique advantage for code - it's a lot harder to "verify" non-code generated prose since you can't execute that and see if you get an error.

> they train on the code itself, the unit tests for that code

If understanding the code was enough, we wouldn't have any bugs or counterintuitive behaviors.

> and - for popular projects - thousands of examples of that library being used (and unit tested) "in the wild".

If people stopped contributing to forums, we won't have any such data for new things that are being made.

The examples I'm talking about come from openly licensed code in sources like GitHub, not from StackOverflow.

I would argue that code in GitHub is much more useful, because it's presented in the context of a larger application and is also more likely to work.

> Sometimes online forums are the only place where you can find solutions for niche situations and edge cases.

That's the most valuable aspect of it. When you find yourself in these niches situations, it's nice when you see someone has encountered it and has done the legwork to solve it, saving you hours and days. And that's why Wikis like the Arch Wiki are important. You need people to document the system, not just individual components.

the Gentoo Wiki is also high on my list, actually first because Gentoo is my preferred choice, equal in terms of useful content, only second in userbase - to Arch. Gentoo also implemented a policy prohibiting people from contributing LLM output. Theres not a culty rule-following edit-sanitization mod team, and the wiki doesnt require special permission, just the honor system, but so far its enough to make me like contributing to it is a form of charity to help future mankind.

> that's not how humans learn

I've been thinking about this a lot lately. Could we train an AI, e.g. using RL and GAN, where it gets an IT task to perform based on a body of documentation, such that its fitness would then be measured based on both direct success on the task, and on the creation of new (distilled and better written) documentation that would allow an otherwise context-less copy of itself to do well on the task?

Just to add to your point, consider a book like "Finite and Infinite" games. I think I can "recreate" the knowledge and main thesis in the book by my readings from other areas.

'Listening to different spiritual gurus saying the same thing using different words' is like 'watching the same coloured glass pieces getting rearranged to form new patterns in kaleidoscope'

Only half true as maybe reasoning and actual understanding is not the strength of LLMs but it is fascinating that they actually can produce good info from everything they have read - unlike me who only read a fraction of that. Maybe dumb, but good memory.

So I think future AI has to read also everything if it is used like ChatGPT these days by average people to ask for advice about almost anything.

> (which is theft, to be clear)

> Another issue is how much IP consumption constitutes theft? If an LLM watches every movie ever made, that's probably theft.

It's hard to reconcile those two views, and I don't think theft is defined by "how much" is being stolen.