Hacker News Reader

Comment by ftxbro | Hacker News Reader

The context is that John Carmack got $20M to try to make AGI by 2030, based on non-LLM methods. In this tweet he is boosting the idea that there is a gigantic learning gap between the statistical efficiency of human and LLM learning, even though there may not be such a huge gap in standardized test results of LLMs and humans. This means that he might yet find a cool idea that could justify the time and money he is spending to find secret knowledge of the ancients from like pre-1990s AI publications that didn't have GPU access. Maybe it will happen!

https://the-decoder.com/john-carmacks-general-artificial-int...

View on HN

In a way LLMs are the worst thing that could have happened in the search for artificial intelligence, instead of the harsh winter from the 1990s we are entering now a climate changed winter: warm, products to market race winter.

Funnily enough, the Geoffrey Hinton of today is probably some symbols researcher, shouting in the desert, like Hinton shouted in the 1980s, that we need more than matmul.

One interesting, biology-inspired mechanism, would be quorum sensing [1]: a basal cognition-like, decision-making function in which decentralized systems (bacteria, cells) start building functionality (sensing/decision) from the bottom up. There is no hint for this sort of mechanism in our current artificial 'neural networks'. Not that it should, but our cells and in general cells use these kind of 'tricks' to solve problems in all kinds of spaces (transcriptomics, morphogenetics, etc.) without requiring ridiculous amounts of energy, time, or other resources.

[1] https://en.wikipedia.org/wiki/Quorum_sensing

LLMs may be terrible for the search for AGI but that may be good thing. If, mind you if, LLMs are always going to semi-smart "parrots", then society will get a picture of what an AI might without the AI being able to skynet-style takeover people are maybe justifiably worried.

Now, as far as whether LLM or offshoots can get to AGI, I know plenty of good arguments for them not being able to do that. I don't think the claim that they're just accumulating more abilities in each iteration in an inexplicable way is true. But I've lived long enough to know that you should never get too cocky when one is "arguing with success". So maybe.

Moreover, given that LLM programming is basically just bucket chemistry, if an LLM can the ability to competently pursue long term goals, it seems like it will have a good chance of some of its goals being random cruft that will make it quite dangerous.

When some hybrot [1] will write the history of the 2000s, after AES 256 is broken, we will probably find out that the Skynet takeover already happened, it just wasn't as dramatic as a James Cameron movie: it was a takeover through financialization (with enough money you can rewrite the past, so Mr. Cameron was right on the money about the time machine), it involved some suits, a glass building, and a generally apathetic, divided by design society: "as of 2020, Aladdin managed $21.6 trillion in assets" [2]. Nowadays, Mr. Sam Altman, who probably has good intentions, is speaking of OpenAI and $100 trillion [3] being unleashed into the market. There will be people wanting those $100+ trillion for themselves, and our current system is being designed by those kinds of people, biased towards and fostering their success (Blackrock, Vanguard, State Street currently barely have $30 trillion under management, with world GDP being at $101.5 trillion in 2022 [4]).

Equating LLMs to parrots is offensive, to the parrots. Well, maybe not parrots, but crows are intelligent: "Scientists [5] demonstrate that crows are capable of recursion—a key feature in grammar. Not everyone is convinced" [6].

[1] https://en.wikipedia.org/wiki/Hybrot

[2] https://en.wikipedia.org/wiki/Aladdin_(BlackRock)

[3] https://www.nytimes.com/2023/03/31/technology/sam-altman-ope...

[4] https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nomi...

[5] "Recursive sequence generation in crows", https://www.science.org/doi/10.1126/sciadv.abq3356

[6] https://www.scientificamerican.com/article/crows-perform-yet...

Carmack is no doubt brilliant and it shows through Oculus as a product. But AGI is not about algorithm optimization the way 3D graphics were in the 90s. I am not sure the approach he takes (let’s find a way to write this in assembly) can apply here. He’s said before that AGI is going to be a few thousands of lines of code, ultimately [1], and it’s hard to believe that’s really the case.

That said, I did get a chance to speak with him one on one last year and he really emphasized a few things; the need for being product oriented and giving customers what they want rather than chasing cool engineering (über)solutions; not being careless with resources just because we have more power with modern hardware (he poked fun at React where you spin up a new thread just for an interactive button); and being aware of the inefficiencies brought on by infinite resources (# of engineers and/or funding, which make you think less critically about timelines and delivering within bounded means)

[1] https://youtu.be/I845O57ZSy4

The guy also tried to build a rocket to reach space via "first principles." It failed, utterly.

They had a low budget relative to most of the others and reached break-even profitability but he moved on to other stuff. Where is the trying to reach space by "first principles" quote from?

It did the thrust vectoring thing. The rocket was ahead of the competition for a moment.

He said games were one of the most complex things humans build and (with implied comparison) the mathematics and physics of rocketry hadn't changed much since the 1960s. Sounds true to me for the 2000s.

Where was the utter failure and first principles?

>> But AGI is not about algorithm optimization the way 3D graphics were in the 90s

Wow, you think his ability is straight performance optimisation? A big part of his early fame was from doing research to find good algorithms to achieve his goals. He also had that stint in real time control systems... flying hovering rockets before spaceX even existed.

He's a problem solver with a strong ability to sift through possible solutions for what actually works, and quite capable of devising his own solutions when is research comes up empty.

But yeah, he can write assembler too.

Many problems can be cast as optimization problems, can the problem of AGI be cast as one? Let's let carmack answer.

It’s definitely Good to not put all our eggs in the LLM basket. Even openAI says it’s only a part of the solution - but it has scaled more than most anyone expected. Will that continue forever and we keep getting more impressive emergent behaviors from it? I doubt that but I could certainly be wrong.

It still seems to be missing any sense of what is True, not sure if that’s possible to embed in that model or if we’ll just have some human feedback hacks and eventually get a better model

That debate is becoming irrelevant. Sure it would be great if GTP-6 cost 50k to train instead of 100M. But even if costs 100M it will produce many billions of dollars of value, so the original investment is still relatively small.

Yeah but the difference between those two numbers is one is accessible to individuals (albeit wealthy ones), the other is accessible only to large-ish companies.

> Will that continue forever and we keep getting more impressive emergent behaviors from it? I doubt that

I feel like putting the word 'forever' ruins the point. It's the most extreme strawman.

It's amazing how the scaling has unlocked the emergent behaviors! When I look at the scaling graphs, I see that the ability to reduce 'perplexity' is continuing with scale and capital investment with no sign of slowing yet (it will slow eventually). I also see that reducing perplexity is continually unlocking new emergent behaviors. So I would guess that scaling will probably unlock so many more new emergent behaviors before it eventually plateaus!

IMO, the capabilities have not improved that much between GPT-3 vs GPT-4, the primary difference is just in the size of the context the model can handle at any one time (Max tokens).

While I definitely get "longer" and slightly more "in-depth" answers from GPT-4 vs 3, it already feels like that capability growth curve is starting to plateau.

> IMO, the capabilities have not improved that much between GPT-3 vs GPT-4

I strongly disagree. Anyone who wants to look for themself can see the GPT 4 technical report.

https://arxiv.org/pdf/2303.08774.pdf

Page 9 pretty much stopped me dead in my tracks. There's no shortage of humans, even technically-savvy ones, who wouldn't answer that question correctly.

That is incredibly impressive. Reminds me of these other examples from Microsoft’s “sparks of AGI” paper (summary link): https://ibragimov.org/2023/03/24/sparks-of-agi.html

The distinction may appear subtle in many cases, but in others GPT4 blows 3 away. E.g. recognizing when it doesn't know and admitting it's hypothesising is something I've run into with 4 but not even 3.5.

The capability curve will necessarily appear to plateau when the starting point is as good as it is now. The improvements we recognize will be subtler. Halving the remaining error rate will look less impressive for each step.

It depends how you see good. The capacities of Gpt-3.5/ChatGPT not close to perfect.

ChatGPT is quite good at something like "putting together facts using logic". Things medical diagnosis or legal argument. However, if the activity is "reconciling a summary with details", ChatGPT is pretty reliably terrible. My general recipe is "ask for a summary of a work of fiction, then ask about the relationship of the summary to details you know in the work." The thing reliably spits out falsehood in this situation.

I am yet to see GPT-4 admit to not knowing something rather than just hallucinating made up shit.

If it could reliably tell me when it DOESN'T know something, I'd have a lot more respect for it's capabilities. As it stands today, I'd feel I need to fact check nearly anything it gave me if I'm in an environment that requires high levels of factual accuracy.

Edit: To be clear, I mean it telling me it doesn't know something BEFORE hallucinating something incorrect and being caught out on it by me. It will admit that it lied, AFTER being caught, but it will never (in my experience) state that it doesn't have an answer for something upfront, and will instead default to hallucinating.

Also - even when it does admit to lying, it will often then correct itself with an equally convincing, but often just as untrue "correction" to its original lie. Honestly, anyone who wants to learn how to gaslight people just needs to spend a decent amount of time around GPT-4.

Well, I have. It still certainly goes the way you're describing often as well, to the point I was flabbergasted when it happened, but it did. Specifically I asked it to give me examples of using an NPM module published after the knowledge cutoff.

[To be clear, I did not tell it it was from after the cutoff]

Is there any information available about the non-LLM methods Carmack's AGI startup is researching?

Reply on news.ycombinator.com