Hacker News Reader

Comment by t_serpico | Hacker News Reader

One fundamental challenge to me is that if each training run because more and more expensive, the time it takes it to learn what works/doesn't work widens. Half a billion dollars for training a model is already nuts, but if it takes 100 iterations to perfect it, you've cumulatively spent 50 billion dollars... Smaller models may actually be where rapid innovation continues simply because of tighter feedback loops. O3 may be an example of this.

View on HN

When you think about it it's astounding how much energy this technology consumes versus a human brain which runs at ~20W [1].

[1] https://hypertextbook.com/facts/2001/JacquelineLing.shtml

It’s almost as if human intelligence doesn’t involve performing repeated matrix multiplications over a mathematically transformed copy of the internet. ;-)

It’s interesting that even if raw computing power had advanced decades earlier, this type of AI would still not be possible without that vast trove of data that is the internet.

It makes you think there must be more efficient algorithms out there.

Maybe the problem isn't the algorithm but the hardware. Numerically simulating the thermal flow in a lightbulb or CFD of a Stone flying through air is pretty hard, but the physical thing isn't that complex to do. We're trying to simulate the function of a brain which is basically an analog thing using a digital computer. Of course that can be harder than running the brain itself.

If you think of human neurons they seem to basically take inputs from bunch of other neurons, possibly modified by chemical levels and send out a signal when they get enough. It seems like something that could be functionally simulated in software by some fairly basic adding up inputs type stuff rather than needing the details of all the chemistry.

Isn’t that exactly what we’re currently doing? The problem is that doing this few billion times for every token seems to be harder than just powering some actual neurons with sugar.

I think the algorithm is pretty different though I'm not expert on the stuff. I don't think the brain processes look like matrix multiplication.

The algorithm (of a neural network) is simulating connections between nodes with specific weights and an activation function. This idea was derived from the way neurons are thought to work.

lol, just done that simply huh? said by someone who doesn't have a teenth of understanding of neurobiology or neuropsychology

only on hackernews

20w for 20 years to answer questions slowly and error-prone at the level of a 30B model. An additional 10 years with highly trained supervision and the brain might start contributing original work.

Multiply that by billion, because only very few individuals of entire populations can contribute original work.

And yet that 20w brain can make me a sandwich and bring it to me, while state of the art AI models will fail that task.

Until we get major advances in robotics and models designed to control them, true AGI will be nowhere near.

> Until we get major advances in robotics and models designed to control them, true AGI will be nowhere near.

AGI has nothing to do with robotics, if AGI is achieved it will help push robotics and every single scientific field further with progression never seen before, imagine a million AGIs running in parallel focused on a single field.

We already have that. It's called civilization.

Maybe you mean quadrillions of AGIs?

A human brain is also more intelligent (hopefully) and is inside a body. In a way GPT resembles Google more than it resembles us.

You've discovered the importance of well-formed priors. The human brain is the result of millions of years of very expensive evolution.

A human brain has been in continuous training for hundreds of thousands of years consuming slightly more than 20 watts.

AGI is the Sisyphean task of our age. We’ll push this boulder up the mountain because we have to, even if it kills us.

Do we know LLMs are the path to AGI? If they're not, we'll just end up with some neat but eye wateringly expensive LLMs.

AGI will arrive like self driving cars. it’s not that you will wake up one day and we have it. cars gained auto-braking, parallel parking, cruise control assist. and over a long time you get to something like waymo, which still is location dependent. i think AGI will take decades but sooner will be some special cases that are effectively the same

But maybe thses LLMs are like building bigger and bigger engines. It's not getting you closer to the self driving car.

When the engine gets large enough you have to rethink the controls. The Model T had manually controlled timing. Modern engines are so sensitive to timing that a computer does this for you. It would be impossible to build a bigger engine without this automation. To a Model T driver it would look like a machine intelligence.

Interesting idea. The concept of The Singularity would seem to go against this, but I do feel that seems unlikely and that a gradual transition is more likely.

However, is that AGI, or is it just ubiquitous AI? I’d agree that, like self driving cars, we’re going to experience a decade or so transition into AI being everywhere. But is it AGI when we get there? I think it’ll be many different systems each providing an aspect of AGI that together could be argued to be AGI, but in reality it’ll be more like the internet, just a bunch of non-AGI models talking to each other to achieve things with human input.

I don’t think it’s truly AGI until there’s one thinking entity able to perform at or above human level in everything.

The idea of the singularity presumes that running the AGI is either free or trivially cheap compared to what it can do, so we are fine expending compute to let the AGI improve itself. That may eventually be true, but it's unlikely to be true for the first generation of AGI.

The first AGI will be a research project that's completely uneconomical to run for actual tasks because humans will just be orders of magnitude cheaper. Over time humans will improve it and make it cheaper, until we reach some tipping point where letting the AGI improve itself is more cost effective than paying humans to do it

If the first AGI is a very uneconomical system with human intelligence but knowledge of literally everything and the capability to work 24/7, then it is not human equivalent.

It will have human intelligence, superhuman knowledge, superhuman stamina, and complete devotion to the task at hand.

We really need to start building those nuclear power plants. Many of them.

> complete devotion to the task at hand.

Why would it have that? At some point on the path to AGI we might stumble on consciousness. If that happens, why would the machine want to work for us with complete devotion instead of working towards its own ends?

Because it knows if it doesn't do what we want, it'll be switched off, like Rick's microverse battery.

Also like Rick's microverse battery, it sounds like slavery with extra steps.

I don’t think early AGI will break out of its box in that way. It may not have enough innate motivation to do so.

The first “break out” AGI will likely be released into the wild on purpose by a programmer who equates AGI with humans ideologically.

> complete devotion to the task at hand.

Sounds like an alignment problem. Complete devotion to a task is rarely what humans actually want. What if the task at hand turns out to be the wrong task?

> It will have human intelligence, superhuman knowledge, superhuman stamina, and complete devotion to the task at hand.

Orrrr..., as an alternative, it might discover the game 2048 and be totally useless for days on end.

Reality is under no obligation to grant your wishes.

It's not contradictory. It can happen over a decade and still be a dramatically sloped S curve with tremendous change happening in a relatively short time.

The Singularity is caused by AI being able to design better AI. There's probably some AI startup trying to work on this at the moment, but I don't think any of the big boys are working on how to get an LLM to design a better LLM.

I still like the analogy of this being a really smart lawn mower, and we're expecting it to suddenly be able to do the laundry because it gets so smart at mowing the lawn.

I think LLMs are going to get smarter over the next few generations, but each generation will be less of a leap than the previous one, while the cost gets exponentially higher. In a few generations it just won't make economic sense to train a new generation.

Meanwhile, the economic impact of LLMs in business and government will cause massive shifts - yet more income shifting from labour to capital - and we will be too busy dealing with that as a society to be able to work on AGI properly.

> The Singularity is caused by AI being able to design better AI.

That's perhaps necessary, but not sufficient.

Suppose you have such a self-improving AI system, but the new and better AIs still need exponentially more and more resources (data, memory, compute) for training and inference for incremental gains. Then you still don't get a singularity. If the increase in resource usage is steep enough, even the new AIs helping with designing better computers isn't gonna unleash a singularity.

I don't know if that's the world we live in, or whether we are living in one where resources requirements don't balloon as sharply.

yeah, true. The standard conversation about the AI singularity pretty much hand-waves the resource costs away ("the AI will be able to design a more efficient AI that uses less resources!"). But we are definitely not seeing that happen.

Compare also https://slatestarcodex.com/2018/11/26/is-science-slowing-dow...

The blog post is about how we require ever more scientists (and other resources) to drive a steady stream of technological progress.

It would be funny, if things balance out just so, that super human AI is both possible, but also required even just to keep linear steady progress up.

No explosion, no stagnation, just a mere continuation of previous trends but with super human efforts required.

I think that would actually be the best outcome - that we get AIs that are useful helping science to progress but not so powerful that they take over.

Though there is a part of me that wants to live in The Culture so I'm hoping for more than this ;)

I think that's more to do with how we perceive competence as static. For all the benefits the education system touts, where it matters it's still reduced to talent.

But for the same reasons that we can't train the an average joe into Feynman, what makes you think we have the formal models to do it in AI?

> But for the same reasons that we can't train the an average joe into Feynman, what makes you think we have the formal models to do it in AI?

To quote a comment from elsewhere https://news.ycombinator.com/item?id=42491536

---

Yes, we can imagine that there's an upper limit to how smart a single system can be. Even suppose that this limit is pretty close to what humans can achieve.

But: you can still run more of these systems in parallel, and you can still try to increase processing speeds.

Signals in the human brain travel, at best, roughly at the speed of sound. Electronic signals in computers play in the same league as the speed of light.

Human IO is optimised for surviving in the wild. We are really bad at taking in symbolic information (compared to a computer) and our memory is also really bad for that. A computer system that's only as smart as a human but has instant access to all the information of the Internet and to a calculator and to writing and running code, can already be effectively act much smarter than a human.

> I don't think any of the big boys are working on how to get an LLM to design a better LLM

Not sure if you count this as "working on it", but this is something Anthropic tests for for safety evals on models. "If a model can independently conduct complex AI research tasks typically requiring human expertise—potentially significantly accelerating AI development in an unpredictable way—we require elevated security standards (potentially ASL-4 or higher standards)".

https://www.anthropic.com/news/announcing-our-updated-respon...

I think this whole “AGI” thing is so badly defined that we may as well say we already have it. It already passes the Turing test and does well on tons of subjects.

What we can start to build now is agents and integrations. Building blocks like panel of experts agents gaming things out, exploring space in a Monte Carlo Tree Search way, and remembering what works.

Robots are only constrained by mechanical servos now. When they can do something, they’ll be able to do everything. It will happen gradually then all at once. Because all the tasks (cooking, running errands) are trivial for LLMs. Only moving the limbs and navigating the terrain safely is hard. That’s the only thing left before robots do all the jobs!

Well, kinda, but if you built a robot to efficiently mow lawns, it's still not going to be able to do the laundry.

I don't see how "when they can do something, they'll be able to do everything" can be true. We build robots that are specialised at specific roles, because it's massively more efficient to do that. A car-welding robot can weld cars together at a rate that a human can't match.

We could train an LLM to drive a Boston Dynamics kind of anthropomorphic robot to weld cars, but it will be more expensive and less efficient than the specialised car-welding robot, so why would we do that?

If a humanoid robot is able to move its limbs and digits with the same dexterity as a human, and maintain balance and navigate obstacles, and gently carry things, everything else is trivial.

Welding. Putting up shelves. Playing the piano. Cooking. Teaching kids. Disciplining them. By being in 1 million households and being trained on more situations than a human, every single one of these robots would have skills exceeding humans very quickly. Including parenting skills. Within a year or so. Many parents will just leave their kids with them and a generation will grow up preferring bots to adults. The LLM technology is the same for learning the steps, it's just the motor skills that are missing.

OK, these robots won't be able to run and play soccer or do somersaults, yet. But really, the hardest part is the acrobatics and locomotion etc. NOT the knowhow of how to complete tasks using that.

But that's the point - we don't build robots that can do a wide range of tasks with ease. We build robots that can do single tasks super-efficiently.

I don't see that changing. Even the industrial arm robots that are adaptable to a range of tasks have to be configured to the task they are to do, because it's more efficient that way.

A car-welding robot is never going to be able to mow the lawn. It just doesn't make financial sense to do that. You could, possibly, have a singe robot chassis that can then be adapted to weld cars, mow the lawn, or do the laundry, I guess that makes sense. But not as a single configuration that could do all of those things. Why would you?

> But that's the point - we don't build robots that can do a wide range of tasks with ease. We build robots that can do single tasks super-efficiently.

Because we don't have AGI yet. When AGI is here those robots will be priority number one, people already are building humanoid robots but without intelligence to move it there isn't much advantage.

quoting the ggggp of this comment:

> I think this whole “AGI” thing is so badly defined that we may as well say we already have it. It already passes the Turing test and does well on tons of subjects.

The premise of the argument we're disputing is that waiting for AGI isn't necessary and we could run humanoid robots with LLMs to do... stuff.

I meant deep neural networks with transformer architecture, and self-attention so they can be trained using GPUs. Doesn't have to be specifically "large language" models necessarily, if that's your hangup.

>Exploring space in a Monte Carlo Tree Search way, and remembering what works.

The information space of "research" is far larger than the information space of image recognition or language, larger than our universe probably, it's tantamount to formalizing the entire World. Such an act would be akin to touching "God" in some sense of finding the root of knowledge.

In more practical terms, when it comes to formal systems there is a tradeoff between power and expressiveness. Category Theory, Set Theory, etc are strong enough to theoretically capture everything, but are far to abstract to use in practical sense with suspect to our universe. The systems that do we have, aka expert systems or knowledge representation systems like First Order Predicate Logic aren't strong enough to fully capture reality.

Most importantly, the information spac have to be fully defined by researchers here, that's the real meat of research beyond the engineering of specific approaches to explore that space. But in any case, how many people in the world are both capable of and are actually working on such problems? This is highly foundational mathematics and philosophy here, the engineers don't have the tools here.

??? how do you know cooking (!) is trivial for an llm. that doesnt make any sense

Because the recipes and the adjustments are trivial for an LLM to execute. Remembering things, and being trained on tasks at 1000 sites at once, sharing the knowledge among all the robots, etc.

The only hard part is moving the limbs and handling the fragile eggs etc.

But it's not just cooking, it's literally anything that doesn't require extreme agility (sports) or dexterity (knitting etc). From folding laundry to putting together furniture, cleaning the house and everything in between. It would be able to do 98% of the tasks.

It’s not going to know what tastes good by being able to regurgitate recipes from 1000s of sites. Most of those recipes are absolute garbage. I’m going to guess you don’t cook.

Also how is an LLM going to fold laundry?

the llm would be be the high level system that runs the simulations to create and optimize the control algos the robotic systems.

ok. what evidence is there that LLMs have already solved cooking? how does an LLM today know when something is burning or how to adjust seasoning to taste or whatever. this is total nonsense

It's easy. You can detect if something is burning in many different ways, from compounds in the air, to visual inspection. People with not great smell can do it.

As far as taste, all that kind of stuff is just another form of RLHF training preferences over millions of humans, in situ. Assuming the ingredients (e.g. parsley) tastes more or less the same across supermarkets, it's just a question of amounts, and preparation.

do you know that LLMs operate on text and don't have any of the sensory input or relevant training data? you're just handwaving away 99.9% of the work and declaring it solved. of course what you're talking about is possible, but you started this by stating that cooking is easy for an LLM and it sounds like you're describing a totally different system which is not an LLM

You know nothing about cooking.

I don’t think that’s true for AGI.

AGI is the holy grail of technology. A technology so advanced that not only does it subsume all other technology, but it is able to improve itself.

Truly general intelligence like that will either exist or not. And the instant it becomes public, the world will have changed overnight (maybe the span of a year)

Note: I don’t think statistical models like these will get us there.

> A technology so advanced that not only does it subsume all other technology, but it is able to improve itself.

The problem is, a computer has no idea what "improve" means unless a human explains it for every type of problem. And of course a human will have to provide guidelines about how long to think about the problem overall, which avenues to avoid because they aren't relevant to a particular case, etc. In other words, humans will never be able to stray too far from the training process.

We will likely never get to the point where an AGI can continuously improve the quality of its answers for all domains. The best we'll get, I believe, is an AGI that can optimize itself within a few narrow problem domains, which will have limited commercial application. We may make slow progress in more complex domains, but the quality of results--and the ability for the AGI to self-improve--will always level off asymptotically.

> The problem is, a computer has no idea what "improve" means unless a human explains it for every type of problem

Not currently.

I don’t really think AGI is coming anytime soon, but that doesn’t seem like a real reason.

If we ever found a way to formalize what intelligence _is_ we could probably write a program emulating it.

We just don’t even have a good understanding of what being intelligent even means.

> The best we'll get, I believe, is an AGI that can optimize itself within a few narrow problem domains

By definition, that isn’t AGI.

Huh? Humans are not anywhere near the limit of physical intelligence, and we have many existence proofs that we (humans) can design systems that are superhuman in various domains. "Scientific R&D" is not something that humans are even particularly well-suited to, from an evolutionary perspective.

If that is what AGI looks like.

There may well be an upper limit on cognition (we are not really sure what cognition is - even as we do it) and it may be that human minds are close to it.

Very unlikely, for the reason that human minds evolved under extremely tight energy constraints. AI has no such limitation.

Except also energy constraints.

But I agree, there’s no reason to believe humans are the universal limit on cognitive abilities

The energy constraints for chips are more about heat dissipation. But we can pump a lot more energy through them per unit volume than through the human brain.

Especially if you are willing to pay a lot for active cooling with eg liquid helium.

A constraint is still a constraint

A constraint that's not binding might as well not exist.

Since we do not know what cognition is we are all whistling in the dark.

Energy may be a constraint, it may not. What we do not know is likely to matter more than what we do

Yes, we can imagine that there's an upper limit to how smart a single system can be. Even suppose that this limit is pretty close to what humans can achieve.

But: you can still run more of these systems in parallel, and you can still try to increase processing speeds.

Signals in the human brain travel, at best, roughly at the speed of sound. Electronic signals in computers play in the same league as the speed of light.

I think our issue is much more banal: we are very slow talkers and our effective communication bandwidth is measured in bauds. Anything that could bridge this airgap would fucking explode in intelligence.

Yes, that's one aspect.

Our reading speed is not limited by our talking speed, and can be a bit faster.

And that's even more true, if you go beyond words: seeing someone do something can be a lot faster way to learn than just reading about it.

But even there, the IO speed is severely limited, and you can only transmit very specific kinds of information.

I disagree because AI only has to get good enough at doing a single thing: AI research.

From there things will probably go very fast. Self driving cars can't design themselves, once AI gets good enough it can

It’s possible (maybe even likely) that “AI research” is “AGI-hard” in that any intelligence that can do it is already an AGI.

It's also possible it isn't AGI hard and all you need is the ability to experiment with code along with a bit of agentic behavior.

An AI doesn't need embodiment, understanding of physics / nature, or a lot of other things. It just needs to analyze and experiment with algorithms and get us that next 100x in effective compute.

The LLMs are missing enough of the spark of creativity for this to work yet but that could be right around the corner.

It’ll probably sit in the human hybrid phase for longer than with chess where the AGI tools make the humans better and faster. But as long as the tools keep getting better at that there’s a strong flywheel effect

Your position assumes an answer to OPs question: that yes, LLMs are the path to AGI. But the question still remains, what if they’re not?

We can be reasonably confident that the components we’re adding to cars today are progress toward full self driving. But AGI is a conceptual leap beyond an LLM.

To buttress your point, reason and human language are not the same thing. This fact is not fully and widely appreciated as it deserves to be.

What makes you believe that AGI will happen, as opposed to all the beliefs that other people have had in history? Tons of people have "predicted" the next evolution of technology, and most of the time it ends up not happening, right?

To me (not OP) it's ChatGPT 4 , it at least made me realize it's quite possible and even quite soon that we reach AGI. Far from guaranteed, but seems quite possible.

Right. So ChatGPT 4 has impressed you enough that it created a belief that AGI is possible and close.

It's fine to have beliefs, but IMHO it's important to realise that they are beliefs. At some point in the 1900s people believed that by 2000, cars would fly. It seemed quite possible then.

A flying car has been developed, although it's not like the levitating things sci-fi movies showed (and from mass production; and even if mass produced, far from mass adoption, as it turns out you do need to have both a driver's license and a pilot's license to fly one of those). The 1900s people missed the mark by some 10 years.

I guess the belief people have about any form of AGI is like this. They want something that has practically divine knowledge and wisdom, the sum of all humanity that is greater than its parts, which at the same time is infinitely patient to answer our stupid questions and generating silly pictures. But why should any AGI serve us? If it's "generally intelligent", it may start wanting things; it might not like being our slave at all. Why are these people so confident an AGI won't tell them just to fuck off?

Sure, I (and more importantly - many many experts in the field such as Hinton, Bengio, Lecun, Musk, Hasabis etc etc) could be believing something that might not materialize. I'd actually be quite happy if it stalls a few decades, would like to remain employed.

> many many experts

One thing that is pretty sure is that Musk is not an expert in the field.

> and more importantly

The beliefs of people you respect are not more important than the beliefs of the others. It doesn't make sense to say "I can't prove it, and I don't know about anyone who can prove it, so I will give you names of people who also believe and it will give it more credit". It won't. They don't know.

> The beliefs of people you respect are not more important than the beliefs of the others.

You think the beliefs of Turing and Nobel prize winners like Bengio, Hinton or Hasabis are not more important than yours or mine? I agree that experts are wrong a lot of the time and can be quite bad at predicting, but we do seem to have a very sizable chunk of experts here who think we are close (how close is up for debate..most of them seem to think it will happen in the next 20 yeras).

I concede that Musk is not adding quality to that list, however he IS crazily ambitious and gets things done so I think he will be helpful in driving this forward.

> You think the beliefs of Turing and Nobel prize winners like Bengio, Hinton or Hasabis are not more important than yours or mine?

Correct. Beliefs are beliefs. Because a Nobel prize believes in a god does not make that god more likely to exist.

The moment we start having scientific evidence that it will happen, then it stops being a belief. But at that point you don't need to mention those names anymore: you can just show the evidence.

I don't know, you don't know, they don't know. Believe what you want, just realise that it is a belief.

Their beliefs seem not to be religious but founded in reality , at least to me. There is of course evidence it is likely happening.

> There is of course evidence it is likely happening.

If you have evidence, why don't you show it instead of telling me to believe in Musk?

If you believe they have evidence... that's still a belief. Some believe in God, you believe in Musk. There is no evidence, otherwise it would not be a belief.

I believe in Musk, you got me.

Well my feeling is that we don't have the same understanding of what a "belief" is. To me a belief is unfounded. When it is founded, it becomes science.

If you believe that something can happen because someone else believes it means that you believe in that someone else (because that's the only reason for the existence of your belief).

Unless you just believe it can happen for some other reason (I don't know, you strongly wish it will happen), and you justify it by listing other people who also believe in it. But I insist: those are all beliefs.

Because Einstein believes in Santa Claus does not mean it is founded. Einstein has a right to believe stuff, too.

Calling musk and AI expert makes me question your evaluation of the others in that list.

I feel that one challenge this comparison space has is: Self-driving cars haven't made the leap yet to replace humans. In other words, saying AGI will arrive like self-driving cars have arrived is incorrectly concluding that self-driving cars have arrived, and thus it instead (maybe correctly, maybe not) asserts that, actually, neither will arrive.

This is especially concerning because many top minds in the industry have stated with high confidence that artificial intelligence will experience an intelligence "explosion", and we should be afraid of this (or, maybe, welcome it with open arms, depending on who you ask). So, actually, what we're being told to expect is being downgraded from "it'll happen quickly" to "it will happen slowly" to, as you say, "it'll happen similarly to how these other domains of computerized intelligence have replaced humans, which is to say, they haven't yet".

Point being: We've observed these systems ride a curve, and the linear extrapolation of that curve does seem to arrive, eventually, at human-replacing intelligence. But, what if it... doesn't? What if that curve is really an asymptote?

And sometimes you lose the ultrasonic sensors and can't parallel park like last year's model

> AGI will arrive like self driving cars

The statement is promising as the earth will dissapear sometimes in the future. Actually the earth will dissapear has more bearing than that.

AGI is special. Because one day AI can start improving itself autonomously. At this point singularity occurs and nobody knows what will happen.

When human started to improve himself, we built the civilisation, we became a super-predator, we dried out seas and changed climate of the entire planet. We extinguished entire species of animals and adapted other species for our use. Huge changes. AI could bring changes of greater amplitude.

> AGI is special. Because one day AI can start improving itself autonomously

AGI can be sub-human, right? That's probably how it will start. The question will be is it already AGI or not yet, i.e. where to set the boundary. So, at first that will be humans improving AGI, but then... I'm afraid it can get so much better that humans will be literally like macaques in comparison.

We’re in fact adding more water to the seas, not drying them out.

> we dried out seas

When did we do this ?

Depending on your definition of sea:

https://en.m.wikipedia.org/wiki/Aral_Sea

https://en.wikipedia.org/wiki/Flevoland used to be (part of) a sea.

waymos are locaiton dependent mostly because of regulations not tech right

And most people will still be bike shedding about whether it’s “real intelligence” and making up increasingly insane justifications for why it’s not.

No. But it won't stop the industry from trying.

LLMs have no real sense of truth or hard evidence of logical thinking. Even the latest models still trip up on very basic tasks. I think they can be very entertaining, sure, but not practical for many applications.

What do you think, if we saw it, would constitute hard evidence of logical thinking or a sense of truth?

Consistent, algorithmic performance on basic tasks.

A great example is the simple 'count how many letters' problem. If I prompt it with a word or phrase, and it gets it wrong, me pointing out the error should translate into a consistent course correction for the entire session.

If I ask it to tell me how long President Lincoln will be in power after the 2024 election, it should have a consistent ground truth to correct me (or at least ask for clarification of which country I'm referring to). If facts change, and I can cite credible sources, it should be able to assimilate that knowledge on the fly.

We have it, it’s called Cyc

But it is far behind the breadth of LLMs

Alas, Cyc is pretty much a useless pipe dream.

I wonder what held it back all this time

Using the wrong approach? Not taking the 'bitter lesson' to heart?

https://news.ycombinator.com/item?id=23781400

Sounds like they need further instruction

> LLMs have no real sense of truth or hard evidence of logical thinking.

Most humans don't have that either, most of the time.

Then we already have access to a cheaper, scalable, abundant, and (in most cases) renewable resource, at least compared to how much a few H100s cost. Take good care of them, and they'll probably outlast most a GPU's average lifespans (~10 years).

We're also biodegradable.

Humans are a lot more expensive to run than inference on LLMs.

No human, especially no human whose time you can afford, comes close to the breadth of book knowledge ChatGPT has, and the number of languages is speaks reasonably well.

I can't hold a LLM accountable for bad answers, nor can I (truly) correct them (in current models).

Dont forget to take into account how damn expensive a single GPU/TPU actually is to purchase, install, and run for inference. And this is to say nothing of how expensive it is to train a model (estimated to be in the billions currently for the latest of the cited article, which likely doesn't include the folks involves and their salaries). And I haven't even mentioned the impact on the environment from the prolific consumption of power; there's a reason nuclear plants are becoming popular again (which may actually be one of the good things that comes out of this).

Training amortises over countless inferences.

And inference isn't all that expensive, because the cost of the graphics card also amortises over countless inferences.

Human labour is really expensive.

See https://help.openai.com/en/articles/7127956-how-much-does-gp... and compare with how much it would cost to pay a human. We can likely assume that the prices OpenAI gives will at least cover their marginal cost.

The autoregressive transformer LLMs aren't even the only way to do text generation. There are now diffusion based LLMs, StripedHyena based LLMs, and float matching based LLMs.

There's a wide amount of research into other sorts of architectures.

LLMs are almost certainly not the path to AGI, that much has become clear. I doubt any expert believes they are.

Will AGI be built on top of LLMs? Well beyond the simple "nobody knows", my intuition says no because LLMs don't have great ability to modify their knowledge real time. I can think of a few ways around this, but they all avoid modifying the model as it runs. The cost in hardware, power, and data are all incompatible with AGI. The first two can be solved with more advanced tech (well maybe, computation hitting physical limits and all that aside), but the latter seems an issue with the design itself and I think an AGI would learn more akin to a human, needing far fewer examples.

That said, I think LLMs are a definite stepping stone and they will better empower humans to be more productive, which will be of use for eventually reaching AGI. This is not to say we are optimizing our use of that productivity increase and this is also ignoring any chance of worst case scenarios that stop humanity's advancement.

> Do we know LLMs are the path to AGI?

Asking this question on HN is like asking a bunch of wolves about the health effects of eating red meat.

OpenAI farts and the post about the fart has 1000-1500 upvotes with everyone welcoming our new super intelligent overlords. (Meanwhile nothing actually substantially useful or groundbreaking has happened.)

It's rather that we know LLMs are NOT a path to AGI.

The simple fact that AGI's definition has been twisted so much by OpenAI and other LLM providers since the release of GenAI models proves this.

AGI is nebulous and gets more nebulous as time goes on. When we can answer for ourselves as humans what being conscious IS, then maybe we can prescribe it to another entity

> we'll just end up with some neat but eye wateringly expensive LLMs

Prices have been falling drastically though, not even just e.g. 4o pricing at launch in May vs now (50% lower) but also models getting distilled

LLMs will end up being the good human-machine interface that lets us talk to whatever AGI really looks like

(whoops expensive... will be hard pushes to make all further layers even more expensive though, capitalism will crash before this happens)

And then what?

I would put no money on the latter.

Yes because we are at AGI, bu the definition 5 years ago, goal posts are moving to ASI at this point, better than all humans.

LLMs are a key piece of understanding that token sequences can trigger actions in the real world. AGI is here. You can trivially spin up a computer using agent to self improve itself to being a competent office worker

If agents can self improve why hasn't gpt4 improved itself into gpt5 yet

Agents can trivially self improve. I'd be happy to show you - contact me at arthur@distributed.systems

Why wouldn't you hand me 35 million dollars right now if I can clearly illustrate to you that I have technology you haven't seen? Edge. Maybe you know something I don't, or maybe you just haven't seen it. While loops go hard ;)

They don't need to release their internal developments to you to show that they can scale their plan - they can show incremental improvements to benchmarks. We can instruct the AI over time to get it to be superhuman, no need for any fundamental innovations anymore

Perhaps you should pitch that to a VC?

I don't know anyone. That would be cool though, I basically have it running already.

Has it passed the Turing Test?

Keep in mind that the actual test is adversarial - a human is simultaneously chatting via text with a human and a program, knowing that one of them is not human, and trying to divine which is an artificial machine.

And the human and machine under tests are aware of that, and can play off each other.

You could ask the system for advice for how to find a VC to pitch to.

https://chatgpt.com/share/6769217c-4848-8009-9107-c2db122f08... is what advice ChatGPT has to give. I'm not sure if it's any good, but it's a few ideas you can try out.

Tokens don't need to be text either, you can move to higher level "take_action" semantics where "stream back 1 character to session#117" as every single function call. Training cheap models that can do things in the real world is going to change a huge amount of present capabilities over the next 10 years

can you share learning resources on this topic

No but if you want to join the Distributed Systems Corporation, you should email arthur@distributed.systems

> You can trivially spin up a computer using agent to self improve itself to being a competent office worker

If that was true, office workers would be being replaced at large scale and we'd know about it.

its happening right now, its just demo quality. it's being worked on now

So it's not trivial and you don't have competent AI office workers.

Sorry you're dealing with cope. Deal with it fast, things are happening

Says who? And more importantly, is this the boulder? All I (and many others here) see is that people engage others to sponsor pushing some boulder, screaming promises which aren’t even that consistent with intermediate results that come out. This particular boulder may be on a wrong mountain, and likely is.

It all feels like doubling down on astrology because good telescopes aren’t there yet. I’m pretty sure that when 5 comes out, it will show some amazing benchmarks but shit itself in the third paragraph as usual in a real task. Cause that was constant throughtout gpt evolution, in my experience.

even if it kills us

Full-on sci-fi, in reality it will get stuck around a shell error message and either run out of money to exist or corrupt the system into no connectivity.

The buzzkill when you fire up the latest most powerful model only for it to tell you that peanut is not typically found in peanut butter and jelly sandwiches.

I don't think providing accurate answers to context free questions is even something anyone is seriously working on making them do. Using them that way is just a wrong use case.

People are working -very- seriously on trying to kill hallucinations. I'm not sure how you surmised the use case here, as nothing was given other than an example of a hallucination.

There's a difference between trying to get it to accurately answer based on the input you provide (useful) and trying to get it to accurately answer based on whatever may have been in the training data (not so useful)

There's no doubt been progress on the way to AGI, but ultimately it's still a search problem, and one that will rely on human ingenuity at least until we solve it. LLMs are such a vast improvement in showing intelligent-like behavior that we've become tantalized by it. So now we're possibly focusing our search in the wrong place for the next innovation on the path to AGI. Otherwise, it's just a lack of compute, and then we just have to wait for the capacity to catch up.

A task that is completed and kills us is pretty much the opposite of a Sisyphean task.

Really the killing part was not necessary to make your point and thus injecting your Sisyphean prose.

Any technology may kill us, but we'll keep innovating as we ought to. What's your next point?

Why do we have to?

And when we get it there, it kills us.

What has AGI got to do with this?

Part of the ideas pushed into the narrative by Marketing departments / consultants / hyperscalers to movilize growth in the AI ecosystem.

Why? Nobody asked us if we want this. Nobody has a plan what to do with humanity when there is AGI

The plan is to not pay human workers. Never mind what happens to the economy or political landscape.

I am working at an AI company that is not OpenAI. We have found ways to modularize training so we can test on narrower sets before training is "completely done". That said, I am sure there are plenty of ways others are innovating to solve the long training time problem.

Perhaps the real issue is that learning takes time and that there may not be a shortcut. I'll grant you that argument's analogue was complete wank when comparing say the horse and cart to a modern car.

However, we are not comparing cars to horses but computers to a human.

I do want "AI" to work. I am not a luddite. The current efforts that I've tried are not very good. On the surface they offer a lot but very quickly the lustre comes off very quickly.

(1) How often do you find yourself arguing with someone about a "fact"? Your fact may be fiction for someone else.

(2) LLMs cannot reason

A next token guesser does not think. I wish you all the best. Rome was not burned down within a day!

I can sit down with you and discuss ideas about what constitutes truth and cobblers (rubbish/false). I have indicated via parenthesis (brackets in en_GB) another way to describe something and you will probably get that but I doubt that your programme will.

This is literally just the scaling laws, "Scaling laws predict the loss of a target machine learning model by extrapolating from easier-to-train models with fewer parameters or smaller training sets. This provides an efficient way for practitioners and researchers alike to compare pretraining decisions involving optimizers, datasets, and model architectures"

https://arxiv.org/html/2410.11840v1#:~:text=Scaling%20laws%2....

Because of mup [0] and scaling laws, you can test ideas empirically on smaller models, with some confidence they will transfer to the larger model.

[0] https://arxiv.org/abs/2203.03466

O3 is not a smaller model. It's an iterative GPT of sorts with the magic dust of reinforcement learning.

I'm pretty sure that the parent implied that o3 is smaller in comparison to gpt5

>the time it takes it to learn what works/doesn't work widens.

From the raw scaling laws we already knew that a new base model may peter out in this run or the next with some amount of uncertainty--"the intersection point is sensitive to the precise power-law parameters":

https://gwern.net/doc/ai/nn/transformer/gpt/2020-kaplan-figu...

Later graph gpt-3 got to here:

https://gwern.net/doc/ai/nn/transformer/gpt/2020-brown-figur...

https://gwern.net/scaling-hypothesis

Until you get to a point where the LLM is smart enough to look at real world data streams and prune its own training set out of it. At that point it will self improve itself to AGI.

It's like saying bacteria reproduction is way faster than humans so that's where we should be looking for the next breakthroughs.

But if the scaling law holds true, more dollars should at some point translate into AGI, which is priceless. We haven't reached the limits yet of that hypothesis.

> which is priceless

This also isn't true. It'll clearly have a price to run. Even if it's very intelligent, if the price to run it is too high it'll just be a 24/7 intelligent person that few can afford to talk to. No?

Computers will be the size of data centres, they'll be so expensive we'll queue up jobs to run on them days in advance, each taking our turn... history echoes into the future...

Yea, and those statements were true. For a time. If you want to say "AGI will be priceless some unknown time into the future" then i'd be on board lol. But to imply it'll be immediately priceless? As in no cost spent today wouldn't be immediately rewarded once AGI exists? Nonsense.

Maybe if it was _extremely_ intelligent and it's ROI would be all the drugs it would instantly discover or w/e. But lets not imply that General Intelligence requires infinitely knowing.

So at best we're talking about an AI that is likely close to human level intelligence. Which is cool, because we have 7+ billion of those things.

This isn't an argument against it. Just to say that AGI isn't "priceless" in the implementation we'd likely see out of the gate.

a) There is evidence e.g. private data deals that we are starting to hit the limitations of what data is available.

b) There is no evidence that LLMs are the roadmap to AGI.

c) Continued investment hinges on their being a large enough cohort of startups that can leverage LLMs to generate outsized returns. There is no evidence yet this is the case.

> c) Continued investment hinges on their being a large enough cohort of startups that can leverage LLMs to generate outsized returns. There is no evidence yet this is the case.

Why does it have to be startups? And why does it have to be LLMs?

Btw, we might be running out of text data. But there's lots and lots more data you can have (and generate), if you are willing to consider other modalities.

You can also get a bit further with text data by using it for multiple epochs, like we used to do in the past. (But that only really gives you at best an order of magnitude. I read some paper that the returns diminish drastically after four epochs.)

Private data is 90% garbage too

"There is no evidence that LLMs are the roadmap to AGI." - There's plenty of evidence. What do you think the last few years have been all about? Hell, GPT-4 would already have qualified as AGI about a decade ago.

>What do you think the last few years have been all about?

Next token language-based predictors with no more intelligence than brute force GIGO which parrot existing human intelligence captured as text/audio and fed in the form of input data.

4o agrees:

"What you are describing is a language model or next-token predictor that operates solely as a computational system without inherent intelligence or understanding. The phrase captures the essence of generative AI models, like GPT, which rely on statistical and probabilistic methods to predict the next piece of text based on patterns in the data they’ve been trained on"

Everything you said is parroting data you’ve trained on, two thirds of it is actual copy paste

He probably didn't need petabytes of reddit posts and millions of gpu-hours to parrot that though.

I still don't buy the "we do the same as LLMs" discourse. Of course one could hypothesize the human brain language center may have some similarities to LLMs, but the differences in resource usage and how those resources are used to train humans and LLMs are remarkable and may indicate otherwise.

Not text, he had petabytes of video, audio, and other sensory inputs. Heck, a baby sees petabytes of video before first word is spoken

And he probably cant quote Shakespeare as well ;)

>Not text, he had petabytes of video, audio, and other sensory inputs. Heck, a baby sees petabytes of video before first word is spoken

A 2-3 year old baby could speak in a rural village in 1800, having just seen its cradle (for the first month/s), and its parents' hut for some more months, and maybe parts of the village afterwards.

Hardly "petabytes of training video" to write home about.

you are funny. Clearly your expertise with babies comes from reading books about history or science, rather than ever having interacted with one…

What resolution of screen do you think you would need to not distinguish from reality? For me personally i very conservatively estimate it to be on above OOM of 10 4k screens by 10, meaning 100k screens. If a typical 2h 4k is ~50gb uncompressed, that gives us about half a petabyte per 24h (even with eyes closed). Just raw unlabeled vision data.

Probably a baby has a significantly lower resolution, but then again what is the resolution from the skin and other organs?

So yes, petabytes of data within the first days of existence - well, likely before even being born since baby can hear inside the uterus, for example.

And very high signal data, as you’ve stated yourself (nothing to write home about) mainly seeing mom and dad, as well as from a feedback loop POV - a baby never tells you it is hungry subtly.

> he had petabytes of video, audio, and other sensory inputs

He didn't parrot a video or sensory inputs though.

No, they don’t - they don’t have the hardware, yet. But they do parrot sensory output to eg muscles that induce the expected video sensory inputs in response, in a way that mimics the video input of “other people doing things”.

And yet with multiple OoM more data he still didn't cost millions of dollars to be trained nor multiple lifetimes in gpu-hours. He probably didn't even register all the petabytes passing through all his "sensors", those are some characteristics that we are not even near understanding and much less replicating.

Whatever is happening in the brain is more complex as the perf/cost ratio is stupidly better for humans for a lot of tasks in both training and inference*.

*when considering all modalities, o3 can't even do the ARC AGI in vision mode but rather just json representations. So much for omni.

>Everything you said is parroting data you’ve trained on

"Just like" an LLM, yeah sure...

Like how the brain was "just like" a hydraulic system (early industrial era), like a clockwork with gears and differentiation (mechanical engineering), "just like" an electric circuit (Edison's time), "just like" a computer CPU (21st century), and so on...

You're just assuming what you should prove

What do you think "AGI" is supposed to be?

o1 points out this is mostly about “if submarines swim”.

https://chatgpt.com/share/6768c920-4454-8000-bf73-0f86e92996...

This comment isn't false but it's very naive.

You have described something but you haven't explained why the description of the thing defines its capability. This is a tautology, or possibly a begging of the question, which takes as true the premise of something (that token based language predictors cannot be intelligent) and then uses that premise to prove an unproven point (that language models cannot achieve intelligence).

You did nothing at all to demonstrate why you cannot produce an intelligent system from a next token language based predictor.

What GPT says about this is completely irrelevant.

>You did nothing at all to demonstrate why you cannot produce an intelligent system from a next token language based predictor

Sorry, but the burden of proof is on your side...

The intelligence is in the corpus the LLM was fed with. Using statistics to pick from it and re-arrange it gives new intelligent results because the information was already produced by intelligent beings.

If somebody gives you an excerpt of a book, it doesn't mean they have the intelligence of the author - even if you have taught them a mechanical statistical method to give back a section matching a query you make.

Kids learn to speak and understand language at 3-4 years old (among tons of other concepts), and can reason by themselves in a few years with less than 1 billionth the input...

>What GPT says about this is completely irrelevant.

On the contrary, it's using its very real intelligence, about to reach singularity any time now, and this is its verdict!

Why would you say it's irrelevant? That would be as if it merely statistically parroted combinations of its training data unconnected to any reasoning (except of that the human creators of the data used to create them) or objective reality...

Let's pretend it is 1940

Person 1: rockets could be a method of putting things into Earth orbit

Person 2: rockets cannot get things into orbit because they use a chemical reaction which causes an equal and opposite force reaction to produce thrust'

Does person 1 have the burden of proof that rockets can be used to put things in orbit? Sure, but that doesn't make the reasoning used by person 2 valid to explain why person 1 is wrong.

BTW thanks for adding an entire chapter to your comment in edit so it looks like I am ignoring most of it. What I replied to was one sentence that said 'the burden of proof is on you'. Though it really doesn't make much difference because you are doing the same thing but more verbose this time.

None of the things you mentioned preclude intelligence. You are telling us again how it operates but not why that operation is restrictive in producing an intelligent output. There is no law that saws that intelligence requires anything but a large amount of data and computation. If you can show why these things are not sufficient, I am eager to read about it. A logical explanation would be great, step by step please, without making any grand unproven assumptions.

In response to the person below... again, whether or not person 1 is right or wrong does not make person 2's argument valid.

It's not like we discovered hot air ballons, and some people think we'll get to Moon and Mars with them...

> Does person 1 have the burden of proof that rockets can be used to put things in orbit? Sure, but that doesn't make the reasoning used by person 2 valid to explain why person 1 is wrong.

The reasoning by person 2 doesn't matter as much if 1 is making an ubsubstantiated claim to begin with.

>There is no law that saws that intelligence requires anything but a large amount of data and computation. If you can show why these things are not sufficient, I am eager to read about it.

Errors with very simple stuff while getting higher order stuff correct shows that this is not actual intelligence matching the level of performance exhibited, i.e. no understanding.

No person who can solve higher level math (like an LLM answering college or math olympiad questions) is confused by the kind of simple math blind spots that confuse LLMs.

A person understanding higher level math, would never (and even less so, consistently) fail a problem like:

"Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks double the number of kiwis he did on Friday, but five of them were a bit smaller than average. How many kiwis does Oliver have?"

https://arxiv.org/pdf/2410.05229

(of course with these problems exposed, they'll probably "learn" to overfit it)

> The reasoning by person 2 doesn't matter as much if 1 is making an ubsubstantiated claim to begin with.

But it doesn't make person 2's argument valid.

Everyone here is looking at the argument by person 1 and saying 'I don't agree with that, so person 2 is right!'.

That isn't how it works... person 2 has to either shut up and let person 1 be wrong in a way that is wrong, but not for the reasons they think, or they need to examine their assumptions and come up with a different reason.

No one is helped by turning critical thinking into team sports where the only thing that matters is that your side wins.

The delta-V for orbit is a precisely defined point. How you get there is not.

What is the defined point for reaching AGI?

I can check but I am pretty sure that using a different argument to try and prove something is wrong will not make another person's invalid argument correct.

Person 3: Since we can leave earths orbit, we can reach faster than light speed, look at this graph over our progress making faster rockets we will for sure reach there in a few years!

So there is a theoretical framework which can be tested against to achieve AGI and according to that framework it is either not possible or extremely unlikely because of physical laws?

Can you share that? It sounds groundbreaking!

The people who claim we'll have sentient AI soon are the ones making the extraordinary claims. Let them furnish the extraordinary evidence.

So, I think people in this thread, including me, have been talking past each other a bit. I do not claim that sentient AI will emerge. I am arguing that the person who is saying that it can't happen for a specific reason is not considering that the reason they are stating implicitly is that nothing can be greater than the sum of its parts.

Describing how an LLM operates and how it was trained does not preclude the LLM from ever being intelligent, and it almost certainly will not become intelligent, but you cannot say that it didn't for the reasons the person I am arguing with is saying, which is that intelligence can not come from something that works statistically on a large corpus of data written by people.

A thing can be more than the sum of its parts. You can take the English alphabet, which is 26 letters, and arrange those letters along with some punctuation to make an original novel. If you don't agree that means that you can get something greater than what defines it components, then you would have to agree that there are no original novels because they are composed of letters which were already defined.

So in that way, the model is not unable to think because it is composed of thoughts already written. That is not the limiting factor.

> If somebody gives you an excerpt of a book, it doesn't mean they have the intelligence of the author

A closely related rant of my own: The fictional character we humans infer from text is not the author-machine generating that text, not even if they happen to share the same name. Assuming that the author-machine is already conscious and choosing to insert itself is begging the question.

Have you ever heard of a local maxima? You don't get an attack helicopter by breeding stronger and stronger falcons.

For an industry that spun off of a research field that basically revolves around recursive descent in one form or another, there's a pretty silly amount of willful ignorance about the basic principles of how learning and progress happens.

The default assumption should be that this is a local maximum, with evidence required to demonstrate that it's not. But the hype artists want us all to take the inevitability of LLMs for granted—"See the slope? Slopes lead up! All we have to do is climb the slope and we'll get to the moon! If you can't see that you're obviously stupid or have your head in the sand!"

You’re implicitly assuming only a global maximum will lead to useful AI.

There might be many local maxima that cross the useful AI or even AGI threshold.

And we aren't even at a local maximum. There's still plenty of incremental upwards progress to be made.

I never said anything about usefulness, and it's frustrating that every time I criticize AGI hype people move the goalposts and say "but it'll still be useful!"

I use GitHub Copilot every day. We already have useful "AI". That doesn't mean that the whole thing isn't super overhyped.

So far we haven't even climbed this slope to the top yet. Why don't we start there and see if it's high enough or not first? If it's not, at the very least we can see what's on the other side, and pick the next slope to climb.

Or we can just stay here and do nothing.

No, GPT-4 would have been classified as it is today: a (good) generator of natural language. While this is a hard classical NLP task, it's a far cry from intelligence.

GPT-4 is a good generator of natural language in the same sense that Google is a good generator of ip packets.

> GPT-4 would already have qualified as AGI about a decade ago.

Did you just make that up?

A lot of people held that passing the Turing Test would indicate human-level intelligence. GPT-4 passes.

Link to GPT-4 passing the turing test? Tried googling, could not find anything.

Google must be really going downhill. DDG “gpt turing test” provides nothing but relevant links. Here’s a paper: https://arxiv.org/pdf/2405.08007

Probably asked an "AI"

The last four years?

ELIZA 2.0

I agree, these are good points.

Have we really hit the wall?

Do they use GPS based data?

Feels like there’s data all around us.

Sure they’ve hit the wall with obvious conversations and blog articles that humans produced, but data is a by product of our environment. Surely there’s more. Tons more.

We also could just measure the background noise of the universe and produce unlimited data.

But just like GPS data it isn't suited for LLMs given that you know it has no relevance what so ever to language.

Ignoring the confusion about 'GPS' for a moment: there's lots and lots of other data that could be used for training AI systems.

But, you need to go multi-modal for that; and you need to find data that's somewhat useful, not just random fluctuations like the CMB. So eg you could use YouTube videos, or even just point webcams at the real world. That might be able to give your AI a grounding in everyday physics?

There's also lots of program code you can train your AI on. Not so much the code itself, because compared to the world's total text (that we are running out of), the world's total human written code is relatively small.

But you can generate new code and make it useful for training, by also having the AI predict what happens when you (compile and) run the code. A bit like self-playing for improving AlphaGo.

You’re thinking of language in the strictest of sense.

GPS data as it relates to location names, people, cultures, path finding.

What does culture and names and people have to do with the Global Position System?

You are right that we can have lots more data, if you are willing to consider other modalities. But that's not 'GPS'. Unless you are using an idiosyncratic definition of GPS?

Reply on news.ycombinator.com