Hacker News Reader

Comment by skepticATX | Hacker News Reader

Eschatological cults are not a new phenomenon. And this is what we have with both AI safety and e/acc. They’re different ends of the same horseshoe.

Quite frankly, I think for many followers, these beliefs are filling in a gap which would have been filled with another type of religious belief, had they been born in another era. We all want to feel like we’re part of something bigger than ourselves; something world altering.

From where I stand, we are already in a sort of technological singularity - people born in the early 1900s now live in a world that has been completely transformed. And yet it’s still an intimately familiar world. Past results don’t guarantee future results, but I think it’s worth considering.

View on HN

> Eschatological cults

TIL: https://en.wikipedia.org/wiki/Eschatology

Thanks for this comment. Personally I have had trouble reconciling the arguments between academics and business people shouting about AIG from upon their ivory tower. It has felt like SO much hubris and self aggrandizing.

Candidly a vector map and rand() doesn't strike me as the path to AGI.

Agreed. Memory and increasing capacity to act in the physical world are necessary conditions.

It won't be a single system.

What about people shouting about AGI from the halls of the most advanced research labs in the field?

When has an ivory tower not involved the ‘most advanced research labs’?

Or do you mean largest implementors?

Ivory tower is usually a pejorative term meaning people who are isolated from the reality of the situation.

So it doesn’t include the most advanced research labs when those research labs are in touch with the reality of the situation, like right now with AI.

Nitpick “most advanced research labs” if you want, it’s obvious what I mean: it is clear that the “shouting” is not coming solely from people who don’t know what’s going on.

Eh, from folks isolated from the constraints and practicalities of the real world.

Which OpenAI definitely is, and has been for some time.

Or do you think infinite money and ‘do whatever you want’ is constraining?

What does constraints have to do with it? The question is who has the best grasp on the trajectory of the technology.

If not the researchers who are at the frontier of building and deploying these systems, playing with next generation iterations and planning several generations forward, then who?

Andreessen? People who like playing with LLMs? People who called the OpenAI API?

Because if you don’t have real world constraints (like needing to be profitable, or paying the bill for thermodynamic reality), then ‘anything is possible’.

Also if you aren’t dealing with those, then problems that come from that never occur to the person involved. So the concerns are abstract and not based on real limits or real problems.

Hence ivory tower.

Okay so the server bills or thermodynamic bills mean we need not worry. Open to hearing the argument: tell me how!

Not at all what I'm saying. What I'm saying is, no one knows where the line between actually economic/useful/effective and 'not worth the trouble' actually is, let alone 'could be done without boiling the oceans'.

Right now it's all hand wavey, could do everything, etc. etc.

That uncertainty is called risk! That is exactly the whole point!

What does that have to do with this thread? You know the ivory tower one?

People who worry about AI taking over the world need to worry about the world continuing to work.

Your average gas station gets gas 2x a day. The electric grid has 100's of faults that are worked around and fixed every day. Delivery still are loaded and driven by people.

Any AGI that is a threat to humanity has to be suicidal to act on it because of how the world works.

Furthermore, if you crack open an LLM and turn OFF the random noise (temperature) it becomes very deterministic. You think our accent to godhood is on the back of a vector map with some noise, I dont know what to tell you. That doesn't mean all this ML can't change the world, can't foster less BS jobs and enable creativity... but I have massive doubts that sentience and sapience are on this path.

Sure hope that intelligence doesn’t roughly mean “ability to find solutions to problems that less intelligent entities can’t discover.”

I have no idea what “our ascent to godhood” entails, and it’s hilarious that doomers get criticized for being religious about all this and then you’ll say something like that with a straight face. That’s neat that you “have doubts” sentience and sapience are on this path. You also don’t actually know.

> Any AGI that is a threat to humanity has to be suicidal to act on it because of how the world works

A "threat to humanity" need not mean "killing all humans next year". It could just mean some entity gaining permanent control. That situation would be irreversible and humanity would be at the mercy of that entity. Chances are, the entity would have emerged victorious from an evolutionary struggle for power and thus would not care about human flourishing.

That entity need not be a single AI. It could be a system like a large corporation or a country where initially a lot of "small" decisions are made by AIs. Over time, the influence of humans on this system might diminish and the trend might become irreversible.

Currently, no individual and no group, (organization, country...) on the planet has anywhere close to complete control over humanity. Further, even if some country managed to conquer everything, it could not hope to maintain that power indefinitely. An immortal system capable of complete surveillance, however, may be able to maintain power. It's a new thing, we don't know. "Sentience" doesn't matter one bit for any of this.

Such a system might take centuries to form or it might go quickly. Humans might also go extinct before something like this comes about. However, that doesn't mean people who think about such possibilities are stupid.

> That entity need not be a single AI. It could be a system like a large corporation

This is not a new problem, then. Let's tackle the issue of corporations, rather than chase an AI boogeyman that doesn't fundamentally change anything.

Look at oil companies, for example. They have humans in the loop at every level, and yet those humans do little to prevent profit incentives from leading them to destroy the planet. A broken reward function is a broken reward function, AI-assisted or not.

An organization's policies are still implemented and maintained by humans. No matter how powerful, the power is transient. You have churn, corruption, incomplete knowledge transfer, etc. AI systems in effective leadership could be able to maintain goals and accumulate power. That's what's new.

I argue this already happened. Facebook Inc (therefore Zuck personally) in 2015 effectively wielded total control of the world. Adjusting weights in ML systems there could determine who would be President of the United States.

Only when that power became clear to the common man, and to Congress, did any effort to rein it in take place. That fear is why their Stablecoin effort was completely destroyed by the US Government. Too much power concentration. It's why Jack Ma was pulled away to some black site and presumably beaten with reeds for a few months.

Effective and manifested power cannot be reined in. A power that is reined in is only as powerful as its reined in form. Most likely, there wasn't any power to start with and it's just a stupid conspiracy theory...

You can imagine yourself or your tribe or your political party to have absolute power. You can imagine Big Brother, or the Illuminati, or the Teletubbies to have power... It's not True! Today, nobody has absolute power! Let's hope we can either keep it that way or share power amongst ourselves.

> What about people shouting about AGI from the halls of the most advanced research labs in the field?

The most practiced researchers of alchemy were convinced that they could turn lead into gold. By itself, this argument is unconvincing. When it comes to the potential for fabulous wealth and/or unimaginable power, incentives distort and people are inclined to abandon their scruples.

Many people who are actually making the things closest to what we currently call “AI” are concerned.

I don’t care to engage in these nitpicky hypotheticals. We all know what I’m saying.

If the scientists building these systems aren’t credible and the commentators not building these systems aren’t credible, then who is?

If the answer is “no one,” okay, cool, that inscrutability is another reason for, not against, caution.

>> Many people who are actually making the things closest to what we currently call “AI” are concerned.

See grey goo: https://en.wikipedia.org/wiki/Gray_goo

How about nukes and atmospheric fire: https://www.insidescience.org/manhattan-project-legacy/atmos...

> scientists ... credible

The science ends at the math. Go play with an LLM at home. Run it on your GPU, see what it does, what its limits are. Realize that it's a system that never "grows", that lacking noise it becomes deterministic. Once you look behind the curtain and find out what's there, a lot of the hype to call it AI looks dumb.

I have run an LLM at home, up to 70B models. While I agree with some of what you said, I don't really know how it prevents LLMs from being useful.

Yes, LLMs currently don't learn. This is also true for humans who cannot form new memories. A human who cannot learn anything new besides the contents of their short-term memory is at a huge disadvantage, but (hopefully) no less capable if given access to a scratchpad and more thinking time to compensate.

As for the part that they are deterministic lacking noise... Why would you not provide it with noise? Generating randomness is something that modern machines are quite good at, so there isn't much of a reason not to inject it at inference time.

Ok then... if current generation is not threat, what about setting up at least some rules for whatever comes next, before it becomes unstoppable force?

I mean we are talking about future existential threats to mankind and the best response would be 'trust me bro'?

Sorry but trust in such matters is not something given freely nor easily. People are generally pretty conservative when their existence may be in stake. Not so much those whose income is tightly coupled with pushing things fast and basically ignoring those concerns. This is where SV's 'push fast and break things' hits a titanium wall of general mankind.

Or maybe folks can be bribed / attention diverted with endless stream of photorealistic cat videos, what do I know.

I’m intuitively not all that worried about LLMs in particular. I am worried about the commercial imperative to produce AI (by whatever path we may eventually find it) being at odds with the technical and sociopolitical ability to steer that AI toward good outcomes (whatever that may mean).

I have no clue whether LLMs will get us there (neither do you) and I have no clue whether we’re 6 months away or 600 years away (neither do you).

What LLMs have revealed is that very reasonable checkpoints will be blown through for commercial purposes. We are reliably producing unexpected emergent capabilities with each new scale up. We are reliably failing to get these systems to behave as their creators intend. We are continually deploying these systems to the public anyway, and we are continually connecting them to other, increasingly vital, systems.

If your argument is “we’ll never reach a superhuman or self-improving AI,” I’m open to hearing that argument. Tell me how you know. If nothing precludes us from building it, then I know that we will eventually build it due to the commercial incentives.

If your argument is “we’ll get there, but don’t worry because we’ll stop minutes before deployment and figure out all the alignment/emergent behaviors/social distribution/airgapping problems before pressing ‘deploy’,” then I ask what is your evidence? What capability have we shown to make such a decision at such a moment?

> Tell me how you know.

Congratulations, you're performing one of the major functions of intellect that an LLM can't. Learning. You aren't a deterministic system. LLM's aren't a model that is suddenly going to change all the past failures of "make it bigger and AGI will emerge from it"... At its core what an LLM is produces something interesting, but it's NOT a path to intellect... It's more or less turning language into a mathematical model. Are we so foolish, so hubris driven that something bigger will emerge from the digitized version of our language, and our vision alone, we made god in our image and how did THAT work out?

And if a super intelligent AI emerges tomorrow so what? Is it going to be able to manipulate us all based on a deep understand of people, that it gained from the piles of bad psych research? Is it going to melt down reactors and launch nukes, cause we live in a brittle world and that is suicide for your AGI (and improbable to impossible). Is it going to kill us with paperclips... cause we're so stupid we keep feeding it metal till we drown ourselves.

If AGI shows up tomorrow how many decades to a tipping point where it could "take over" where it could "cause harm"? Walk me through that process where the world is automated enough that paperclips are what kills us. That AGI is gonna be stuck in a cradle, a data center for a LONG time. It's gonna need to be fed power, hardware, water (for cooling). A can of gas and a bag of rags will stop, A few people with shovels or one person with a backhoe cuts it off from the world.

It takes 50 to 100 years for that AGI to get to a place where it does not need us at all. I would be more worried about a doomer mudering it in its cradle than it going off the rails in that time frame.

Okay, let's say it takes 100 years. Hey let's say it takes 200 years for AGI to get to a place where it doesn't need us at all.

What then?

Nuke it.

It exists in a physical (if distributed) location and requires inputs and outputs in electric cables and fibre cables.

Destroy the brain.

> It exists in a physical (if distributed) location and requires inputs and outputs in electric cables and fibre cables

"Nuke all of civilization right now!"

Those people (like most people who talk AGI) live in a world far "above the metal." They, like the rationalist bloggers, live in a "software world," far above the constraints of physical reality. This causes an understandable blind spot: the understanding that computers are, fundamentally, physical machines.

It's quite easy to imagine that you could conjure superintelligence when you are conjuring a pod of 8 quadrillion transistors in a single shell script. The illusion breaks down when you get closer to the physical reality. Reality is that those transistors are made of real materials, with real impurities, that have to be broken down and painstakingly physically, electrically, chemically, debugged, in the world below the 1's and 0's abstraction.

The commonality that both sides of this silly debate share is that neither really deals with entities at the layer below abstract software (or essays). To them, the perfect world of software is fundamental reality (they spend all of their time there!), and any crystal prism that can be constructed in software can be constructed in reality.

Of course, such prisms are instantly shattered when they come into contact with the real world: an impurity in the wafer, a static shock on the PCB before installation, a flood in the datacenter. This software stuff is fragile by default, it actually takes a tremendous amount of active work in the physical reality to conjure it into existence. I can do something that no language model can do, and is showing no signs of being able to do any time soon: drive a car to the datacenter, turn the door handle, and unplug the rack.

> turn the door handle, and unplug the rack.

There are door handles that you cannot turn, and racks you cannot unplug: ones buried deep in the Cheyenne Mountains and hidden far away in the Siberian tundra. They are protected by unfathomably powerful systems, with the support of countless people and backed by the whims global economic power.

I say this as someone who likely agrees with you. I think the power of the real world, that of companies, and governments and militaries should decrease our concern with AGI gaining power itself.

But I don’t think it’s as obvious as pointing to the fragility of software. Our human systems are fragile too, and subject to manipulation in not-so-different a way as data centers. I think you should not be so quick to discount the voices of many smart people shouting.

Human systems. They are protected by human systems. Human beings are the ones who press the big red button.

We probably do agree. I'm not saying that this tech won't be used by bad actors. I already told my elderly relatives that if they haven't seen someone in person, they shouldn't talk on the phone.

But what I'm talking about is categorically, fundamentally, not the eschaton!

Yes, they are human indeed, but those humans cannot agree on whether the plug should or shouldn't be pulled.

So your argument is that we need not worry because Sutskever and Hinton aren’t aware you can destroy data centers with water or static electricity?

The same Hinton who was certain that radiology was solved?

The same Sutskever who led OpenAI when they released a neutered GPT-2 because the full model was too dangerous?

The problem with the AI safety argument is that it can be boiled down to:

1) Assume that we build an all powerful god machine

2) The all powerful god machine has the ability to exterminate humanity

All of their work revolves around different ways that 2) might occur. But what I, and many other people, take issue with is whether 1) is even possible, and if it is, if we’re even remotely close to building it.

No one ever explains how 1) will be achieved beyond vague handwaving, because no one knows how to build a god machine.

(1) will happen if the following is true:

A. Intelligence is substrate independent I.e. there’s nothing mythically special about biological neurons that allow them to “be intelligent.”

B. Intelligence is economically valuable

C. We don’t destroy ourselves first e.g. via nukes or asteroid

We know for a fact that you can produce human-level intelligence that runs on very, very little power, and you can train that intelligence up to human-level in the course of 10-50 years of navigating the real world. Why do we know that? Because evolution did it already.

Now is there something so remarkable about the tissue we happen to have in our brain that means it is literally impossible to replicate this success? I can’t seem to find what it is.

Is there some reason to believe that the human brain is the absolute upper limit of what type of intelligence is possible in this universe? I can’t seem to find why that would be the case.

Obviously everyone who needs to agree on the commercial incentive already does agree on the commercial incentive (see: bottomless VC and soon govt funding), so that’s a moot point.

So explain how if these things are true: it’s possible, there’s immense incentives, and we have time enough to do it, that we don’t end up accomplishing it?

> I can do something that no language model can do, and is showing no signs of being able to do any time soon: drive a car to the datacenter, turn the door handle, and unplug the rack.

Steven Hawking was generally intelligent and struggled to do these things. He'd also struggle to move the goal posts for AGI as much as this.

(edit: to clarify I think AI Safety concerns are fever dreams of people who don't get out in the real world. But I don't think that reflects on what AGI looks like)

I'm not concerned that Stephen Hawking will destroy all matter in the light cone*.

Remember, if this conversation wasn't about eschatology, "AGI" would be just another synonym for "smart, useful machines." I didn't set the terms of discussion.

* or prevent humanity's technological transformation into the divine. Choose your flavor.

Nah. We'll have robots for that.

> TIL: https://en.wikipedia.org/wiki/Eschatology

The term I learned for this was "millennial", but today that tends to be interpreted as a reference to someone's age.

https://en.wikipedia.org/wiki/Millennialism

People who are concerned about global warming or nuclear weapons are also in cults?

The problem is we have stigmatized the concept of cults into more or less any belief system we disagree with. Everyone has a belief system and in my mind is a part of a kind of cult. The more anyone denies this about themself, the more cultlike (in the pejorative sense) their behavior tends to be.

The tech cults of the bay area are at least amongst the most obnoxious.

They’re nowhere near as obnoxious as evangelicals in small towns in the Bible Belt.

Sure but I don't have to live in the Bible belt.

What sort of influence do evangelicals in small towns wield compared to tech giants?

Well a woman was forced to go to the state Supreme Court this week to get a routine medical procedure, and thanks to evangelicals was denied.

Are nuclear weapons and their effects only hypothesized to exist? You could still create cults around them, for example, by claiming nuclear war is imminent or needed or some other end-of-times view.

There were people who were concerned about global annihilation from pretty much the moment the atom was first split. Those people were correct in their concerns and they were correct to act on those concerns.

In a way, they were not correct so far (in part also because of their actions, but not only due to their actions).

Pretty much exclusively due to their actions and some particularities of the specific technology which don’t seem to apply to AI.

If you put, for example, US presidents in the concerned group, i.e., actual decision makers then fair enough. But it wasn't just concerned scientists and the public.

Uhh correct. Unsurprisingly though, many of the people with the deepest insight and farthest foresight were the people closest to the science. Many more were philosophers and political theorists, or “ivory tower know-nothings.”

Maybe. There were also those scientists working actively on various issues of deterrence, including on how to prevail and fight if things were to happen - and there were quite a few different schools of thought during the cold war (the political science of deterrence was quite different from physical science of weapons, too).

But the difference to AI is that nuclear weapons were then shown to exist. If the lowest critical mass had turned out to be a trillion tons, the initial worries would have been unfounded.

None of whom were people saying "there's no risk here just upside baby!"

People were on totally opposing sides on how to deal with the risk, not dissimilar to now (with difference that the existential risk was/is actual, not hypothetical).

Sure, there are also some (allegedly credible) people opening their AI-optimist diatribes with statements of positive confidence like:

“Fortunately, I am here to bring the good news: AI will not destroy the world”

My issue is not with people who say “yes this is a serious question and we should navigate it thoughtfully.” My issue is with people who simply assert that we will get to a good outcome as an article of faith.

I just don't see the point in wasting too much effort on a hypothetical risk when there are actual risks (incl. those from AI). Granted, the hypothetical existential risk is far easier to discuss etc. than to deal with actual existential risks.

There is an endless list of hypothetical existential risks one could think of, so that is a direction to nowhere.

All risks are hypothetical

Many items on the endless list of hypothetical x-risks don’t have big picture forces acting on them in quite the same way e.g. roughly infinite economic upside by getting within a hair’s breadth of realizing the risk.

No, some risks are known to exist, other just might exist. If you walk across a busy street without looking, there is a risk of being run over - nothing hypothetical about that risk. In contrast, I might fear the force gravity suddenly disappearing but that isn't an actual risk as far as we understand our reality.

Not sure where infinite economic upside comes from, how does that work?

Being concerned and making it part of your identify are two very different things. For the latter, yes it's basically a religion.

I don't think we're dealing with "concerned" citizens in this thread, but with people who presuppose the end result with religious certainty.

It's ok to be concerned about the direction AI will take society, but trying to project any change (including global warming or nuclear weapons) too far into the future will put you at extremes. We've seen this over and over throughout history. So far, we're still here. That isn't because we weren't concerned, but because we dealt with the problems in front of us a day at a time.

The people who, from Trinity (or before), were worried about global annihilation and scrambled to build systems to prevent it were correct. The people saying “it’s just another weapon” were incorrect.

It's kind of infuriating to see people put global thermonuclear conflict or a sudden change in atmospheric conditions (something that has cause 4 of the 5 biggest mass extinctions in the entire history of the planet) on the same pedestal as a really computationally intense text generator.

It’s kind of infuriating to see people put trench warfare or mustard gas on the same pedestal as a tiny reaction that couldn’t even light a lightbulb.

There are different sets of concerns for the current crop of “really computationally intense text generators” and the overall trajectory of AI and the field’s governance track record.

My worries about AI are more about the societal impact it will have. Yes it's a fancy sentence generator, the problem is that you already have greedy bastards talking about replacing millions of people with that fancy sentence generator.

I truly think it's going to lead to a massive shift in economic equality, and not in favor of the masses but instead in favor of the psychopathic C-suite like Altman and his ilk.

I'm personally least worried about short-term unemployment resulting from AI progress. Such structural unemployment and poverty resulting from it happens when a region loses the industry that is close to the single employer there and people affected don't have the means to move elsewhere or change careers.

AI is going to replace jobs that can be done remotely from anywhere in the world. The people affected will (for the first time in history!) not mostly be the poorest and disenfranchised parts of society.

Therefore, as long as countries can maintain political power in their populations, the labor market transition will mostly be fine. The part where we "maintain political power in populations" is what worries me personally. AI enables mass surveillance and personalized propaganda. Let's see how we deal with those appearing, which will be sudden by history's standards... The printing press (30 years war, witch-hunts) and radio (Hitler, Rwandan genocide) might be slow and small innovations compared at what might be to come.

With respect to communication innovation, I think AI Hitler put it best: "we can't rewind we've gone too far."

Here's a link to the relevant historical record: https://www.youtube.com/watch?v=25GjijODWoI&t=93s

I don't think existing media channels will continue to be an effective way to disseminate information. The noise destroys the usefulness of it. I think people will stop coming to platforms for news and entertainment as they begin to distrust them.

The surveillance prospect however, is frightening.

I think people aren't thinking about these things in the aggregate enough. In the long term, this does a lot of damage to existing communication infrastructure. Productivity alone isn't necessarily a virtue.

I've recently switched to a dumb phone. Why keep an internet browsing device in my pocket if the internet's largest players are designing services that will turn a lot of its output into noise?

I don't know if I'll stick with the change, but so far I'm having fun with the experience.

The Israel/Gaza war is a large factor - I don't know what to believe when I read about it online. I can be more slow and careful about what I read and consume from my desktop, from trusted sources. I'm insulated from viral images sent hastily to me via social media, from thumbnails of twitter threads of people with no care if they're right or wrong, from texts containing links with juicy headlines that I have no hope of critically examining while briefly checking my phone in traffic.

This is all infinitely worse in a world where content can be generated by multi-modal LLMs.

I have no way to know if any of the horrific images/videos I've already seen thru the outlets I've identified were real or AI generated. I'll never know, but it's too important to leave to chance. For that reason I'm trying something new to set myself up for success. I'm still informed, but my information intake is deliberately slowed. I think that others may follow in time, in various ways.

...you do realize that a year or two into the earliest investigations into nuclear reactions what you would have measured was less energy emission than a match being lit, right?

The question is, "Can you create a chain reaction that grows?", and the answer is unclear right now with AI, but it's hard to say with any confidence that the answer is "no". Most experts five years ago would have confidently declared that passing the Turing test was decades to centuries away, if it ever happened, but it turned out to just require beefing up an architecture that was already around and spending some serious cash. I have similarly low faith that the experts today have a good sense that e.g. you can't train an LLM to do meaningful LLM research. Once that's possible, the sky is the limit, and there's really no predicting what these systems could or could not do.

It seems like a very flawed line of reasoning to compare very early days nuclear science to an AI system that has already scaled up substantially.

Regarding computing technology, I think the positive feedback you're describing happened with chip design and vlsi stuff, eg. better computers help design the next generation of chips or help lead to materials breakthroughs. I'm willing to believe LLMs have a macro effect on knowledge work in a similar way search engines, but as you said, it remains to be seen whether the models can feed back into their own development. From what I can tell, gpu speed and efficiency along with better data sets are the most important inputs for these things. Maybe synthetic data works out, who knows.

It's a perfectly fine comparison to make until it is proven that AI has no potential to continuously improve upon itself.

The people who thought Trinity was “scaled up” were also wrong.

The only reason we stopped making larger nuclear weapons is because they were way, way, way beyond useful for anything. There’s no reason to believe an upper bound exists in the physical universe (especially given how tiny and energy efficient the human brain is, we’re definitely nowhere near it) and there’s no reason to believe an upper bound exists on the usefulness of marginally more intelligence. Especially when you’re competing for resources with other nearly-as-intelligent superintelligences.

I think if it had turned out differently, and it had been possible to make an H-Bomb that produced no fallout, they would have continued to scale up.

There were lots of plans for using nukes for large-scale engineering projects.

- Make a protected harbor from a straight coastline

- Get that mountain that's in the way of your railroad flattened

- Add a freshwater lake to this plain

- Mine an entire vein at once

Reality got in the way and we found out that we don't want to be doing that.

Maybe if we get practical antimatter bombs we'll revisit those plans.

Not at all. But I think one's feelings on global warming & nukes can be influenced by previous exposure to eschatology. I was raised in American evangelicalism which puts a heavy emphasis on the end of the world stuff. I left the church behind long ago. But the heavy diet of Revelations, etc. has left me with a nihilism I can't shake. That whatever humanity does is doomed to fail.

Of course, that isn't necessarily true. I know there's always a chance we somehow muddle through. Even a chance that we one day fix things. But, emotionally, I can't shake that feeling of inevitable apocalypse.

Weirdly enough, I feel completely neutral on AI. No doomerism on that subject. Maybe that comes from being old enough to not worry how it's going to shake out.

It's obvious to me that humans will eventually go extinct. So what? That doesn't mean we should stop caring about humanity. People know they themselves are going to die and that doesn't stop them from caring about things or make them call themselves nihilists...

All I was doing in my comment was describing how an overdose of eschatology as a child can warp how one views the future. I never said I didn't care.

Singularity means more than that - an unlimited burst in information. Not just a world transformed; an infinite world of technology.

Whenever I see comments like this I wonder if anyone making them has taken a course in Thermodynamics.

Like the folks pushing the grey goo panic? In my experience, nope.

Everything’s possible until you have to actually make it work, then surprisingly the actual possibilities end up quite limited.

That said, despite micro-nuclear propulsion being a fantasy, the bomb was real.

AI is real too, but the computational capacity of the human race is not limitless. AI is expensive as hell to run, and more advanced systems might require more and more computation. We can only build so many GPUs, and the rate at which those GPUs get faster is likely to slow down over the next decade in the absence of new physics. Infinite is impossible.

I also have to wonder how far throwing a shitload of data at a statistical machine will get us, without a much stronger understanding of how these systems work and what the goals should be to achieve "AGI", whatever that means.

That depends on what you mean by AI, no?

Sure, but it seems fairly safe to assume that better models or more sophisticated training algorithms will require more compute.

I agree with you about thermodynamics. When people use words like “limitless,” it’s hard to take what they are saying seriously.

But,

> AI is expensive as hell to run, and more advanced systems might require more and more computation. We can only build so many GPUs

While certainly not using techniques like LLMs, we do know you can get some decent intelligence out of a 23 watt analog computer, the human brain.

So it is plausible that we could find algos + hardware which use 100W of power, and are quite a bit smarter than a human. Multiply that times a multi-megawatt datacenter, and the availability of some level of super intelligence might appear to be relatively “limitless.”

Unfortunately the human brain can't be powered by a 23 W windmill. We have to run an enormous infrastructure project (agriculture and logistics) that most of the world seems to be dedicated to, so people can think about making fart-joke TikToks.

Just explaining what 'singularity' means. Thanks for slanging me for no reason! Always raised the level of conversation.

My bad. I misread your comment as someone suggesting this stuff could scale forever.

Sorry for responding rudely as well. That didn't help either.

Could you explain what you mean by this? Not sure if this is even what you're talking about, but I can't imagine how a thermodynamic limit could apply here. Isn't there virtually infinite negentropy we can borrow from nearby to use in the places that matter to us most?

Reply on news.ycombinator.com