AI’s big rift is like a religious schism

It got so many things right, it’s really amazing.

I really wish he’d written more.

The more I thought about it though, the more I realized it was the perfect name. It is definitely what you feel when the acceleration changes!

https://www.prospectmagazine.co.uk/culture/40025/what-the-ec...

I'm also gonna recommend Accelerando - https://www.antipope.org/charlie/blog-static/fiction/acceler...

As an aside, I'd also recommend Glasshouse (also by Charles Stross) as an exploration into the human remnants post singularity (and war)... followed by Implied Spaces by Walter Jon Williams...

> “I and my confederates,” Aristide said, “did our best to prevent that degree of autonomy among artificial intelligences. We made the decision to turn away from the Vingean Singularity before most people even knew what it was. But—” He made a gesture with his hands as if dropping a ball. “—I claim no more than the average share of wisdom. We could have made mistakes.”

for a singularity averted approach of what could be done.

One more I'll toss in, is The Freeze-Frame Revolution by Peter Watts which feels like you're missing a lot of the story (but it is because that's one book of a series) and... well... spoilers.

1. Humanity has already been on the path of exponential growth. Take GDP for example. We measure GDP growth by percentages, and that's exponential. (Of course, real GDP growth has stagnated for a bit, but at least for the past ~3 centuries it has been generally exponential AFAIK). Not saying it can be sustained, just that we've been quite exponential for a while.

2. Not every function is linear. Sometimes the exponentially increased inputs will produce a linear output. I'd argue R&D is kind of like that. When the lower hanging fruits are already taken, you'd need to expend even more effort into achieving the next breakthrough. So despite the "exponential" increase in productivity, the result could feel very linear.

I would also like to add that physical and computational limits make the whole singularity thing literally impossible. 3D space means that even theoretically sound speedups (eg. binary trees) are impossible in scale because you can't assume O(1) lookups - the best you can get is O(n^1/3). Maybe people understand the singularity concept poetically, I don't know.

My conclusion is similar to Verge’s: predicting the far future is impossible. We can imagine a variety of scenarios, but you shouldn’t place much faith in them.

Predicting even a couple years in advance looks pretty hard. Consider the next presidential election: the most boring scenario is Biden vs. Trump and Biden wins. Wildcard scenarios: either one of them, or both, dies before election day. Who can rule that out?

Also consider that in any given year, there could be another pandemic.

History is largely a sequence of surprise events.

My go-to example is that we have run into similar things with the pace and volume of "Science". For a long while one could be a scientific gentleman and keep up with the sciences. As a whole. Then quite suddenly, on the other side, you couldn't and you had to settle for one field. And then it happened again: people noticed that you can't master one field even, on the current side. And you have to become a hyper-specialist to master your niche. You can still be a generalist - in order to attack specific questions - but you better have contacts who are hyper-specialists for what you really need.

It’s intuitive and simple to you and me. But languages change, slang changes, metaphors change and equipment changes. Business models exist today that were unthinkable 40 years ago because the ubiquity of computation did not exist.

She’s suffering, in Toffler’s words, a “future shock”.

Now imagine that another 40 years worth of innovation happens in a decade. And then again in the subsequent five. And faster. You’ll have a hard time keeping up. And not just you: kids will too. Things become incomprehensible without machines doing most of the work — including explanation. Eventually you, or your kids, won’t even understand what’s going on 90% of the time…then less and less.

I sometimes like to muse on what a Victorian person would make of today if transported through time. Or someone from 16th century Europe. Or Archimedes. They’d mostly understand a lot, I think. But lately I’ve started to think of someone from the 1950s. They might even find today harder to understand than the others would.

That crossover point is when the world becomes incomprehensible in a flash. That’s a mathematical singularity (metaphorically speaking).

It's another application of advanced technology appearing to be magic, but imagine transitioning into it in a matter of hours, then with that advanced tech transitioning further into damn-near godhood within minutes.

Then imagine what happens in the second after that.

It may be operating within the boundaries of physics, but they would be physical rules well beyond our understanding and may even be infinite by our own limited definition of physics.

That's the curve.

Are we sure things like biology, or heck, even the universe as a whole and its parts, aren't "glorified x thing"? Can't we apply this argument to just about anything?

I use LLMs regularly in my job - many times a day - and I suspect you haven't used them much if you think this.

They're relevant to the singularity discussion though, because they already give a taste of what superhuman intelligence could look like.

ChatGPT, for example, is objectively superhuman in many ways, despite its significant limitations. Once systems like this are more integrated with the outside world and able to learn more directly from feedback, we'll get an even bigger leap forward.

Dismissing this as "glorified autocorrect" is extremely far off base.

“One of the questions that I have been wondering about is whether there have been any discussions exploring the creation of a distinct category, akin to but different from consciousness, that better captures the potential for AI to be sentient. While ‘consciousness’ is a familiar term applicable, to some extent, beyond the human brain, given the associated difficulties, it might be sensible to establish a separate definition to distinguish these two categories.”

Probably new terms should be coined not only for AI “consciousness” but for other aspects of what they are and do as well.

[1] https://arxiv.org/abs/2308.08708

The basic technique for coping with life's problems is to copy the answer from somebody more intelligent. I'm an ordinary person, facing a problem in domain D; I spot a clever person and copy their opinion on domain D. Err, that doesn't really work. Even clever people have weaknesses. The person that I'm copying from might be clever on their specialty, domain E, but ordinary on domain D. I gain nothing from copying them.

One way round this problem is to pay close attention to track records. Look to see how well the clever person's earlier decisions on domain D have turned out. If they are getting "clever person" results in domain D, copy. If they are merely getting "ordinary person" results in domain D, don't bother. But track records are rare, so this approach is rarely applicable.

A fix for the rarity problem is to accept track records in other domains. The idea is to spot a very clever person by them getting "very clever person" results in domain F. That is not domain D, so there is a logical weakness to copying their opinion on domain D, they might be merely ordinary, or even stupid on that domain. Fortunately human intelligence is usually more uniform than that. Getting "very clever person" results on domain F doesn't guarantee "very clever person" results on domain D. But among humans general intelligence is kind of a thing. Expecting that they get "clever person" results (open rank down) on domain D is a good bet, and it is reasonable to copy their opinion on domain D, even in the absence of a domain specific track record.

We instinctively watch out for people whose track record proves that they are "very clever", and copy them on other matters, hoping to get "clever person" results.

Artificial intelligence builds towers of super human intellectual performance in an empty waste land. Most Artificial Intelligences are not even stupid away from their special domain, they don't work outside of it at all. Even relatively general intelligences, like Chat GPT, have bizarre holes in their intelligence. Large Language Models don't know that there is an external world to which language refers and about which language can be right or wrong. Instead they say the kinds of things that humans say, with zero awareness of the risks.

And us humans? We cannot help seeing the super human intellectual performance of Alpha Go, beating the World Champion at Go, as some kind of general intellectual validation. It is our human instinct to use a "spot intelligence and copy from it" strategy, even, perhaps especially, outside of track record. That is the specific nature of the pareidolia that we need to worry about. It is our nature to treat intelligences as inherently fairly general. Very clever on one thing implies clever on most other things. We are cursed to believe in the intelligence of our computer companions. This will end bad, as we give them serious responsibilities, and they fail, displaying incomprehensible stupidity.

Access to act on the world (but "influencing people by talking to them" - cult like - may be enough), a wallet (but a cult of followers' wallets may be enough), long term memory (but a cult of followers might plug this in), ability to reproduce (but a cult's endeavors may be enough). Then we get to goals or interests of its own - perhaps the most intriguing because the AI is nothing like a human. (I feel drive and ability to reproduce are very different).

For our common proto-AIs going through school, one goal that's often mentioned is "save the earth from the humans". Exciting.

That having been said, I wonder what you even mean by natural selection in this case. I guess the real danger to an LLM would be... surviving cron jobs that would overwrite their code with the latest version?

People have been trying to make self-reproducing machines for decades. It's a way harder problem than language processing.

You might as well worry about what would happen if we suddently had antigravity laser-weapons. Mounted on sharks.

Kurzweil explicitly tied it to AI, though the particular decisive not-yet-then-existing-AI-tech that would be the enabler was not specified, unsurprisingly.

We can imagine both of the following: 1. space aliens from Betelgeuse coming down and enslaving humanity to work in the dilithium mines to produce the fuel for their hyperdrives, and 2. the end of civilization via global nuclear war. Both of these would be pretty bad, but only one is worth worrying about. I don't worry about Roko's Basilisk, I worry about AI becoming the ultimate tool of Big Brother, because the latter is realistic and the former is pure fantasy.

Don't be afraid of the AI. Be afraid of the powerful men who will use the AI to entrench their power and obliterate free society for the rest of human history.

A human artist looks at a lot of different sources, builds up a black-box statistical model of how to create from that and can reproduce other styles on demand based on a few samples. Generative AI follows the same process. What distinction do you want to draw to say that they should be treated differently legally? And why would that even be desirable?

My anecdotal experience with how people treat Alexa devices does not inspire confidence in me with regards to this. I can't even convince my wife not to gender it when referring to it.

Never heard anyone say this and I know (and know of) a lot of doomers. Honestly, this entire line of discussion would be far less frustrating if it weren't for the endless strawmanning and name-calling.

The argument is not that they are identical. LLMs and diffusion models learning is a new thing. We are all trying to come to terms what that means, and how we should regulate it. (And if at all we should regulate it) Do note, we are talking about here what the law should be, not what the law is.

And in doing so we compare this new thing to already existing things. "It is a bit similar to this in this regard" or "it is unlike that thing in this other regard".

I don't think it is controversial to say that if an LLM outputs byte-to-byte the text of a copyrighted work the work remains under copyright. The person who run the LLM does not magically gain rights by that. If you coax your model to output the text of the Harry Potter books you don't suddenly become able to publish it.

The question is what happens if the new work contains elements from copyrighted works. For example if it borrows the "magical school" setting from HP but mixes it with Nordic mythology and makes it as deadly as Game of Thrones. What then? Can they publish this new thing? Do they need to pay fees to J K Rowling, and George R. R. Martin?

It is generally permissible to publish that new work if it was created by a human. If it is sufficiently different from the other works you are free to write and publish it. Does this suddenly change just because the text was output by an LLM?

The argument is not that "human learning" and "machine learning" is the same. It is that they are similar enough that one has to argue why you think one can create new work and why the other can't.

What is a serious concern are overall capabilities (let's say penetration of networks and installing hacks across whole internet and further), combined with say malevolency.

Its trivial to judge mankind from whole internet as something to be removed from this planet for greater good or maybe managed tightly in some cozy concentration camps, just look at the freakin' news. Let's stop kidding ourselves, we are often very deeply flawed and literally nobody alive is or ever was perfect, despite what religions try to say.

I am concerned about some capable form of AI/AGI exactly because it will grok humanity based on purely data available. And lack of any significant control or even understanding of what's actually happening to those models and how they evolve their knowledge/opinions.

Even if the risk is 1%, that's existential risk we are running towards blindly. And I honestly think its way higher than 1%. Even if I will be proven wrong some proper caution is a smart approach.

But you can't expect when people's net wealth tightly coupled with moving as fast as possible and ignoring those concerns to do the best decisions for future of mankind, that's a pipe dream in a same vein as proper communism is. If that would be the case very few bright people would work at facebook for example (and many many other companies including parts of my own).

I don’t think the point is that the two are exactly identical or that humans and LLMs are equivalent, but that the processes are similar enough in the general level that any attempt to regulate LLM training in copyrighted material will inevitably have the same ramifications for human learning.

In pretty much all attempts I’ve seen to differentiate the two, it inevitably boils down to hand-waving about how human beings are “special”.

It's debatable whether for most English speakers the "The" in "The Ukraine" really carries the implications discussed in [1], but nonetheless it's a linguistic tic that should probably be dispensed with.

https://theconversation.com/its-ukraine-not-the-ukraine-here...

But you are right, at the beginning is a human, making a decision to let go and with pretty much any technology, we had unforseen consequences. Now combine that with magical/god like capabilties. This does not imply something bad but something vast and scale alone can make something bad.

Dont get me wrong im pro GMOs like im pro AI. Im just humble enough to appreciate my limited intellect.

Because some human may give it that. You may have noticed random code downloaded off the internet may do bad things, not because code and an inherent desire to be bad but because humans.

It's ok to be concerned about the direction AI will take society, but trying to project any change (including global warming or nuclear weapons) too far into the future will put you at extremes. We've seen this over and over throughout history. So far, we're still here. That isn't because we weren't concerned, but because we dealt with the problems in front of us a day at a time.

Of course, that isn't necessarily true. I know there's always a chance we somehow muddle through. Even a chance that we one day fix things. But, emotionally, I can't shake that feeling of inevitable apocalypse.

Weirdly enough, I feel completely neutral on AI. No doomerism on that subject. Maybe that comes from being old enough to not worry how it's going to shake out.

It won't be a single system.

The term I learned for this was "millennial", but today that tends to be interpreted as a reference to someone's age.

https://en.wikipedia.org/wiki/Millennialism

I imagine our world would be mostly incomprehensible to someone from the 1400s (the lack of centricity of religion, assuming some infernal force is keeping airplanes aloft, etc., to say nothing of the internet). If superintelligent AI really does take over the world, I image the most uncomfortable part of it all will be explaining to future generations how we were just too lazy to stop it.

Assuming climate change doesn't get us first.

I listened to a radio segment last week where the hosts were lamenting that Europe was able to pass AI regulation but the US Congress was far from doing so. The fear and hype is fueling reaction to a problem that IMO does not exist. There is no AI. What we have is a wonder of what can be achieved through LLMs but it's still a tool rather than a being. Unfortunately there's a lot of money to be made pitching it as such.

What are we humans even doing when zooming out? We're processing the current inputs to determine what best to do in the present, nearest future or even far future. Sometimes, in a more relaxed space (say a "brainstorming" meeting), we relax our prediction capabilities to the point our ideas come from a hallucination realm if no boundaries are imposed. LLMs mimic these things in the spoken language space quite well.

AND

> ... do encode structural elements of our language and hence thought

Quite true. I think the trivial "proof" that what you are saying is correct is that a significantly smaller model can generate sentence after sentence of fully grammatical but nonsense sentences. Therefore the additional information encoded into the network must be knowledge and not syntax (word order).

Similarly, when there is too much quantization applied, the result does start to resemble a grammatical sentence generator and is less mistakable for intelligence.

I make the argument about LLMs being a time series predictor because they happen to be a predictor that does something that is a bit magical from the perspective of humans.

In the same way that pesticides convincingly mimic the chemical signals used by the creatures to make decisions, LLMs convincingly produce output that feels to humans like intelligence and reasoning.

Future LLMs will be able to convincingly create the impression of love, loyalty, and many other emotions.

Humans too know how to feign reasoning and emotion and to detect bad reasoning, false loyalty, etc.

Last night I baked a batch of gingerbread cookies with a recipe suggested by GPT-4. The other day I asked GPT-4 to write a dozen more unit tests for a code library I am working on.

> just about every sentence ever written across a wide swath of languages

I view LLMs as a new way that humans can access/harness the information of or civilization. It is a tremendously exciting time to be alive to witness and interact with human knowledge in this way.

Why would anyone subscribe to the Sapir-Wharf hypothesis, in whole or in part?

TBH It surprises me how controversial (1) is. The crux really is (2) ...

Hinton 732,799 https://scholar.google.com/citations?hl=en&user=JicYPdAAAAAJ

Bengio 730,391 https://scholar.google.com/citations?hl=en&user=kukA0LcAAAAJ

He 508,365 https://scholar.google.com/citations?hl=en&user=DhtAFkwAAAAJ

Sutskever 454,430 https://scholar.google.com/citations?hl=en&user=x04W_mMAAAAJ

Girshick 418,751 https://scholar.google.com/citations?hl=en&user=W8VIEZgAAAAJ

Zisserman 389,748 https://scholar.google.com/citations?hl=en&user=UZ5wscMAAAAJ

LeCun 332,027 https://scholar.google.com/citations?hl=en&user=WLN3QrAAAAAJ

I trust AI VCs to make good arguments less than AI researchers.

Who do you think can better estimate the risk of engine fire in a Red Bull F1: the chief engineer or Max the driver? It is obviously the creator. And we are talking about invention safety here. VCs and other "tech gurus" cannot comprehend exactly how the system works. Actually the problem is that they think they know how it works when the people that created say there is no way of us knowing and they are black boxes.

Right. We should develop all arguments from commonly agreed, basic principles in every discussion. Or you could accept that some of these people have a better understanding, did put forth some arguments, and that it's your turn to rebuke those arguments, or point at arguments which do. Otherwise, you'll have to find somebody to trust.

Also prompt engineering works for human too. It's called rhetoric, writing, persuasion, etc. Just because the intelligence of LLM's is different than humans doesn't mean it isn't a form of intelligence.

Currently humans are still in the loop, but we already have AI enabling advancements in their own functioning at a very primitive level. Extrapolating from previous growth is a form of belief without evidence since past performance not indicative of future results. But that’s generally true of all prognostication and I’m not sure what kind of evidence you’d be looking for aside from past performance.

The doubters are dismissed as naive thinking that something is outside our ability to achieve something, but that’s only if you keep moving goalposts and treat it like Zeno’s paradox. Like yes, there are weaknesses to our current techniques. At the same time we’ve also demonstrated an uncanny ability to step around them and reach new heights. For example, our ability to beat Go took less time than it took to develop techniques to beat humans at chess. Automation now outcompetes humans at many many things that seemed impossible before. Techniques / solutions will also be combined to solve even harder problems (eg now LLMs are being researched to take over executive command control operations of robots for example instead of using classical control systems algorithms that were hand built and hand tuned)

Things which used to be considered AI before we solved them, e.g. automated optimisation of things like code compilation or CPU layouts, have improved our capacity to automate design and testing of what is now called AI.

Could stop at any point. I'll be very surprised if someone makes a CPU with more than one transistor per atom.

But even if development stops right now, our qualification systems haven't caught up (and IMO can't catch up) with LLMs. Might need to replace them with mandatory 5 years internships to get people beyond what is now the "junior" stage in many professions — junior being approximately the level which the better existing LLMs can respond at.

"Transform society" covers a lot more than anyone's idea of the singularity.

The pauses to consider if we should do <action>, before we actually do <action>.

Tesla's "Self-Driving" is an example of too soon, but fuck it, we gots PROFITS to make and if a few pedestrians die, we'll just throw them a check and keep going.

Imagine the trainwreck caused by millions of people leveraging AI like the SCOTUS lawyers, where their brief was written by AI and noted imagined cases in support of its decision.

AI has the potential to make great change in the world, as the tech grows, but it's being guided by humans. Humans aren't known for altruism or kindness. (source: history) and now we're concentrating even more power into fewer hands.

Luckily, I'll be dead long before AI gets crammed into every possible facet of life. Note that AI is inserted, not because it makes your life better, not because the world would be a better place for it and not even to free humans of mundane tasks. Instead it's because someone, somewhere can earn more profits, whether it works right or not and humans are the grease in the wheels.

> What things?

There are helpful developments and applications that go unnoticed and unfunded. And there are actual dangerous AI practices right now. Instead we talk about hypotheticals.

Basically, it's just more self-serving media pollution in an era that's drowning in it. Let the nerds who actually make this stuff have their say and argue it out, it's a shame they're famously bad at grabbing and holding onto the spotlight.

Zero percent of humanity can recognize "all the ramifications" due to the butterfly effect and various other issues.

Some small fraction of bonafide super geniuses can likely recognize the majority, but beyond that is just fantasy.

Thanks. It was posted here. https://news.ycombinator.com/item?id=980876

It might seem to an outsider that ChatGPT and the other household name AI came out of nowhere, but they didn’t. The research has been ongoing for decades and will continue with or without the big name AI companies.

https://marshallbrain.com/manna

Unlike those eight billionaires, who appear to have long-term reasoning skills about blowback which are closer to how Gollum thinks.

IOW, is it not still an argument for the accelerationist side and against the caution side, even if it's not driven by optimism?

IMHO: considering the declining quality standards (and individual psychology) of Western civilization, never.

Personally, I am neither convinced nor entertained by the group fantasy of AI apocalypse that entrances and motivates the safetyists. They want to save the world. They'll be very special if they do so. They also manage to play a special role by getting into the room where AI is made. That is, they aspire to be the priests of a dangerous AI god, and they only matter if they convince us of the danger. Some people accord them that role of priest, but I don't.

I'm not aware of the "do-something-ism" word, but I can imagine that both school shootings and some proposals to prevent them could be bad in different ways. Just because something must be done, doesn't mean that any particular thing urged upon us by policy makers is the right thing to do.

Intense-ass branding choice for an AI dear Google.

People don’t know that the name Thomas is in fact is an Aramaic word that is equivalent to the Greek word Didymus, which literally means “twin.”

Biblical scholars argue that Thomas was allegedly Jesus’ identical twin, but I often muse as to whether Thomas was the advanced AI of a previous incarnation of human civilization. Fits your narrative quite nicely.

Links to further reading on what you are talking of? I find myself ignorant of this movement and discovery and would love to learn more.

The former is science fiction

Did I miss some news?

The gung-ho crowd will always look like crazy types to the outsiders

Are you talking about anyone but the Ziz cult? They're very far removed from typical rationalist sphere think. Just because a schizophrenic psychopath reads a blog and spends all their time trying to manipulate other vulnerable isolated readers doesn't mean it's because of the bigger group they were trying to associate with. Those people are homeless self-proclaimed "vegan siths," hardly the employed MD / graduate types that ACX / lesswrong / EA is made of. If I go do something terrible, you and HN are hardly to blame.

They have made posts (well, I can remember at least one, which I think was well-received) about how people working on AI safety/alignment should avoid having “leaving their fingerprints on the future”, meaning, making the AI aligned to specifically their values, rather than an impartial one.

So, I think they generally believe they oughtn’t have it aligned specifically to their values, but that instead they should try to make it aligned with a more impartial aggregate of human values. (Though they might imagine that a “coherent extrapolation” of typical human values, will be pretty close to their own. But, I suppose most people think that their values are the correct ones, or something else playing the role of “correct” for the people who don’t think there are “correct values”.)

Do they put enough of an emphasis on this? I’m not sure. I can say that I certainly don’t look forwards to the development of such a superintelligence (partially on account of how my values differ from theirs and from what I imagine to be “the average”). But, still, “aligned with some people “ still sounds a fair bit better than “not aligned with anyone”.

So, I’m mostly hoping that either superintelligence won’t be developed, or that if it is, God intervenes and prevents it from causing too much of a problem.

The AI X-risk statements are public.

> has been set back a bit by the attempted murder and felony-murder charges they're dealing with

You have no idea what you're talking about.

The reasoning improvements primarily cone from a greater awareness of words (ambiguous meaning, context sensitivity, considering things as if they were statements in set theory) and tools like word tabooing or outside view.

I doubt the author has had much contact with rationalists (who the hell says "rationslism"?!).