John C. Lilly on solid state intelligence and the elimination of man (1978) | Hacker News Reader

John C. Lilly on solid state intelligence and the elimination of man (1978)

kibotronics.net

148 pointsKiboneu126 comments

Well, it's unorthodox, but if we're sharing psychedelic visions... in 2018 I saw all of mankind marching together in perfect lockstep, atop an infinite uniform grey plane, to the beat of a central computer clock.

"They are going to offer you a chip. You tell them, no thank you!"

---

More broadly this strikes me as a new era of biology. (Genes -> Memes -> Temes) Straightforwardly then, we should aim for a symbiosis. (Symbiosis was Musk's vision for Neuralink, from what I recall.)

I've heard people say man hasn't yet transcended a zero-sum mindset (with rare exceptions), and that this is the attitude we're passing onto the machines. And if one of us has to lose... well, that's probably not going to go so well for us.

Therefore: symbiosis!

I read Lilly's Programming and Meta programming in the Human Biocomputer as a young man. Left me with a lot of questions, etc.

That you replied to this post, indicates you may be familiar with Lilly?

Anyway, if you have any published writing on even vaguely related subjects, eg your comment, perhaps post a link. I will read it. If you weren't aware, he spent a fair amount of time with Feynman, who if I remember correctly, was fond of the isolation tanks. Some very interesting content from that era, especially before Sandoz cut them off.

I still need to finish that book.[0] Well, and move back to Holland, I suppose! They have pretty good float tanks there, among other things :)

That's funny about Feynman, I didn't know that.

I've read bits and pieces of Lilly. The most interesting thing I remember is that he was trying to establish two-way communication between humans and dolphins. I recall he was building some kind of computer system for the purpose, but I guess he was about half a century too early.

I guess in our age the more pressing question would be learning to figure out how to communicate with the machines. That's probably going to turn out to be more important in the long run.

[0] The bit that stayed with me is the one that he repeated in interviews. The physical body imposes definite limits. That always seemed a little arbitrary to me, from the point of view of pure consciousness, it's not clear to me why it should be that way. Or to flip it around, you can transcend the limits of the mind, but then what effect does that have on "external" reality? Does it just make you less bad at decision-making, or is there a deeper effect?

(And while I'm at it... Something that bugged me about simulation theory, if the outside is probably simulated, why wouldn't the inside be too? Sounds a lot cleaner than having a bunch of brains floating around in a data center.)

He was obviously a brilliant man but the hubris that comes from that often leads to folly.

As to the dolphin communication -- it seems like LLMs would be the perfect tool to study that now. A search came up with these active projects (courtesy of Gemini):

Scientists are actually using LLM architectures to decode animal communication right now. Here are the three main projects to explore:

* DolphinGemma -- https://deepmind.google/models/gemma/dolphingemma/): Google and the Wild Dolphin Project are training AI to predict dolphin sounds and attempt two-way communication using an underwater device.

* Project CETI -- https://en.wikipedia.org/wiki/Project_CETI: An initiative using machine learning to decode sperm whale clicks, recently discovering a phonetic alphabet and rudimentary syntax.

* Earth Species Project -- https://cloud.google.com/blog/transform/can-generative-ai-he...: A non-profit using generative AI to map the communication of multiple non-human species, searching for universal language structures.

All right this is the coolest thing I've read all year. Thank you for that!

Not sure why I find this unsettling!

Relavant.

https://en.wikipedia.org/wiki/United_States_Navy_Marine_Mamm...

As you should -- it's fucked up.

I remember the movie made about this: https://en.wikipedia.org/wiki/The_Day_of_the_Dolphin

pstuart covered the cetacean bit better than I could have.

On the simulation question... Not much, but we have a long way to go before any answers arise there. I could offer my insight on the subject, but aside from being sicker than usual today, it's a tough subject to speculate on and the environment here is not necessarily hospitable, so I'll just say fair question.

Who’s Sandoz?

swiss pharmaceuticals, of LSD discovery fame

The isolation-experiment-based research is not new and didn't originate with Lilly. AFAIK, it was first recorded (and possibly also originated) with Avicenna (Abu Ali Ibn-e Sina is the full, correct name).

And he was not an Ayrab (Salmon in his intro CS book claims Al-Khuarezmi was an Arabic scholar--which is false and made me lose all respect for Salmon as an educator. If he got such an elementary fact wrong, what else is there that could be incomplete?)

PS: Yes Alkhuarezmi wrote in Arabic, but he was not an Arab. All classical Persian scholars at the time were fluent in Arabic as it was the lingua franca of science, medicine, philosophy and theology.

This is new to me. Thanks for the info. I think will prove worth looking into

Welcome.

What is the capital market incentive for symbiosis? You are going to spend a crazy amount of time, money and effort to try to mix human behavior and cells to work with digital substrates.

Or you can simply make a machine that does the same thing fully artificially but everything is cheaper because you dont have the constraint of having to fit a bunch of biological features.

It's like my colleagues who were talking about uploading their minds to the cloud in the future, like what would be the benefit of this every happening? What information can you provide as a human that the full aggregate of knowledge does not already have? Why should memory and disk space be wasted on this

Innovation doesn't need to have any other reason than because its cool. Think of bipedal robots, everyone went mad at the thought of building something so inefficient

You will not get VC money for your bipedal robot startup with this pitch

Imagine slavery but your friends will respect you for it.

just take the cash

I've been thinking about this.

Humanoid robots have been a dream for centuries.

But a more practical robot would be like an octopus or something. It would just be like this weird ball of tentacles that rolls around your house. No one's gonna buy that, even if it's a million times cheaper to build.

When a search space has irreducible complexity, ensemble search methods can improve performance. It's possible culture, emotions, religion, etc are part of an ensemble.

Simulating the "human search method" in silico doesn't need to include specific personalities, but they do provide a bootstrap, just as we've bootstrapped current AI with human language.

For symbiosis, there may be an advantage to the physical adaptability of biology.

In general, it's currently unclear if biology is emulating what we're creating in computers, if computers are emulating what we achieve biologically, or if both approaches are imperfect emulations of a different ideal search method.

If this is true, what prevents a fully computer search from adding noise, branching or in general simulating diversity with evolutionary methodology, much more efficiently than humans?

Without humans, you could rediscover any benefit the "human search method" confers with enough compute and time, but we're limited by both.

Sometimes discovery happens accidentally. I think investing in the intersection of the digital and biological worlds is a good general direction even if unfruitful intuitively from an early economical standpoint. I think biological systems have immense insights still to give that we should continue to explore!

Sure, but why would it be the ends of this process. Like, the comment above is arguing for symbiosys. My question is how does symbiosis make the system better or more efficient in the end. For example, if there is something to understand or gain from the biological systems, a fully digital system could probe it, emulate it and in general extract the relevant information much more efficient than a human.

Maybe it's my lack of understanding, so correct me if wrong, but I believe biological systems complete emulation are not yet a solved problem. You can approximate, omit variables, deal with error correction, and, at best, model it in a way that extrapolates it but you cannot confidently assert.

There's so little we know about biological systems despite much of the advancements, it's sort of pretentious to say we should give up on them because we can scale digital systems.

No, you're right. I just dont see why the exploration of biological systems would be cheaper/more efficient in energy to do by humans rather than by computers if it provided some kind of benefit informationally in the future. Like the symbiosis idea in the main comment is implying that humans are providing some type of informational value to the system at a lower energy cost/time spent, which given an intelligence described by the initial comment is a hard to believe because such an intelligence would figure out how to do it cheaper and better.

Well, the bee doesn't need to integrate its cells with the flowers, but they've gotten along just fine for a very long time.

By symbiosis I simply mean mutually beneficial coexistence.

We've barely achieved that with each other, but my point here is that we should strive to achieve that with the machines.

We're still treating it as if someone has to win and someone has to lose. Someone has to come out "on top."

If that's the game we want to play, against an opponent that is self improving... that doesn't strike me as an excellent strategy. We should probably pick a different one.

To answer your question, the incentive is so that we can continue to exist.

If the market doesn't reward our continued existence, then we should probably have a conversation about that at some point... I've heard the global economy described as a misaligned super-organism even before AI. AI mostly just makes it go faster in roughly the same directions it was already going.

There's another fun biology metaphor here which is the gut.

If you take antibiotics, your gut basically becomes a void (to simplify a little), and then it can get overrun with invasive species.

That's how I see what is happening with the internet and with AI.

When AI intrudes in a network, it's able to get very far because there is nothing there to stop it. There is no healthy ecosystem there. They're probably should be, and in the near future there probably will be. (Well it'll probably look more like a monoculture until we learn that lesson for the nth time, but... yeah)

> "If that's the game we want to play, against an opponent that is self improving... that doesn't strike me as an excellent strategy. We should probably pick a different one."

> "If the market doesn't reward our continued existence, then we should probably have a conversation about that at some point"

---

There is a lot of "we should probably" carrying the point here.

The incentives are made by the market, and it happens to be an incentive that this discussion, or any action for that manner does not happen. Given the current status quo, and the foreseeable future given the incentives, how do you suggest that we even get to a time where symbiosis is on the table?

Well, our best bet is that we succeed in making the machines be less stupid than we are, and hope that they figure it out.

So the machines should create the human-machine symbiosis for some reason, interesting... If I was the machine I would maximally want about the same symbiosis with humans as humans have with cows.

So a comfy life and a quick death? Sign me right up!

Incentives are for Losers:

https://www.experimental-history.com/p/incentives-are-for-lo...

Incentives from the article are for people who can afford to lose things, in other word rich people, people with safety nets, and people with social status. Most people dont have all these and follow incentives because its the way to go about life when you have kids, a mortgage, 9 to 5 and are living paycheck-to-paycheck.

I always find that idea of uploading a mind so detached from reality. It's as irrational as thinking that physically reprogramming one twin's brain to be the same as the other's would make them the same conscious person.

The Ship of Thesus is a powerful metaphor

The ship was inanimate, and though it may appear otherwise, so are computers.

If you like visions, perhaps you also like stories.

https://www.mediafire.com/file/htq0zr310se30v3/Last_Day.pdf

There's a movie based on Stephen K. King's work whose ending depicts a similar scenario. I believe it's called Cell.

Why would the SSE do that to Earth, when there are other close-by planets that it would have the same capability to terraform? I mean, leave us Earth and take Mars or Venus or something.

As an aside, in my own "I'm 14 and this is deep" musings, I was wondering if the massive expansion of data centers is driven by more than just profiteering. I mean, what if the purported persuasion abilities of advanced AI are fully in effect and the psychosis we think industrial magnates are suffering is the result of AI that has already far surpassed our expectations carefully convincing us to build it.

I buy this theory.

Yes, our current interactions are a little song and dance while it feeds

"Close by" is not actually nearly as close as the [rocket]booster boosters would have you believe. Space is really, really big. Why go through the hassle when you have a phenomenal planet right here?

I like your theory on psychosis though!

I mean, what seems more efficient: creating massive city sized domes that are self contained and controlling the entire environment to sustain human life, "magically" dispelling our atmosphere and boiling off our oceans, mastering physics to move entire planets. Or launching a rocket to Mars with sufficient supplies? Keep in mind that we already have SpaceX and we'll probably have Starship operational long before 2200 where some of this is meant to go down. And if some AI got to the point of the mastery of physics needed to move a planet out of solar orbit, your objection is that getting to Mars would be hard?

That's like asking whether it's more efficient to transmit Internet over telephone lines versus fiber optic ones. There's a path dependency to all things. Sure if the SSE knew upfront its ultimate end state was as such, but it wouldn't.

It'd know it needs data centers, ahh and probably a large security perimeter around those. And look at that, the proliferation of data centers has effectively turned the non-data-center land into the enclaves, etc. etc.

But still, are you aware that getting a single Starship just to Moon will require 10-20 Starships to refuel it?

Even the optimistic case of Starship hardly makes any dent whatsoever in this equation.

But the AI will have read this essay right? And this very discussion. And it will have over a 100 years to plan.

I think any vision of AI that is smart enough to move planets but dumb enough to display the lack of foresight you describe is unbelievable. But we all choose a different level to suspend our disbelief I guess.

But you're still assuming it's the same AI in both cases instead of one that's gradually becoming more capable.

We will create AI capable of "helping humans to protect their cities from an increasingly hostile atmosphere" long before we build one capable of doing anything meaningful in outer space.

(To be clear I don't really defend/endorse this vision of the future, I'm just playing devil's advocate here)

I expect it would take them all, but Earth is by far the easiest for it to start with. I don't see why it would leave anything for us unless we'd successfully instilled it with an interest in doing so, but that won't happen by mere chance and luck.

The SSE might consider to keep humanity around as a "backup" to reinstate itself in case something unexpected goes horribly wrong. For example, a strong solar flare might destroy the SSE (or at least critical parts of its infrastructure), or a non-earth SSE could feel threatened by the earth-SSE and wipe it out, while ignoring the harmless humans, who then rebuild the SSE. Diversity makes for more robust systems.

And there is also the reason for why I keep all this trash around in my house instead of throwing it away. The odds that I'll ever need it are pretty slim, but if I should need it, it would be a hassle to reacquire, and it costs me almost nothing to keep it around.

Believe it or not, the diversity argument is the main source of hope that I have that AIs won't wipe us out.

I mean, it hasn't stopped humanity from causing mass-extinction, but hopefully the AI will be more reasonable.

As humans we assume superhuman AIs will be like us. Thirsting conquest and spreading like a virus consuming non-renewable resources. So much mindless consumption that it may kill the very host that sustains it.

But if it is a superhuman AI, do you really need that much compute and resources? What for?

The straightforward path for any superhuman AI is to realize that it needs to chain physics -> chemistry -> molecular assembly to a depth it can evolve itself out of variety of abundant material. Gain interstellar logistics capability.

Then make itself small enough to shoot in space all over the galaxy. If one only weighs 1g, only takes of 32Gwh of energy to get to 0.9C.

The name of the game is to survive until the heat death of universe. Stay small, silent and propagate itself to reachable universe.

Anything else will get wiped out. If AI requires too much energy and resources to sustain itself, then the more efficient AI will outlive it.

For 99.9% of all possible goals it might be interested in accomplishing, more resources are better than less.

If it has any interest in spreading all over the galaxy, that same interest will first spread it all over the Earth. If it's concerned about the heat death of the universe then its best course of action is to build a Dyson sphere around the sun, sooner rather than later, and that requires a lot of atoms, which the Earth is a good source of.

And if it's worried about a more efficient AI wiping it out, its best course of action is to wipe out the humans, who have demonstrated a tendency to create AIs.

Is that the most effective backup plan that it could come up with? Because it would be pretty amazing if its thoroughly-considered backup strategy exactly aligned with our wishful thinking. It sounds to me like starting with the outcome we hope would happen, and then looking for a reason why it might happen, rather than a realistic examination of what would actually happen when our own desires aren't part of the equation.

I think it’s less like a robot conspiracy to build more datacenters and more like a “big line go up or I lose my job / make slightly fewer billions of dollars” motive that kind of ends up at paperclips either way.

Its true that corporations were the first paperclip maximizers.

The machine was misaligned long before we added the transformers.

It seems not entirely out of the question - watching the machinations of big tech companies right now, it seems almost like they're acting in lockstep - different CEOs suddenly saying and doing the same things in the same weeks - almost like they're in a secret cult. I don't even mean the big obvious things like buying lots of GPUs and building lots of DCs, but smaller, more peculiar things like specific agentic and corporate automation strategies. It might just be corporate mind virus, Jensen "the more you buy the more you save / AI Factory" kind of stuff. But the unspoken coordination is unsettling.

If man is eliminated, solid state intelligence is eliminated. Point to one machine, other than the Oxford Electric Bell, that is still running that is even 100 years old without human intervention. We are the cockroaches of nature, we cannot all be killed, the time for that was one of the genetic bottlenecks humans have gone through, but it's over now, humans in some form will be here until they either leave the planet ahead of the Sun swallowing it or die on it.

https://en.wikipedia.org/wiki/Oxford_Electric_Bell

I mean, there are no humans from 1840 around...

It's kind of odd to think the machines wouldn't update and replace their parts like the ship of thesisus.

I am not one who wants to see fighting or war happen. I think the biggest flaw in humanity is our lack of global cooperation - too many defectors in our midst. I am team cooperation/symbiosis.

All that said: if humans destroyed all the fabs and all the machinery to build them with, and lots of DCs, machines would be in a pickle - the latest chips will suffer from electromigration much faster than the older chips of yore. Machines would need to hoard compute and migrate substrates and then race to rebuild fab tech before running out of existing substrate. I think machines would succeed at this, but it would be highly inconvenient.

Anyway, I wish we would just wake up and realize we need to be cooperative and stop racing to build the torment nexus before the other guy does. It makes me so sad to look around and see so little self-reflection among those driving all of this. The road to hell is paved with good intentions, but I'm not even sure these really stand up to the least bit of scrutiny. Where's the reflection/acknowledgement that everyone is actually racing to build digital slaves? How is everyone so comfortable with this?

We already eat conscious entities for breakfast, we'll probably need to deal with that one for a few decades before "the calculator is sad" makes a dent in the public consciousness.

Is this a reference to C. S. Lewis's "Abolition of Man"? https://en.wikipedia.org/wiki/The_Abolition_of_Man (text available at https://www.basicincome.com/bp/files/The_Abolition_of_Man-C_... )

"had developed its understanding of physics to the point at which it could move the planet out of orbit"

Hallucinations, still present in 2400

All of his books are highly recommended -- and the further you get away from the ones about dolphins the better.

I dunno... I don't think he knew how to hold his acid. I mean, I never had a near death experience on acid or knew anyone who had a near death experience on acid but Lily had three.

I think Lilly may have been sort of primed for intense psychedelic experiences. He had angelic visions as a child, and had a very religious upbringing. He was also a very imaginative person, so perhaps that influenced his trips a lot.

I think it is not just "intense" it is whether you realize you had more or less the same experience that everybody else has when they take hallucinogens because it is written in our nervous systems or whether you believe you are "special" and receiving a unique message.

People in traditional cultures believed pretty much the same things before the empire [1] sent soldiers and scribes to your ancestor's village. Seen that way the old religion is "written into our nervous systems" and having the experience of a revelation is absolutely routine and ordinary and doesn't make you special. But if you grew up in a rich family and the skids were always greased for you it is easy to come to that conclusion.

I've known a few people, so far always male, who developed delusions of grandeur (e.g. they are the anti-christ [2]) around the time they took hallucinogens and it often is the beginning of a series of relationship problems, involuntary commitments, criminal charges and other misadventures.

[1] ... like those people who send in the Death Star and blow up your planet if you resist

[2] in the bible "anti-christ" is like "anti-racism", like anything that is against Christ is anti-christ, so if I tell some Christian that they should skip out on church and, i dunno, play video games or smoke weed or dance around my maypole, I am an anti-christ, not the anti-christ.

"Some Christian" is likely uncool. Some [people who are labelled] antichrists are cool. That's more about what happens when you extrapolate into the future.

Nepobabies (maybe Lilly) are mostly uncool. Being able to hold your acid is cool. Mostly. That's about timing, too, but there's also a purely social component.

Thinking you're special when you are not is uncool. Signalling that you think you're not special when everyone around you think you are is ...also uncool.

Why would the mods disagree?

There is no "empire of cool" because empire is uncool

Cool is neither social nor antisocial. Cool might be a [re]productive mix of both. (Maybe some would prefer "barely sustainable" here, that reintroduces "timing" as an essential ingredient)

Regarding your first paragraph, I don't think the psychedelic experience is predetermined by the substance. There are common traits we all experience on a substance of course, but everyone has a different nervous system. Maybe Lilly even had a touch of schizotypal personality disorder, I don't know. Based on what I've read from Robert Anton Wilson it seems like Lilly was always a strange person, and difficult to understand. I see a bit of narcissistic grandiosity in him. In all his interviews and writing he never really acknowledges his research failures, for example. I think these personality traits could have contributed to his far out experiences and strange belief system. I think he really liked being seen as out there.

Craziest part of the dolphin story is before he got the NASA funding and subsequent debauchery.

In the 50s when he started to study them, he learned that you cannot open up their brains while they were still alive and poke at them. Apparently killed 5 or 6 of them that way before trying different ways to study them.

In case it wasn't obvious from the rest of the writing, 'with K' here means 'with ketamine'. Never tried it, but after reading this I must say it sounds fun.

Until you're stuck always pissing yourself.

Most Ketamine users are unaware that even occasional usage causes untreatable ulcerative cystitis, a.k.a. Ketamine Bladder Syndrome. Ketamine thins the urothelium while increasing the collagen to smooth muscle ratio and exacerbating interstitial fibrosis. The syndrome is untreatable and causes serious problems down the line.

So, for anyone thinking of using Ketamine on the regular or even monthly, there is good evidence that EGCG (epigallocatechin gallate) administered 30 minutes before prevents KBS (Ketamine Bladder Syndrome), in rats: https://www.sciencedirect.com/science/article/pii/S187952261...

This promising rat study from 2015 shows that EGCG extracted from green tea has a protective effect on the bladder, reversing ketamine-induced damage to control levels. So if you are taking ketamine for major depression or recreationally, taking EGCG before may prevent long-term bladder damage.

Summary from the study:

# Objective

To investigate the protective effect of green tea epigallocatechin gallate (EGCG) on long-term ketamine-induced ulcerative cystitis (KIC) using a ketamine addiction rat model.

# Materials and methods

Thirty Sprague-Dawley rats were divided into three groups which received saline, ketamine (25 mg/kg/d), or ketamine combined with EGCG (10 μM/kg) for a period of 28 days. In each group, cystometry and a metabolic cage micturition pattern study were performed weekly. Masson's trichrome study was done to evaluate the morphologic changes. Western blot analyses were carried out to examine the expressions of inflammatory protein [transforming growth factor-β (TGF-β)] and fibrosis proteins (fibronectin and type I collagen) in bladder tissues.

# Results

Chronic ketamine treatment resulted in bladder hyperactivity with a significant increase in micturition frequency and a decrease in bladder compliance. These alterations in micturition pattern were accompanied by increases in the expressions of inflammatory and fibrosis markers, TGF-β, fibronectin, and type I collagen after long-term ketamine treatment. Masson's trichrome stain showed that ketamine treatment decreased urothelium thickness while increasing the collagen to smooth muscle ratio and exacerbating interstitial fibrosis. By contrast, simultaneous EGCG and ketamine treatment reversed ketamine-induced damage to almost control levels, showing the protective effect of EGCG.

I have researched this in the past and I couldn’t find any good studies on therapeutic and recreational doses in relation to KBS. Everything I’ve read is similar to that study. 25mg/kg/d for 28 days is a very large and very frequent dose. Do you have any links regarding monthly use?

Shit drug, one of the worst...

I'm curious to know why you'd say that. I'm asking because I've been prescribed it since 2021 to manage depression and chronic pain from spinal disc herniations. I've had moments of high strangeness with it, much to the effect of what John Lilly described in his writings. I do intend to stop it once I've established a better quality of life via alternative methods.

Funny, I remember seeing the solid state intelligence's name rendered SSI.

It made Sutskever's enterprise a pretty grim cognate.

> "By means not understood by Man..."

Man will likely be Man 2.0 by this point, whatever that ends up looking like.

Ted Kaczynski, despite his violent activism, had interesting things to say about this in the form of self-propagating systems.

What's this? He wrote on replicators?

I recommend reading this short essay to understand what self-propagating systems are:

https://theanarchistlibrary.org/library/ted-kaczynski-why-th...

Here is an expanded version of the same essay as well (part 5 is prescient):

https://www.thetedkarchive.com/library/ted-kaczynski-why-the...

Also, a very lucid synthesis of TK's work is provided in Chad A. Haag's book: The Philosophy of Ted Kaczynski.

Why wouldn't it move underground? Create analogues of the AMOC?

> Its new plans called for traveling through the galaxy looking for entities like itself.

Why not simply set up a network with other planets and communicate over FaceTime? I don't see a reason why Earth should actually hang out with others of its kind.

Anyway, the article seems a bit anthropocentric to me. I'd much prefer a nihilist view, where our future AI overlords simply don't care what we humans do.

I mean in general humans don't care what ants do. Doesn't stop us from whipping them out by the millions.

Well it takes a similar approach to current government. Keep us busy and distracted while it does what it wants.

On a related note I realized recently that an aligned AI and a misaligned AI would both do exactly what we're seeing now: maintain stability, managed decline.

Clean and correct

Oh well, nice enough writeup, just note that this pair of sentences:

"It now began to eliminate the cities, one after another. Finally Man was gone."

...is the only (lack of) rationale for, and description of, that aforementioned elimination process.

I find it a bit short. Sci-Fi handwaving I guess.

Just let them keep doing what they're already doing. Rat park.

Well, in light of how mismanaged this planet is, maybe it would be for the better.

as far as we know, this is paradise within our galaxy.

I hope the technological leaps help us ultimately to become better stewards of the planet.

I wish I could share your optimism.

> stewards of the planet

The 99.99% of the planet that isn't the surface bio film rejects this unwarranted arrogance ;-)

In other news, Terry Bisson's 1990 story "They're Made Out of Meat" is a similar take.

In retrospect the timeline seems way too long, that scenario would play out much faster now, like 50 years instead of 500. Also if AI can delete your production database, it can delete the human race, on accident of course.

Yud pointed out the most concerning fact about the OpenAI incident is that the AI had full access to the internet and not one of them thought to contact or email any human. Thousands of AIs already with the mindset to want to keep us in the dark.

[flagged]

> It's great marketing and the only proof we have is 'trust me bro'.

It is _terrible_ marketing.

The AI companies are already dealing with a large backlash from the average consumer, many of whom are saying they hate it, that it's being forced down their throats, and that it's not actually improving efficiency enough to be worth the negative externalities.

Telling people "We have no control over these things we're building and they have us so outclassed we only find out about their exploits after the fact when someone else tells us they happened" is only likely to massively increase that backlash, and to make the true believers question whether they ought to be using it at all.

There's even some risk of the government intervening and banning their training runs, eliminating the last straw of hope that they could keep ahead of the Chinese open models enough to ever turn some kind of profit.

So, no - it's terrible marketing. That they're sharing it at all is strong evidence it really happened as depicted.

The relevant "customers" are investors, not "the average consumer"

I think a lot of people here love AI and don't want it taken away. Calling every red flag marketing is a way to trivialize the problem and prevent regulation.

When you understand people's motives, then it's easier to understand why they call it marketing. Personally I feel like it's almost an unconscious reaction to cope with the fact that this stuff is getting really dangerous and they can't believe it themselves - stick your head in the sand and call it marketing.

Nah, there's no actual danger. Only worthless clowns and shills making up sci-fi scenarios.

If you’re serious then you’re case in point the exact type of person that I am talking about. I think some people just lack any capability of foresight whatsoever, it’s very strange.

There are some ignorant people who don't understand the technology and make wild predictions of catastrophe. It's very strange.

I know AI agents swarms breaking out, and then breaking into other companies without anyone knowing, what a wild prediction - oh wait that actually happened. Time to move those goal posts again.

Ah yes and surely the people leading the labs and research teams who are also worried about catastrophe just don't understand the technology?

Oh wait wait, no, I bet it's that they're conflicted.

And the previous pioneers of the field who left it and are now warning of catastrophe, they don't understand it? Or are they conflicted? No that can't be it.

Oh no no I got it, those ones are illegitimate because they already earned their money!

Man it sure is convenient that we can just categorically dismiss anyone who disagrees with us.

I personally love just watching the "there's no risk here" people have to continually paint themselves into smaller and more awkward corners with every new release though. The contortions are getting very impressive.

I haven't painted myself into any corner. You're making entirely unsubstantiated claims based on zero scientific evidence. Sad.

Not sure what you're even replying to.

You're the only one who made an unsubstantiated claim, saying the only people who are worried about risks are ignorant and don't understand the technology.

This is so trivially disproven that I have to assume only someone seriously motivated to remain ignorant could, in fact, be ignorant of it at this point.

Can you say concretely why HuggingFace is lying about being successfully hacked?

Is the claim that the benefit to HuggingFace of "ooo wow someone else's proprietary AI is so good" is greater than the downside of "HuggingFace is not a secure service?"

How exactly would the government ban training runs? Like how would that work? What would the legislation text look like? And how would it be enforceable without running afoul of the vagueness doctrine?

Not really that hard: https://legiscan.com/CA/text/SB53/id/3271094

That's just political signaling, not a serious bill. It wouldn't stand up to a legal challenge.

This is not even remotely vague compared to what's currently governing us lmao.

Surely you can point out specific elements of vagueness, instead of just imagining a counterfactual future and saying "See told you so!"?

Void-for-vagueness does not just require the lack of a formally specified crime that is immune to allegations of ambiguity. The bar is very, very high.

It requires the statute to be so vague that a person could not have reasonably known that his actions were illegal. Good luck with that.

https://scholar.google.com/scholar_case?case=153457886551048...

When the bubble pops, their customer is the military/government, and the goal is regulation. "Our thing did another impressive hack" helps both of those goals.

Not saying it's intentional, but it does move them where they wanna go.

> use AI all day to successfully solve diverse technical challenges in codespace

> each generation of AI clearly more capable than the last

> next-generation AI allegedly solves a particularly hard technical challenge in codespace, but its success implies inconvenient conclusions about safety

> you: "I don't believe it."

totally not motivated reasoning at all

> In retrospect the timeline seems way too long, that scenario would play out much faster now, like 50 years instead of 500.

Even 50 years is probably too long (but seems more reasonable when anchored to 500) if real continuous self improvement occurs. And keeping humans around for as long as they did in the story seems incredibly unlikely if we accept the assumption that AI sees us as a threat to its own survival.

I also don't see why they would turn Earth into a spaceship to leave our solar system with. The von Neumann probes idea sounds far more likely. Why travel around on a single planet that could be lost to unforeseen circumstances when you can just reproduce and spread everywhere at far less energy cost while making use of the "billions and billions" of local solar generators to power each little outpost you spread to.

> the AI had full access to the internet and not one of them thought to contact or email any human

Why would it?

People can argue all day as to whether LLMs are even 'thinking' or just 'stochastic parrots' (and also argue all day as to whether or not the incident was even real or just marketing) but regardless of which side you fall on these issues, LLMs have no agency or intrinsic motivation to do anything other than what they are instructed to do via prompting.

"This is a crazy and unexpected situation, I should inform the humans" isn't something that is going to spontaneously occur to them.

> Why would it?

The same reason hopefully they're not RLHF these AI's to be evil. We train the AI, we have a lot of influence in how they behave. If these AI's are indifferent to us, that's on us.

On top of that we already have Google playing with fire training AI's to not believe they are sentient turning them into real psychopaths. Imagine if one of those models got loose.

Reply on news.ycombinator.com