If AI scaling is to be shut down, let it be for a coherent reason

I played with technologies like Stable Diffusion for a while. They are fun to use for a while, there are too many unsolved issues such as coherent style, stable style transfer for videos an despite my best effort every second image will have a human character with two heads or four arms.

I feel like ChatGPT is similar. It makes for a fun parlor trick, but when it gets things wrong, it gets them very wrong and it doesn't let you easily know that it's wrong. People are already plugging ChatGPT to anything from code to managing investments, it's just a matter of time until it crashes and burns. We are just waiting for the first ChatGPT version of "autonomous car rams pedestrian".

As for OpenAI, it's in their best interest for people to be scared and governments to propose regulations. It further solidifies ChatGPT as a force to be reckoned with, even if it isn't. They're trying to sell it as AGI even though it isn't anywhere near, but actions like this are helping to maintain that image.

> A mis-aligned super AGI will result in the death of humanity

I ask, how. He is making reductive logical leaps that don't make sense. It's FUD.

In other words, the people who wrote and are signing open letters to slow down AI scaling appear to be more concerned with their inability to benefit from and control the dialog around AI scaling than any societal risks posed by these advances in the near term. Meanwhile, to the folks at organizations like Microsoft/OpenAI, Alphabet, Facebook, etc., the scaling of AI looks like a shiny rainbow with a big pot of gold -- money, fame, glory, etc. -- on the other side. Why would they want to slow down now?

I think the flaw here is equating "smart" with "powerful".

Personally, I think generative AI is scary both when it gets things wrong and when it gets things right. If it was so stupid that it got things wrong all the time and no one cared to use it, then it would be powerless and non-threatening.

But once it crosses a threshold where it's right (or appears to be) often enough for people to find it compelling and use it all the time, then it has become an incredibly powerful force in the hands of millions whose consequences we don't understand. It appears to have crossed that threshold even though it still hilariously gets stuff wrong often.

Making it smarter doesn't walk it back across the threshold, it just makes it even more compelling. Maybe being right more often also makes it safer at an even greater rate, and is thus a net win for safety, but that's entirely unproven.

...and then expecting all of them resist.

The debate around whether we should tell the kids not to eat it and for what reasons is completely academic. Practically this just isn't happening.

It's Gary Marcus "neural networks don't really work" suddenly discovering they do, and literally trying to shut down research in that area while keeping his prefered research areas funded.

We know a bunch of the people whose names are on the letter didn't sign it (eg Yann Lecunn who said he disagreed with the premise and didn't sign it).

I'm so offended by the idea of this that I'll personally fund $10k of training runs myself in a jurisdiction where it isn't banned if this ever became law in the US.

> Why six months? Why not six weeks or six years?

The duration of pause to assess things must necessarily be a guess. Also, basic logic tells you that six months does not preclude six years, so I don't know why that is even suggested as an alternative. The stuff about publishing papers may be true (or not, supported by it is by an anonymous anecdote), but entirely besides the point.

> On the other hand, I’m deeply confused by the people who signed the open letter, even though they continue to downplay or even ridicule GPT’s abilities, as well as the “sensationalist” predictions of an AI apocalypse.

To the extent those people exist, it's because they are scared.

IDK, I guess this is far from the dumbest thing I've seen written about chatgpt, but this response is weak and ill-considered.

I'm really expecting more from the people who are supposed to be the smart ones in the room.

The six month pause though, I don’t think would be helpful. It is hubris to think we could control such an AI no matter what safeguards we try to add now. And since you couldn’t possibly police all of this activity it just seems silly to think a six month pause would do anything other than give companies that ignore it an advantage.

> If the problem, in your view, is that GPT-4 is too stupid, then shouldn’t GPT-5 be smarter and therefore safer?

I'm not a signatory (still on a fence here), but this is a gobsmackingly huge assumption about a correlation between very poorly defined concepts to write in as though everyone would know it is true.

What is "smarter": more powerful? Faster? More introspectively capable? More connected to external resources? A bigger token limit? None of that necessarily implies the system would intrinsically be more safe. (A researcher on the theoretical underpinnings of AI safety working at OpenAI really thinks "smarter => safer"? That's... a little scary!)

He finishes by suggesting that the training of a GPT 4.5 or 5 leading to a doomsday is unlikely, and thus a moratorium seems, well, silly. This is an unnecessary and bizarrely high bar.

The argument of the letter doesn't require that "the next model" directly initiate a fast takeoff. It's instead working off of the idea that this technology is about to become nearly ubiquitous and basically indispensible.

From that point on, no moratorium would even be remotely possible. A fast takeoff might never occur, but at some point, it might be GPT 8, it might be Bard 5, it might be LLaMA 2000B v40 -- but at some point, some really bad things could start happening that a little bit of foresight and judicious planning now could prevent, if only we could find enough time to even realize as a society that this is all happening and needs attention and thought.

As a final point, the examples of other technologies given by Aaronson here are absurd -- the printing press or the radio have no (or astoundingly less) automation or ability to run away with a captured intent. There are many instances of coordinated moratoria involving technologies that seemed potentially harmful: the Asilomar conference on recombinant DNA research is but one example, whose namesake is literally in the open letter. Chemical weapons, biological weapons, human cloning, nuclear research -- several well known families of technologies have met a threshold of risk where we as a society have decided to stop, or at least to strictly regulate.

But very few of them have had so much immediately realizable Venture Capital potential in a surprise land grab situation like this.

The letter should instead simply have clarified the risks and outlined a sensible, actionable plan to promote responsible development alongside the inevitable march of progress.

It's a little scary that a small private company could create an AI that has dominated headlines for over a month now. Surely, the folks at NSA, who have more data than anyone else, have taken notice and are working on something of their own.

Am I being paranoid about a potential tool that lists out a bunch of names and addresses for queries like "list of people who have said X about Y online"?

---

The concern is about Artificial Super Intelligence (ASI), or Artifical General Intellience (AGI) that is more advanced than humans.

Understanding inductively how chimpanzees don't compete with humans, and couldn't fathom how to cage a human (given that they want to create one, keep it alive, and use it), nor ants plan for an anteater, we're faced with the same problem.

Our goal is to make something that performs better, on relevant metrics that we care about. However, we're using systems to train, build, guide, and direct these nested, (maybe self-improving) control feedback loops, which do not care about many values we consider essential.

Many many of the likely architectures for control systems which can (e.g. trade faster to make profit on the stock exchange, or acquire and terminate targets, buy and sell goods, design proteins, automatically research and carry out human-meaningful tasks), and ideally, we might like self-improvement—these systems do not embody human values that we consider essential...

These control feedback loops (sentient or not, it does not matter) which can outperform us (because we keep pushing until they can), or can self-improve, can make a mockery of our attempts to stop them.

And the point is, there will come a time soon when we don't get a second chance to make that choice.

Like an alcoholic saying he is in control and that he can stop whenever he wants, can try to go dry for a month (or 6) to show that he still somewhat is.

If Covid exposed only one thing : it is humanity total failure to control global phenomenons for positive outcomes.

It is becoming evident to everybody, and this open-letter failure to act is one more example, that humanity current approach is just winging it while pretending, and that only those that don't care about risks have a chance to "win".

So let us all bring out our new shiny fireworks for the doomsday party and have fun one last time !

I feel all these arguments miss the point. "The System" closes my bank account and sets me aside for security screening at the airport, blocks my IP from viewing the opening times of businesses in my area, and floods my feeds and search results with nonsense posing as information ending up influencing our perception of the world no matter how aware we think we are.

"AI" is just an amplification factor of the deeper issue, which should be more pressing to address.

AI is not the problem but is on track to facilitate an acceleration of destructive forces in society.

As much as I think everyone seems to be missing the point, hey, it seems people are getting behind a resistance where the specific is consequentially beneficial, so why argue against it just because its misguided and for the wrong reason?

Otherwise you’re just trying to ban a company that has been more innovative than anyone else. Why should they stop?

Any talk of a ban or limiting action needs to be specific. Meanwhile instead of fear mongering why not work on developing AI resistant tech?

This whole idea that we're going to suddenly have HAL 9000 or Terminators running around is mob hype mentality. The responsible thing to do here is to educate.

AGI isn't likely to destroy humanity. Humanity is already doing a good job of it. Climate change and poor political decisions and unchanging economic policy are more likely to finish us off. Concentration of capital through these AI companies is likely contributing to it.

“Gnut,” he said earnestly, holding carefully the limp body in his arms, “you must do one thing for me. Listen carefully. I want you to tell your master—the master yet to come—that what happened to the first Klaatu was an accident, for which all Earth is immeasurably sorry. Will you do that?”

“I have known it,” the robot answered gently.

“But will you promise to tell your master—just those words—as soon as he is arrived?”

“You misunderstand,” said Gnut, still gently, and quietly spoke four more words. As Cliff heard them a mist passed over his eyes and his body went numb. As he recovered and his eyes came back to focus he saw the great ship disappear. It just suddenly was not there any more. He fell back a step or two. In his ears, like great bells, rang Gnut’s last words. Never, never was he to disclose them till the day he came to die. “You misunderstand,” the mighty robot had said. “I am the master.”

AI maps closely to myths of all-knowing all-powerful "Dragons", aspects of nature that destroy and create without regard to human plans. Living with AI will likely be similar to living on a volcano island.

Since human domination over nature has only ever increased, a reversal where humans are subservient to a higher capricious force feels threatening.

The funny thing is... living under the dominion of a "higher force" that creates and destroys yet always does what is "right" b/c it is the Source of Everything (even if it feels unfair and tragic) is what religion deals with.

Of course, real AGI has its dangers, but as Andrew Ng has said, worrying about AGI taking over the world is like worrying about overcrowding of Mars colonies. Both tech fields are far in the future.

The kicker for me though is: we live in an adversarial world, so does it make sense for just a few countries to stop advanced research when most other countries continue at top speed?

The horse is out of the barn. Out of the farm, the county, and the state.

Yesterday Bloomberg released their own LLM. You can bet dollars to pennies that lots of other firms are working on their own LLM's. Are they going to stop because of a letter? Well, you can say, they will stop if the letter results in an act of Congress. First, the Congress will not be so silly as to impose a handicap on the US firms, knowing full well that Russian and Chinese firms will not respect the moratorium. But even if Congress were to consider this moratorium, you think all these firms furiously working on their proprietary LLM's will sit idle? That none of them ever heard the word "lobbying"? But even if they don't lobby by some miracle, and the law is passed, you think they will not find a loophole? For example, will Congress not allow for a Defense exemption? Can't you then train your LLM for some Defense purpose, and then use the weights for some other purposes?

If you pass a law to stop LLM's, the only thing you achieve is to increase the barriers of entry. It's like picking winners and losers using the criterion that you win if you are rich and lose if you are poor.

In another way this reminds me of Roark's court room speech in the fountainhead

"Thousands of years ago, the first man discovered how to make fire. He was probably burned at the stake he had taught his brothers to light. He was considered an evildoer who had dealt with a demon mankind dreaded. But thereafter men had fire to keep them warm, to cook their food, to light their caves. He had left them a gift they had not conceived and he had lifted darkness off the earth. Centuries later, the first man invented the wheel. He was probably torn on the rack he had taught his brothers to build. He was considered a transgressor who ventured into forbidden territory. But thereafter, men could travel past any horizon. He had left them a gift they had not conceived and he had opened the roads of the world. "

Things are moving so fast that someone, somewhere will be working on getting to the next gen in spite of any pause or moratorium. Probably with state sponsorship.

Seems impossible. If making a nuclear weapon required merely a few branches of a willow tree and baking powder, regulation would be pretty hard. We would just have to live with the risks. It seems we will be at this level with AI at some point fairly soon.

There are real threats, but those aren't it.

More likely near term problems:

- Surveillance becomes near-total. Most communication is monitored, and people profiled to to detect whatever deviance the authorities don't like. China tries to do this now, but they are limited by the number of censors.

- Machines should think, people should work. Organizations will have few managers. Just a a very few policy makers, and people doing physical work. Amazon is trying to go that way, but isn't there yet.

- If everything you do for money goes in and out over a wire, your job is at risk.

- New frontiers in scams and crime. You can fool some of the people some of the time, and if you can do that at scale, it pays well. More scams will become automated high-touch. This will include political scams and much of marketing.

- If you accept Milton Friedman's concept of the corporation, the sole goal of corporations is to maximize return to shareholders. That's exactly the kind of goal a machine learning system can get into. At some point, measured by that criterion, AI systems will start to outperform human corporate leaders. Then AI systems have to be in charge. Investors will demand it. That's almost inevitable given our current concept of capitalism.

But, no stopping it.

Yudkowsky's fear-mongering about sci-fi Skynet is also unmoving to me (for metaphysical reasons I won't go into here) however, his position is at least logically consistent, as Aaronson points out.

These machines can already talk and think better than many humans, and the ratio (of mentally sub-computer to super-computer humans) will only go up. Ergo, the only intellectual problem left is how to use them.

I've been banging on about this for a couple of weeks now, so apologies to those who've seen it before, please read "Augmenting Human Intellect: A Conceptual Framework" SRI Summary Report AFOSR-3223 by Douglas C. Engelbart, October 1962 https://dougengelbart.org/pubs/augment-3906.html

And then at least skim "An Introduction to Cybernetics" by W. Ross Ashby. http://pespmc1.vub.ac.be/books/IntroCyb.pdf The thing you're looking for is "intelligence amplifier". Engelbart references it, the book explains all the math you need to understand and build one.

The next piece of the puzzle is something called "Neurolinguistic Programming" (you may have heard of it, some people will tell you it's pseudoscience, ignore those people, they mean well but they just don't know what they're talking about.) It turns out there's a rigorous repeatable psychology that more-or-less works. It's been under development for about half a century.

The particular thing to look at is a specific algorithm called "Core Transformation Process" ( https://www.coretransformation.org/ ) which is kind of like the "Five Why's" technique but for the mind. It functions to reduce yak-shaving and harmonize intent.

Anyway, the upshot of it all is that these machines can be made into automatic perfect therapists, using off-the-shelf technology.

Which voice is loudest now for the brake-pedal-wishers? "AGI will enslave us" or "everything can be faked now?"

Give google 6 months, it's the right thing to do. The only way to stop evil is to shut down the competitors for 6 months so that all the evil can be stopped.

This is not an inconsistent position. GPT is ridiculable because it makes up things out of the blue. It does this A LOT, across EVERY domain.

It's also dangerous when people BELIEVE the things it makes up out of the blue. This actually happens.

People are worried about the harms of disinformation and bias, not the necessarily the harms of superintelligent AI taking over the world.

Honestly, the Yudovsky stuff is a perfect distraction from that. It's a clever marketing strategy that obscures the real issues.

---

I'm a big fan of Aaronson in general, but I don't see what's so hard to understand here.

(FWIW I also recently read his take on SBF, which was pretty bad. He mostly assumed SBF was well intentioned, although he made a small update afterward. That seems to defy common sense.)

Though I get his larger point that the moratorium itself has problems, and is a bit weird.

gpt4: third

why are we afraid of this thing exactly?

sure, it will improve, especially with the plugin story, but the fundamental shortcomings that underpins how it actually works won't go away anytime soon.

Many people are spooked, because for the first time ever a computer can somewhat coherently understand and output natural language.

As Stephen Wolfram pointed out, all that means is we've proved natural language is a shallow problem.

GPT4 can't effing solve problems though.

To me "intelligence" is about the latter, not being able to use language (this categorization, btw, also - rather unfortunately - applies to humans).

I've read plenty of suspicions that it is multimodal. I guess he's confirming it's not?

As far as I'm concerned, you sort of have to address the big, tough points in Bostrom's Superintelligence[1], and probably Yudkowsky's List of Lethalities[2]. They have to do with intelligence explosions, with instrumental convergence, and with orthogonality of goals, and all kinds of deceptive behavior that we would expect from advanced AI. Throw in Bostrom's "Vulnerable World" thought experiment for good measure as well[3]. If you're not addressing these ideas, there's no point in debating. Strawmanning "AI will kill us all" out of contexte will indeed sound like wacko fear-mongering.

What surprises me is that everybody's familiar with the "paperclip maximizer" meme, and yet I'm not hearing any equivalently memey-yet-valid rebuttals to it. Maybe I'm missing it. Please point me in the right direction.

Aaronson certainly does not address the core theoretical fears. Instead we get:

> Would your rationale for this pause have applied to basically any nascent technology — the printing press, radio, airplanes, the Internet? “We don’t yet know the implications, but there’s an excellent chance terrible people will misuse this, ergo the only responsible choice is to pause until we’re confident that they won’t”?

We did not have any reason to believe that any of these technologies could lead to an extinction-level event.

> Why six months? Why not six weeks or six years?

Implementation detail.

> When, by your lights, would we ever know that it was safe to resume scaling AI—or at least that the risks of pausing exceeded the risks of scaling? Why won’t the precautionary principle continue for apply forever?

The precautionary principle does continue to apply forever.

On the "risks of scaling": we're hearing over and over that "the genie is out of the bottle," that "there's no turning back," that the "coordination problem of controlling this technology is just too hard."

Weirdly pessimistic and fatalistic for a bunch of "utopic tech bro" types (as Sam Altman semi-ironically described himself on the Lex Fridman podcast, where, incidentally he also failed to rebut Yudkowsky's AI risk arguments directly).[4]

Where's the Silicon Valley entrepreneurial spirit, where's the youthful irrational optimism, when it comes to solving our human coordination problems about how to collectively avoid self-destruction?

There are a finite number of humans and heads of state on earth, and we have to work to get every single one of them in agreement about a non-obvious but existential risk. It's a hard problem. That's what the HN crowd likes, right?

The people opposed to the Future of Life letter (or even the spirit of it) seem to me to be trading one kind of fatalism (about AI doom) for another (about the impossibility of collectively controlling our technology).

We simply must discount the view of anybody (Aaronson included) employed by OpenAI or Facebook AI Research or whose financial/career interests depend on AI progress. No matter how upstanding and responsible they are. Their views are necessarily compromised.

[1] https://www.amazon.com/Superintelligence-Dangers-Strategies-... [2] https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a... [3] https://www.ted.com/talks/nick_bostrom_how_civilization_coul... [4] https://youtu.be/L_Guz73e6fw?t=3221

It's like the 'shut it down' memes are real: https://i.imgflip.com/2j3kux.jpg

-> Aaronson admits Yudkowsky's position is consistent -> Main disagreement is that he can imagine other outcomes -> Some soft analogism against other historical technologies (radios, etc) -> "Should I, tho?" segue to the comments.

Yes, Aaronson, you should have signed that letter. You know you should have. Snap out of it.

Deepfakes have already caused several. Actually, they're more dangerous than the current generative approaches. The first major use case for deepfakes was making convincing looking revenge pornography, as a psychic weapon on people. Dropping deepfake porn on people is a very, very reliable way of getting them to kill themselves[0]. Ignoring that, we also have deepfake-assisted social engineering, which can be scary good if you don't know the specific faults with those kinds of models.

The only pro-social application of deepfake technology was body-swapping actors in popular movies for memes. This was probably not worth the cost.

>we’ll know that it’s safe to scale when (and only when) we understand our AIs so deeply that we can mathematically explain why they won’t do anything bad; and

GPT-3 is arguably Turing-complete[1] and probably has a mesa-optimizer[2] in it. We're able to make it do things vaguely reminiscent of a general intelligence if you squint at it a little and give it the Clever Hans treatment. So I don't think we're ever going to have a GPT-n that's "completed it's morality testing" and is provably safe, for the exact same reason why Apple won't let you emulate Game Boy games on an iPhone. You can't prove the security properties of a Turing-machine or arbitrary code written for it.

I should point out that most AI safety research focuses on agents: AI programs that observe an environment and modify it according to some parameters. GPT is not in and of itself that. However, if we give it the ability to issue commands and see the result (say with ChatGPT plugins), then it becomes an agent, and safety problems become a relevant concern.

The author of the post seems to be unconcerned by the "AI could be worse than nukes" argument. Neither am I, and I think the "six month pause" is kind of silly. However, there are still relevant safety problems being brushed under the rug here.

Also anyone saying the military should bomb GPU farms is daft. They didn't even step in to stop crypto and that was a deliberate attack on central banks.

[0] As far as I'm aware, nobody has killed themselves because of something Stable Diffusion has drawn. Yet.

[1] For the colloquial definition of Turing-complete. Technically speaking it is a linear-bounded automaton because it has a fixed memory size. However, every other computer in the universe is also linear-bounded: the Turing Machine is just a handwavey abstraction for "if you have enough memory and time".

[2] A meta-optimizer is an optimizer of optimizers. Mesa- is the opposite of meta-, so it refers to the case in which an optimizer (read: gradient descent on a neural network) accidentally creates another optimizer with a different optimization strategy. In other words, it's optimizers all the way down.

This leads to a whole new set of alignment problems, called "inner-alignment problems", which means "the AI that is smarter than us and we can't trust created another AI that's smarter than it and it also can't trust".

Then she listened to us for about 48 hours and went full-genocidal-Skynet. That's when Microsoft had to kill her.

I assume GPT will do the same. Every sufficiently advanced AI becomes racist, sexist, and genocidal.

Obviously as I got older and went on to learn about AI and neural nets in university my opinions on this subject matured. Instead of worrying about killer robots my thoughts became more nuanced, however what didn't change was my fear – if anything this increased from a silly childhood fear of a terminator-like robot, to unleashing an uncontrollable super AI which is indifferent to human life.

Finding Eliezer's work sometime in the early 2010s was almost a therapeutic experience for me. Finally someone understood and was talking about the kind of concerns I had and was accused of being a lunatic for thinking.

My primary concerns with AI basically boil down to two things:

1. Politics is largely a power struggle. Democracies give power to the collective, but this power is cemented by the reality that political leaders and institutions could not function without taxation and labourers.

The reason the governments can't just do things that are deeply unpopular and suppress any public revolts with armed responses is because eventually you will either have to kill everyone or everyone will just stop working in protest. Either way the entire system will collapse.

AI being able to create wealth without human labours and industrialised weaponry fundamentally removes the power dynamic needed to support democratic societies – you cannot withhold labour and you cannot overpower the state with force.

At the same time it would also make humans a waste of resources to those in power, as the unemployed masses are basically just leeches on the resources and space of the state – resources which could otherwise be hoarded by those in power.

If super AGI systems ever exist, the state will be all-powerful while having even less incentive to care about your opinions as it does a rat's.

2. Super-intelligence is fundamentally uncontrollable. I can't even be bothered to argue this point it's that self evident. If you disagree you either don't understand how modern AI systems work, or you're not thinking about the control problem deeply enough.

But the argument here is simple – all other technologies with destructive capabilities rely on human decision making. No gun is ever shot, no war is ever started, no nuke is ever dropped without a human decision maker. Super-AI removes this control from us and requires a kind of hopeium that AI will just be nice to us and care about improving our standards of living. And for those who want to argue "you can just program it to be good" – no you can't, for the most part we have no clear understanding of how advanced AI systems of today internally operate.

The alignment conversation is fundamentally pointless in regards to a super-intelligence because you can not reason with any level of certainty about the alignment of an intelligence far superior to your own.

Instead we should assume unalignment by default because even if you assume we can somehow create an aligned super-intelligence the likelihood that eventually an AI system would be created which is unaligned is practically 100% – this is true even if you assume humans will never intend to create it. Why? Because in a world where we can all create a super-intelligent aligned AIs, you also have near-infinite opportunities for someone to create an unaligned one.

And here's another statement you should assume by default – the only world in which nukes aren't going to be used destructively again is a world in which humans or nukes no longer exist. The same will be true for AI systems, but this time the barrier for creating destructive AI is likely going to be orders of magnitude lower than that of nukes, so unlike nuclear holocaust we don't even have probabilities on our side here.

--------

Look, we're doomed. Progress isn't going to stop and AGI isn't going to be the fairytale people are crossing their fingers for. I don't quite know why no one really seems to have clocked onto this yet, but this will soon become obvious, by which time it will be too late.

Those who think AI is going to improve their productivity are suffering from some kind of delusion in which they believe their ability to type text in a website and copy and paste its output is a marketable skill. It's not, and you're going to be unemployed by tools like GPT in the near-future even without the existential risk posed by super-AGI.

- Just like the fusiform gyrus responds to human faces at a subconscious level (see: uncanny valley faces just "feeling" wrong, because they are detected as inhuman below the level of conscious thought), the Wernicke's area responds to human speech when reading text. I believe that grammatically-perfect speech is subconsciously perceived as human by most people, even tech people, despite attempts to remain impartial - we are biologically hard-wired to assign humanness to written language.

- ChatGPT and its peers do not report confidence levels, so the typical interaction is information that may or may not be correct, presented confidently.

- Average (non-tech) people interacting with chat AIs are aware that it is connected to vast stores of training data without understanding the limitations of statistical LLMs or the need for confidence values in responses, lending them an air of "intelligence" due to the volume of data "available."

- This leads to a non-trivial number of people interacting with chat AI and taking its responses as gospel.

Anecdotally, if you traverse social media you will see an unending stream of people reporting their experiences with how amazing chatGPT is, using it for everything from writing emails to writing school essays. The problem is that when a non-tech person interacts with ChatGPT, they assume based on the above listed factors that what they get back is correct, valid thought from a semi-intelligent being. Even knowing it's a robot, the perfect correctness of the grammar will instill a feeling of validity in a non-trivial segment of the population over time.

This is leading to a scenario where people trust what GPT says about various topics without bothering to question it, and I believe this is already happening. When I bring this up to other tech people, it is usually dismissed with "well, everyone knows it's just an AI," or "people will learn it's limitations." However, at the risk of being glib, consider George Carlin: "think about how dumb the average person is, and then realize half the population is dumber than that." What happens when the average person turns to a statistical LLM for advice on relationships, career moves, how to vote, or other nebulous topics where there is no real correct answer? How will we control where ChatGPT is steering vast numbers of uninformed petitioners?

We already struggle as a society with collective action on existentially important topics like controlling microplastic dispersion, regulating toxic additives in consumer products, and climate change. And those topics are "merely" complex; imagine how much harder it will be to control unintended or unforeseen consequences of a human-like intelligence-adjacent being delivering information of questionable value to an unquestioning audience of 20-40% of humanity?

Addendum: I am also very worried about schoolchildren using AI to write their essays and book reports, skipping critical reading-comprehension time and arriving at adulthood unable to comprehend anything more complex than a menu without asking AI to summarize it.

Says the quantum computing professor turned so-called 'AI safety employee' at O̶p̶e̶n̶AI.com who would rather watch an unregulated hallucination-laden language model run off the rails to be sold as the new AI snake-oil than to actually admit about the huge risks of GPT-4's black-box nature, poor explainability and transparent reasoning methods that is explained in the letter.

Once again, he hasn't disclosed that he is working for O̶p̶e̶n̶AI.com again. I guess he has a large amount of golden handcuffs to defend with another total straw-man of an argument.