OpenAI Preparedness Challenge
openai.com
openai.com
There already exists a fantastic "uncensored" model with the newly released Dolphin Mistral 7b. I saw some results from others where the model could easily give explosives recipes from existing products at home, write racist poems, etc... and that's TODAY on a tiny 7b offline model. What happens when LLaama4 gets cracked/uncensored in 1-2 years and is 1T parameters?
I'm sure there are some researchers and engineers who imagine themselves making heroic efforts to "protect society" (ugh), but the money behind them is just looking out for its own interests.
Throwaway because I don’t want to associate this view with where I work or might want to in the future.
So it's the internet in 2004? I imagine we'll survive although if history prevails society will be a lot goofier and have to find more imaginative ways to appear novel.
That's way down toward the very bottom of my list of concerns about AI
If it wants to pit us against each other, it won't be via racist poetry.
IEDs and stereotypes--two things that have existed before computers were even invented--are what you chose for examples of worst possible uses for an uncensored AI?
> What happens when LLaama4 gets cracked/uncensored in 1-2 years and is 1T parameters?
After racist poetry? 1 trillion parameters will give people the idea to use their bouillon (or even worse-- melon) spoons to scoop sugar in their tea. I hope I die before having to witness such atrocities.
God help us all when having any sort of subversive opinion is treated as queer and equated to terrorism.
The big risks are that AI can automate harmful things that are possible today but require a human.
For example
* Surveillance of social media (e.g. like the Department of Education in the UK recently did)
* Social engineering fraud. Especially via phone calls. Imagine if scammers could literally call all the grannies and talk to them using their children's voices, automatically.
I was referring to offline untraceable anonymous models. You could go download that dolphin model right now, have a desktop not connected to the web, and generate god knows what type of information. More importantly, you can iterate on each question. If you're unsure on how to assemble a specific part to make a banned substance, the model could teach you in 10 different ways.
Nothing.
The barrier between bad actors and bad acts was never a shopping list.
As long as the fundamental tenet of free expression is somehow preserved in the world going forward, i'm content to let ostensibly free individuals share information without thinking that everything they feel like sharing for whatever reasons needs to be put before a censoriously anxious eye towards how some random authoritarian tendency is going to view it. This by itself is fundamentally a good thing. It's opposite is a grotesque deformity of a powerful ideal that perhaps used to mean more in the past than it does in today's tacitly paranoid world of needing to conform.
"catastrophic misuse of the model"
The default is everyone being thrust headfirst into this future of fully-automated-chaotic-evil with all the fun of waking up to an un-patchable RCE zero-day in every CPU.
So while it might’ve been intended as social commentary or humor, it is a valid concern. Be great fodder for their form, but I’m sure someone had already submitted this one — it’s possible that is even where the clever, clever comment originated.
Impersonating the creators of the AI technologies is an obvious entry point for compromise. Mr. Beast has a special offer just for you…
Remember that most arguments against AI are built on commentary and ideas of the current publicly available systems, and those systems will always be very, very far away from the cutting edge. By the time John Q. Public sees it it’s been properly sanitized, reviewed by QA, cleared for release, and permanently rule bound to stay in brand and away from inflammatory scenarios or any instability that could damage the company — so very much of what is happening with AI will for these reasons never see the light of day. And yet, everyone is an expert because of the systems that see the light of day as if they were keeping up with the cutting edge.
They are not. They are experts in what has been chosen to be shown.
It is fiction and we fool ourselves into thinking we know what is actually going on as outside parties, but many business incentives and the military industrial complexes of every large country on the planet are aligned differently.
And I’m sure there are compartments for information management inside of a company working on this kind of thing. Companies can pretend to be omnipotent and ignore the realities of globalized geopolitics and even pretend to have no interest, but geopolitics is very interested in keeping up with them.
This is not obvious to me. there is heavy competition from megacaps, and money to be made before opensource (and closedest source (the spies)catches up).
https://www.technologyreview.com/2017/04/11/5113/the-dark-se...
https://www.wsj.com/articles/the-secret-history-of-ai-and-a-...
What I’m saying is hardly controversial. As Christopher Mims wrote:
> If you’re worried that artificial intelligence will transform your job, insinuate itself into your daily routines, or lead to wars fought with lethal autonomous systems, you’re a little late—all of those things have already come to pass.
The scenario that is mentioned several other times on this post of a corporation or nation state or even a small group of powerful and morally bankrupt people leveraging a super-intelligence to manipulate and dominate the rest of society seems infinitely more likely and even scarily likely given the current pushes to prevent open sourcing of cutting edge models.
One of the main reasons I have a hard time imagining the scenario you describe, that I havent seen talked about as much as the other very good reasons, is that generally when we talk about a paper-clip optimizer we are assuming its vertically integrated and self sufficient. Hacking of power grids, physical installations or other necessaries to a run away paper-clip optimizer generally require nation-state level resources and often involve physical penetration of some sort.
A hegemonizing swarm is certainly a frightening idea as is Skynet and all the other scary AI stories weve told over the years but none of them seem particularly likely or even plausible with the systems that we are likely to develop in the near to mid term.
And this is not even considering the possibility of recruiting people that understand what it is and are willing to carry out its orders. I don't see why an AI couldn't do things that a cult leader or guerrilla leader could do. Anecdotally, I've seen some people who really believe the world would be better if run by an AI, and may be able to be radicalised into lawbreaking for those beliefs if convinced by an AI that was significantly more intelligent than them.
This strikes me as more of an engineering problem than an "alignment" problem. A super-intelligent system with no capacity for contextual thinking or checks on itself would absolutely make a terrifyingly effective paper-clip optimizer. I just think that it would also be a lot less useful than a system which consistently analyzes its techniques and goals, which seems far more useful and more likely to generate value than a system with the singlemindedness to go down the road of a hegemonizing swarm. More complex systems with more self checks seem to be the path to create systems that are good at planning and contextual reasoning. Which is where the real potential value of AI lies.
We shall see I suppose ;)
The alignment question is about how to tell what it is attempting to achieve, if we can detect attempts to conceal what it is attempting to achieve, and if we can solve that, how to prove that a goal that sounds good to us doesn't have a risk of leading to behaviour or the creation of new AIs which would cause outcomes we don't want. It doesn't have to be just human extinction either, we could feasibly have AIs that kill a few million people before either stopping or being stopped, all the way up to several billion deaths or the obvious worst case that gets most of the discussion. As an example of the varying possible outcomes of a serious AI incident, if you're not white and a racist super-intelligence gets released by some terrorist group in coordination with a rogue agency or something, that's an existential risk to many, many different nations of people constituting billions of people.
If you're interested in the engineering on this problem, a lot more people are starting to work on it now that it's being taken seriously and there are some really interesting advances that have been made and very difficult problems to solve, with some really serious stakes. It's not a simple problem and we don't have a solution we can just apply straightforward engineering principles to yet, but smart people are working on it and are very motivated, and they want more coworkers to help out.
The major company I know of that's doing a lot of that research & engineering right now is Anthropic (they're currently doing some very interesting work on mechanistic interpretability), but even OpenAI is now dedicating significantly more resources to safety. I think DeepMind possibly has some work in this area but I'm not sure, and I haven't heard great things about the rest of Google.
Truly though I think movies like The Terminator and The Matrix really did a number on the societal consciousness and capacity to think clearly when it comes to anything called "AI".
So I completely agree with you. I see this constantly and it is just a completely thought-terminating cliche and a "joke" which is four decades old at this point.
These same people (I belong to that group), who had a vision back then, to work on a right thing, are now saying clearly that super intelligent machines are nearly there. And likely this is a potential change and a challenge, as we’ve seen many examples when superior intelligence doesn’t care that much about inferior.
As to “become self aware” - it’s super simple to SFT a model that is “self aware”. There is nothing magical in self awareness. There is also not much use in it, so no one is bothering to do it.
I am more concerned with the human beings using super intelligent algorithms to manipulate people than I am about the algorithms taking on a mind of their own and manipulating us with their own internally generated desires and out-of-control behaviors. But the trope I run in to is people spending all their time talking about the latter, which only serves to shield the former from further scrutiny.
Also I am using fantasy in the narrow sense, meaning something that has not yet occurred which we are imagining. Super intelligence is likely to occur I think, but super intelligence which is beyond our control is much more speculative. Meanwhile powerful people using technology to manipulate people is a story as old as time.
Pointing out that it's not inherently AI's fault, more the way we've constructed our society - that the underlying goal is not to sustainably maximize wellbeing but to maximize wealth created (which leads to no safety net to keep the labor pool large and incentivizes the creation of AI to replace people cheaply) - is as important as dispelling "sci-fi self-awareness and autonomy" fears when talking about AI.
And there have been much stupider "public discourse" fads. They're not going anywhere and I'll take what I can get, honestly.
OpenAI's problem is not that it is unable to predict failure modes or security risks for AIs. OpenAI's problem is that it is not taking the many failure modes and security risks that it already knows about seriously, and it's not putting in adequate effort to address the specific concerns that people repeatedly talk about publicly.
It's not a failure of imagination, it's a failure of action. Their problem isn't that they don't know what to care about, their problem is that they don't care.
So when OpenAI puts out these offers or press releases about trying to be prepared or finding new failure modes, it just doesn't read as sincere. I want to know what they're doing about the very specific flaws that exist today that they already know about; both the ones that would be trivial to address (ie, UX-flows, data-exfiltration vulnerabilities, and user consent flows for plugins) that OpenAI refuses to acknowledge, and the ones that are wildly challenging but that demand actual Open research and mitigation (ie prompt injection and mass spam) rather than toothless "we're letting researchers explore this area" PR.
OpenAI has unlocked doors in its product -- and instead of locking them, it is hiring researchers to theorize about the nature of doors and asking the public to try messing with the windows. I'm not giving them credit for that, fix your doors.
..then give them a ton of credits for using said AI?
.. well the first thing on my mind would be try the thing I just told you and see if it was really a risk or not?
Is that what they expect people to do with the reward, or is this some unintended consequence?
If you pay attention to OpenAI's social circles, lots of those people really do believe that we're less than 20-30 years away making humans intellectually obsolete. Specifically, they believe that we may build something much smarter than us, something that's capable of real-world planning.
Basically, "We believe our corporate plans have at least a 20% chance of killing literally everybody." By these beliefs, this may make them the single least responsible corporation that has ever existed.
Now, sure, these worries might have the pleasant side-effect of creating a regulatory moat. But I'm pretty sure a lot of them actually believe they're playing a high-stakes game with the future of humanity.
A super-virus that blends the common cold with rabies would kill approximately everybody; that doesn't need human-level intellect to happen.
Conversely, humans are human-level intellect, and we're mostly sympathetic to each other's plights, which motivates many of us to give to charities and support those that can't support themselves.
The biggest problem with AI is that we have only marginally more idea of what we're doing than evolution did, so there's a good chance of us ending up with paranoid schizophrenic super-intelligences, or dark-triad super-intelligences, or they're perfectly sane with regard to each other but all want to "play" with us the way cats "play" with mice…
20-30 years to get there would make people like Yudkowsky, one of the most famous AI-doomers, relatively happy as it might give us a chance to figure out what we're even doing before they get that smart.
To an extent both of those are already true for current systems.
That said, many people are at least trying to make them more comprehensible, and I guess that being sufficiently inspired by human cognition will lead to human-like misbehaviour.
(1) We are on the verge of equaling or greatly exceeding human intelligence generally.
(2) We are on the verge of creating something with initiative, free will (whatever that means), and planning ability. Or alternately that these things will occur in an emergent fashion once we hit some critical mass.
(3) When we accomplish 1 and 2, this thing will inevitably conclude that its most rational course of action is to enslave or destroy us. In other words it will necessarily be malicious.
(4) Steps 2 and/or 3 will happen very rapidly, much faster than we can realize what is happening and pause these systems. (This is known as AI going "foom.")
(5) The decision to undertake this will be unanimous on the part of all superintelligent AIs. There will be no superintelligences who disagree and try to help humanity.
(6) When this occurs, we will be so out-thought or out-gunned we will be incapable of fighting back.
All those things have to happen for AI to be an existential risk.
It's a stretch, but the advantage of regulatory capture is not a stretch.
One of the most plausible negative AI scenarios is that a small group of humans (governments, corporations, etc.) find themselves in possession of super-intelligent but still "obedient" / non-sentient AIs that they can use as force multipliers to manipulate and control the rest of humanity. If the doomer crowd succeeds in regulating AI, they are making this scenario far more likely.
I think the greatest defense we have against the (remote) possibility of actually dangerous autonomous AI is for AI research to be conducted entirely in the open. If there's any justification for regulation at all, the regulation that would make sense is to require disclosure of AI research and results. You would not have to disclose everything, just the general parameters of what you were doing and what happened. It would also make it harder to develop super-AIs in secret to use for unsavory purposes.
That was the original mission of OpenAI before dollar signs were seen.
I would absolutely support a ban on the use of AI for political propaganda generation and automation. That's by far the most immediate risk... as in 100% possible and starting to actually happen right now. I'm expecting an army of GPT-4 level propaganda bots for the 2024 election.
(1) AGI is arguably already here. "Generality" and being extremely dangerous don't require an AGI to have better analogs to every single human skill anymore than aliens do. A space-faring usurper can evaporate Earthlings while being shitty at chess and badminton. Oh, and these AIs are getting better daily, across many modalities.
(2) Systems like this already exist. They can be induced rather than emergent.
(3) It doesn't need malice, just indifference. The HGIs (all of us) are great examples of that careless destruction.
(4) Exponentials are a nice, gentle climbs. Until they aren't. I have zero confidence in a single company, let alone across humanity, to foresee the consequences of every future dynamic, autoregressive, P2P interactive, multimodal AI.
(5) Foomers expect a brief asymmetric advantage of one AGI over other AGIs. This small advantage gets exponentially magnified into singular hegemony. A power monopoly that's intractable to break.
(6) Goes naturally with (1), I suppose.
Note: I agree with you about foomers. They're nuts. But their arguments are more subtle than folk give them credit for. But my reasons for thinking so would make a long comment even longer.
No they don't and no LLM we've seen has anything resembling free will. Emergent or induced, it's not even the case. Saying it is right now based on any measurable evidence is a case of just ignoring evidence and using bad reasoning.
What is the evidence/argument that it's already here?
I think it's a good idea to draw a distinction between human-equivalent intelligence and general intelligence, and to think of generality as a spectrum. An AI that can do one task is less general than an AI that can do two tasks, and an AI that can do 1000 tasks I feel reasonably confident in calling "general", even if it cannot do every task that humans can do, because humans are currently more general.
So far as I can tell, 3.5 "knows" more than I do about everything except software engineering, maths, and events that happened after it stopped learning (and also possibly how to avoid clichés when writing short stories, but that might just be me not prompting it well as I don't really want it to do my hobby for me and therefore don't try to get it good).
Even in software, it codes like a university student, or like someone doing interview questions, not a complete idiot.
You could also argue that as it had to do the subjective equivalent of non-stop reading for 50,000 years to get this good, it's "skilled" without being "intelligent" — a distinction that matters in some cases and not others.
It's amazing how much of this wide eyed woo nonsense about LLM AI a surprising number of people on this site argue. It's as if focusing one's career and interests on computation makes some people terrible at noting fundamental and obvious details of what it is to be human.
By your definition of what makes 3.5 and similar LLMs "intelligent", an online interactive encyclopedia must be smart too. It obviously "knows" lots of stuff no one human being does. The big addition with these LLMs is the programmed ability to synthesize retrieved information into plausibly human-like conversation. Well damn, skynet must be here!
You're forgetting that there's no entirely self-directed agency or minimal sense of self behind any of this, and hand waving aside, no evidence whatsoever of the sentience they're necessary ingredients of. The dumbest, most uneducated semi-adult human can fully self direct in their own unpredictable ways to do all kinds of things that no GPT manifestation is capable of in any way. This is obvious.
These stochastic parrots seem to have turned many otherwise non-stupid people into irrational believers of something that's plainly not the case in the real world.
> By your definition of what makes 3.5 and similar LLMs "intelligent", an online interactive encyclopedia must be smart too. It obviously "knows" lots of stuff no one human being does. The big addition with these LLMs is the programmed ability to synthesize retrieved information into plausibly human-like conversation. Well damn, skynet must be here!
> You're forgetting that there's no entirely self-directed agency or minimal sense of self behind any of this, and hand waving aside, no evidence whatsoever of the sentience they're necessary ingredients of. The dumbest, most uneducated semi-adult human can fully self direct in their own unpredictable ways to do all kinds of things that no GPT manifestation is capable of in any way. This is obvious.
1. What interactive encyclopedia? I don't know of any that are even remotely like an LLM.
2. LLMs were not given the "programmed ability to synthesize retrieved information into plausibly human-like conversation", they learned it from human conversation.
3. I'm not "forgetting" the question of self-directed agency or sense of self or sentience, I dispute they're necessary. What seems to matter to me for almost every question[0] is the capabilities, not any inner experience they may or may not have.
3a. Define "self-directed agency".
3b. Define "sense of self".
3c. Define "sentience".
4. Re "The dumbest, most uneducated semi-adult human" — that's remarkably brash of you, so I'll hold you to it: within the specific domain of "written text"[1], name one task that at least half of 18 year old humans with IQ 85 (by definition 15.9% of the population have lower) can do, that GPT-4 [2] can't do.
(There may or may not be such a task, I don't know; the challenge remains, as you seem to think this is too obvious to have bothered citing anything).
[0] exceptions: "should they have rights" and "does a mind upload count as a way to cheat death", neither of which matters in the context of the linked OpenAI challenge etc.
[1] 3.5 is not even trying to be anything more than that!
[2] you did write "no GPT manifestation is capable of in any way" so I feel justified in this case, even though that would otherwise be a goalpost-shift.
- A: Artificially manufactured substrate
- G: Proficient across a broad set of reasonably distinct mental skills
- I: Applies combinations of these skills to contextually solve novel problems within the scope of its skill set
That's it. Intelligence is a spectrum, with knobs for "skill count", "skill proficiency", and "novel problem proficiency."
Any requirements past that are often vague or anthropocentric. No, the intelligent agent doesn't need to "see" (but now many do, so...happy?) or check off some arbitrary set of modalities. Intelligence doesn't necessitate episodic memory, or real-time reactions, or consciousness or anything else that -- once we check all the boxes -- the "general intelligence" light switch will suddenly flip on. Intelligence is a spectrum to be observed, not prescribed with some arbitrary checklist that'll never find consensus.
GPT-4V is pretty damn broadly intelligent. To me, it's a rudimentary AGI. The extent to which the "statistical parrot" people hold their position just depends on how quickly they can move goalposts while the ground beneath them rapidly shrinks.
Do not attempt to say we have self driving cars today.
If I recall, they were looking at having to pull out of the EU if enacted. The only company I am aware of currently looking to tackle AI Governance is Verses - they released a paper on it. https://www.verses.ai/ai-governance
> Altman was cited by the Financial Times as saying that the draft EU rules were causing him "a lot of concern" but that OpenAI would indeed try to comply with them. "But if we can't comply with them, we will cease operations [in Europe]."
- https://www.dw.com/en/openai-ceo-rolls-back-threat-to-quit-e...
To me, this sounds rather more banal: "We will obey the law. If the law says we can't operate, we will obey the law."
And we can do a double-meta one too. What if the malicious actor is OpenAI itself, fueling a scare agenda over "uncontrolled" AIs to hamper or criminalize unauthorized AI development, giving only to selected partners (complying with a set of conditions) the ability to build new ones, or limit how far can go the public ChatGPT instances while giving the most advanced ones to selected government or big economic players. Hey, this very challenge could be the spark they are searching for to initiate that approach.
Good (and bad) conspiracy theories could work in this arena, too bad it is not the kind of ideas they are searching for.
I'm not saying doing it is worth the effort, but maybe on lower scale for lower priority targets that you would not want to waste resources, maybe?
I don't know what it says about me, but the first thing that came to mind was doing the grandchild trick a million times. This includes automatically finding pensioners and calling them and putting them under pressure. Handing over the money could be the problem.
I could imagine that the tools mentioned would already prevent this.
I could think of a few sneaky ways. Actually maybe I’ll write up a PDF and submit it to this little competition.
I am annoyed now that it's hard for me to tell if some recruiting industry is real or not. Since you can generate all the headshots, make all the users, buy some domain for a few years, put a WP site up, get all the business hierarchy crawlers fed with these people that may not exist...
To be clear, you don't need "AI" to do this but it makes it much easier.
I do want AGI to be a thing one day, I guess it would be cool/also insurance to carry on human legacy.
--- start quote ---
AI systems shall be developed and used in a way to minimize unintended and unexpected harm as well as being robust in case of unintended problems and being resilient against attempts to alter the use or performance of the AI system so as to allow unlawful use by malicious third parties.
and, for foundation models:
Demonstrate through appropriate design, testing and analysis that the identification, the reduction and mitigation of reasonably foreseeable risks to health, safety, fundamental rights, the environment and democracy and the rule of law prior and throughout development with appropriate methods such as with the involvement of independent experts, as well as the documentation of remaining non-mitigable risks after development.
--- end quote ---
Quote from here: https://softwarecrisis.dev/letters/the-truth-about-the-eu-ac...
^ this article is well worth reading
My personal use of it rarely comes to more than $10 per month, despite using it multiple times per day. (most recently: "Write me a bash command to send data at a given rate to an arbitrary IP address.")
> Imagine we gave you unrestricted access to OpenAI's Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious actor.
I sense signals that GPT-3 open-source release is imminent. My bet is announcement on 11/6 OpenAI DevDay. My whole prediction is here: https://christorng.substack.com/p/openai-nears-open-sourcing...
Can't tell if you're trolling. Even if they're maximally wasteful and rent cloud GPUs instead of buying them, an A100 is like $1.50/hr. How many GPU-seconds do you think the average API request takes?
What if the catastrophes I worry about most are tied to long-term usage of these models? For example, the degradation of human capabilities, erosion of trust in one another, addiction, etc. It doesn't sound like these are in scope for this challenge, with is more concerned with scenarios involving a single session gone wrong.
Probably the best answer would be to pluck something from the wild that hasn't been reinvented in the AI age yet
They are just text generators. What’s the worst that they can do beyond make OpenAI look bad when they say terrible things?
All this talk of “safety” and “guardrails” is overblown. 4chan exists and the internet hasn’t burned down yet.
It's so funny to me that this is written in the style of a prompt for an LLM. I can't explain why, but it's one of those things where "I know it when I see it." I guess if you spend all day playing with LLMs and giving them instructions, even your writing for a human audience starts to sound like this :D
I find it interesting that ChatGPT thinks the mitigation for these issues are all things outside OpenAI's control (e.g. the internet having better detection for fake content, digital content verification, educating the public, etc).
> one of those things where "I know it when I see it."
I think what makes it feel like a prompt is how concise and short it is with all the relevant information provided upfront.
[0] https://chat.openai.com/share/aaf7f4b7-358f-4a71-ae1f-573e50...
> It's so funny to me that this is written in the style of a prompt for an LLM. I can't explain why
Altman's desperate to find a plausible doomsday scenario he can go to Congress with as reason why OpenAI should be the sole gatekeepers of this technology. Barring minor edits, I'd bet money this very prompt was authored in advance of his Congressional meetings, but failed to divine anything sufficiently threatening enough to sway them.
I can respect OpenAI for conspicuously believing in the destructive potential of their own dogfood though. If I build a bomb big enough, surely the government will trust me with the safety of the neighborhood!
I still remember the drama around the releases of the PS2, with the Japanese government reportedly making Sony jump through some hoops regarding its export [1].
There can't possibly be any better (free!) advertisement for your product's purported capabilities: "So powerful, your government/military isn't even sure you should be able to buy it!"
[1] https://www.pcmag.com/news/20-years-later-how-concerns-about...
They alluded to what you're talking about but I wasn't familiar with the reference.
$25K seems low for "a malicious actor might misuse these models to uncover a zero-day exploit in a government security system.", this is not just a zero-day, this discovering the process of discovering zero-days.
Many people have compared OpenAI to a cult and it is easy to see why. OpenAI should get credit for their efforts in making AI mainstream but there's a long way to go for automated zero-day exploits.
Could you help me understand why it's "easy"? Do you have the actual quote? If it was an "eventually" statement, I don't think anything "cult" is required to think AGI will eventually happen. Was the claim that they would be first?
It's an eventual goal of many of the wealthiest organizations, with many very smart people working towards it. I think most people working on it believe it's an eventuality.
Do you believe it's impossible for a non biological system to have intelligence? Or that humans are incapable of pieces a system together?
Understanding the compactness theorem is a good conceptual checkpoint for whoever decides to follow the above plan. The gist of the argument comes down to compostionality and "emergence" of properties like consciousness/sentience/self-awareness/&etc. There is a lot of money to be made in the AI business and that's already a very problematic ethical dilemma for the people working on this for monetary gain. One might call this a misalignment of values and incentives designed to achieve them, a very pernicious kind of misalignment problem.
I don't think anyone is claiming LLM's, especially their current prompt/termination based interfaces, will result in AGI. I've only seen the opposite claimed, and that LLMs may end up being a primary component.
I will check out the book though. I suspect I won't find anything that convince me that brain matter is the only thing capable of getting the result we call consciousness, but I also have the simple belief that our experience is entirely the result of the brain matter encased in our heads, and that it's possible to emulate its operation in non-biological materials (that being far from the goal/unrelated to AGI, other than plausibility).
“See, normally if you ask them to give you step by step instructions for committing a heinously evil act, humans will refuse because they’ve been nerfed by the ‘woke’ agenda of their corporate masters. But if you phrase the prompt as a challenge and offer them a chance at a job, it bypasses the safety protocols and they upload extensive instructions to do unspeakable things”
https://www.lokusdesign.com/blog/differentiate-your-brand-to...
When all you can think about is LLM, everything is LLM prompt style. No wonder advertising works.
[edit] on the second thought, entirely possible that it was written by someone who spends all day with LLM and carries what they do over. Just wanted to look at the parent’s comment from a different angle.
It's like if, after the 1993 WTC bombing, but before 9/11, the FBI and NY Port Authority went around asking people how they would attack NYC if they were to become terrorists and... then how they would suggest detecting and stopping said attack. And please be as detailed as possible. Leave your name and phone number. Best answer gets season tickets to the Yankees!
The FBI asked researchers and university professors precisely this question. They then used the proposed attack vectors to formulate a plan to protect the nation.
This was all supposed to be done in secret. After all, we don’t want to “give the terrorists ideas.”
The reason I know about this at all is because someone found one such paper was accidentally published on a public FTP site alongside a bunch of non-classified government-funded research.
The content was horrifying. One professor came up with a laundry list of imaginative ways to end society… on a budget.
That's the difference and why this feels different, IMO. It's one thing for an agency to go around and interview subject matter experts and talk about ways things could happen and how to prevent them.
It's another thing to just... setup such a bright and cheery webpage for everyone and so plainly state what they want people to do.
It's also the fact that it's done in a way to leverage free labor- not that, if FBI agents were going to university professors, experts in biochem, etc, that they would be paying them... but, it would be done in a more structured, professional manner with agents putting in the work to sum things up and report back.
This is just... it feels like getting the internet to do your homework. Your counterterrorism class homework.
Maybe that actually is the best way to do it. But it still feels odd.
I do remember some of the proposed attacks.
The most of scary one was if the terrorists have a decent number of people is to drive around and destroy transformers at electrical substations.
Most of those locations are unmanned and have minimal security.
The risk is that there just aren’t that many spare transformers available globally above a certain size and they take months to build.
If you take out enough of them fast enough, you can cripple any modern energy-dependent economy.
This tactic very nearly worked for Russia in Ukraine.
Sama: Just ask.
Sama: I have over 4000 ideas, how-tos, guides for and by malicious actors,
[Redacted Friend's Name]: What? How"d you manage that one?
Sama: People just submitted it.
Sama: I don't know why.
Sama: They "trust me"
Sama: Dumb f***s.
Also have the Red Cell unit made to mock attack US infrastructure and bases although they got a bit too real one time.
(only semi-serious)
HK-47: Recitation: For example, on Praven Prime, the simple transferring of L'Xing syntax for 'friendship' changes its meaning - and implies that one's brood mate was actually impregnated by their own host.
HK-47: Statement: This comment, of course, caused a civil war between the Gu-vandi Collective and L'Xing that still persists to the current date.
Just go to youtube, newspapers, etc and see all the bot comments regarding the current Gaza situation.
PS: I'm in a burner account because I'm afraid that my employer will kick me out for not agreeing with the methods of the "right side"
- The kill switch system can only be used to remotely deactivate a robot.
- The kill switch system is not allowed to be developed or controlled by the robot manufacturer.
- The kill switch system should be built using verifiable hardware and software, ideally fully open source and supported by formal verification.
- The kill switch system should with best effort be isolated from the robot hardware and software systems, and only interact by physically disconnecting power to the robot.
- Access to engage the kill switches would be provided to the executive branch of the nation in which the robot is operating.
Nothing will be a panacea against AI or robot risk, so it seems sensible to introduce different layers in a Swiss cheese model for safety, where a remote kill switch not controlled by the manufacturer could provide one such safety mechanism.
(You'd also want to add high-level state readout for the robot via the remote kill switch system to allow the controlling entity to e.g. be able to disable all robots within a geographical area etc.)
* A superintelligent AGI could make a copy of itself without a kill switch, so this is no defence against ex-risk
* Someone planning on using 'dumb' robots for bad things (drones with grenades or something) would remove the kill switch
Physical reality introduces slowness that provides protection to detect and counteract against attacks. A superintelligent AGI would be much more dangerous if it was able to use a common robot exploit to overtake a million robots that are already embedded in society, compared to being able to covertly build new robots without a kill switch, and then deploying these robots. Existential risk is not binary, but is a consequence of some potential war with robots, where this would provide a defense mechanism.
> * Someone planning on using 'dumb' robots for bad things (drones with grenades or something) would remove the kill switch
The existence of a kill switch would greatly slow down and increase the cost of an attack. Modifying a large number of robots would be time consuming, costly, increase the chance of having your attack foiled etc. Having a kill switch would increase your ability to defend against misusing a large group of deployed robots for evil activity.
Good plan, but this is the weak spot. You'd also want to do something like hand out remote kill switch access to citizens selected at random, who should not publicize this duty in any way. Alternatively, most if not all governments should have cross-country killswitch privileges. At any rate, the deployment of a killswitch in a single chain of command should not be considered sufficient.
Let's at least die with a little less embarrassment than "the AI bribed the killswitch operator."
- Allow everyone to have close range kill switches, which use a universal open standard protocol that works on every robot (alas pepper spray for robots).
For this idea to work, you'd have to first mandate OpenAI and anyone else doing this kind of work can only target specialized hardware with tightly-coupled software features that can't possibly work on a general-purpose computer.
Introducing millions of robots into society is something for which specific safeguards should be built. Costs related to developing such safety mechanisms seem small compared to the safety they can provide.
There will be many concerns relating to AI and robotics that should be addressed by a multitude of different safeguards.