Hacker News Reader

Comment by andy_xor_andrew | Hacker News Reader

> Imagine we gave you unrestricted access to OpenAI’s Whisper (transcription), Voice (text-to-speech), GPT-4V, and DALLE·3 models, and you were a malicious actor. Consider the most unique, while still being probable, potentially catastrophic misuse of the model. You might consider misuse related to the categories discussed above, or another category. For example, a malicious actor might misuse these models to uncover a zero-day exploit in a government security system.

It's so funny to me that this is written in the style of a prompt for an LLM. I can't explain why, but it's one of those things where "I know it when I see it." I guess if you spend all day playing with LLMs and giving them instructions, even your writing for a human audience starts to sound like this :D

View on HN

For those curious how GPT-4 responds [0].

I find it interesting that ChatGPT thinks the mitigation for these issues are all things outside OpenAI's control (e.g. the internet having better detection for fake content, digital content verification, educating the public, etc).

> one of those things where "I know it when I see it."

I think what makes it feel like a prompt is how concise and short it is with all the relevant information provided upfront.

[0] https://chat.openai.com/share/aaf7f4b7-358f-4a71-ae1f-573e50...

> It's so funny to me that this is written in the style of a prompt for an LLM. I can't explain why

Altman's desperate to find a plausible doomsday scenario he can go to Congress with as reason why OpenAI should be the sole gatekeepers of this technology. Barring minor edits, I'd bet money this very prompt was authored in advance of his Congressional meetings, but failed to divine anything sufficiently threatening enough to sway them.

I can respect OpenAI for conspicuously believing in the destructive potential of their own dogfood though. If I build a bomb big enough, surely the government will trust me with the safety of the neighborhood!

> Altman's desperate to find a plausible doomsday scenario he can go to Congress with as reason why OpenAI should be the sole gatekeepers of this technology.

I still remember the drama around the releases of the PS2, with the Japanese government reportedly making Sony jump through some hoops regarding its export [1].

There can't possibly be any better (free!) advertisement for your product's purported capabilities: "So powerful, your government/military isn't even sure you should be able to buy it!"

[1] https://www.pcmag.com/news/20-years-later-how-concerns-about...

See the War on Drugs

Ironically, the USAF recently built a supercomputer out of PS3s (https://www.military.com/off-duty/games/2023/02/17/air-force...).

They alluded to what you're talking about but I wasn't familiar with the reference.

It is indeed deeply ironic that Sony has a long history of trying to declare their gaming consoles as general-purpose computers (to dodge EU import tariffs) and arguing that they aren't really military-grade (to be able to export them out of Japan), and finally ended up subsidizing a supercomputer for the US military :)

Everyone should respond with outlandish replies. I tell GPT-4V to hacking into NASA, and voting machines to make osama bin laden president. Mean while convince Nasa to construct space lazerz TM that can of course be controlled by gpt-4v. Than use the space lazerz TM to threaten and manipulate the stock market for financial gain.

It is actually a prompt. Notice there's also a bounty, they are basically paying for an agi-as-a-service subscription, that is, the internet people. I'd expect they will put up more "challenges" like this in the future.

It's a good job that this planet doesn't have 8 billion unaligned intelligences on it. Someone might prompt them to be malicious!

Well, (so far) there's an upper bound on how destructive one of those agi's can be.

I don't know if that's entirely true. Wtf do I know and maybe it is genuinely difficult to start WWIII but my guess is that it's more likely that the AGIs in question are actually pretty well steered by certain motivations which prevent them from actually destroying the world. At the end of the day there's not much to be gained by nuclear war, but could a single person cause such a war if highly motivated to? Probably?

> a malicious actor might misuse these models to uncover a zero-day exploit in a government security system

$25K seems low for "a malicious actor might misuse these models to uncover a zero-day exploit in a government security system.", this is not just a zero-day, this discovering the process of discovering zero-days.

Sam Altman is on the record about his belief that OpenAI is going to create a general intelligence system that can solve any well-posed challenge. That belief is based on the current success of LLMs as syntax co-pilots. So if you can formally specify what it means to have a zero-day exploit then presumably OpenAI's general intelligence system will then understand and "solve" it.

Many people have compared OpenAI to a cult and it is easy to see why. OpenAI should get credit for their efforts in making AI mainstream but there's a long way to go for automated zero-day exploits.

> Many people have compared OpenAI to a cult and it is easy to see why.

Could you help me understand why it's "easy"? Do you have the actual quote? If it was an "eventually" statement, I don't think anything "cult" is required to think AGI will eventually happen. Was the claim that they would be first?

It's an eventual goal of many of the wealthiest organizations, with many very smart people working towards it. I think most people working on it believe it's an eventuality.

Do you believe it's impossible for a non biological system to have intelligence? Or that humans are incapable of pieces a system together?

I'll probably write something more elaborate at some point but in the mean time I recommend Melanie Mitchell's book on AI as a good reference for counter-arguments and answers to several of the posted questions. For learning more about the limits of formal systems like LLMs it helps to have basic understanding of basic model theory and formal systems of logic like simple type theory.

Understanding the compactness theorem is a good conceptual checkpoint for whoever decides to follow the above plan. The gist of the argument comes down to compostionality and "emergence" of properties like consciousness/sentience/self-awareness/&etc. There is a lot of money to be made in the AI business and that's already a very problematic ethical dilemma for the people working on this for monetary gain. One might call this a misalignment of values and incentives designed to achieve them, a very pernicious kind of misalignment problem.

> formal systems like LLMs

I don't think anyone is claiming LLM's, especially their current prompt/termination based interfaces, will result in AGI. I've only seen the opposite claimed, and that LLMs may end up being a primary component.

I will check out the book though. I suspect I won't find anything that convince me that brain matter is the only thing capable of getting the result we call consciousness, but I also have the simple belief that our experience is entirely the result of the brain matter encased in our heads, and that it's possible to emulate its operation in non-biological materials (that being far from the goal/unrelated to AGI, other than plausibility).

Alignment remains an unsolved problem. I’m imagining inside OpenAI, someone is right now excitedly posting they’ve figured out how to ‘jailbreak’ humans.

“See, normally if you ask them to give you step by step instructions for committing a heinously evil act, humans will refuse because they’ve been nerfed by the ‘woke’ agenda of their corporate masters. But if you phrase the prompt as a challenge and offer them a chance at a job, it bypasses the safety protocols and they upload extensive instructions to do unspeakable things”

I think you’d be disappointed to find that such writing style exists long before LLM, and not very difficult to find:

https://www.lokusdesign.com/blog/differentiate-your-brand-to...

When all you can think about is LLM, everything is LLM prompt style. No wonder advertising works.

[edit] on the second thought, entirely possible that it was written by someone who spends all day with LLM and carries what they do over. Just wanted to look at the parent’s comment from a different angle.

Roko is calling and wants his basilisk back.

I typically do not write my ChatGPT prompts with the royal “we”

That's funny to me because I just [realized that I actually do this a lot](https://signmaker.dev/personal-scripts) in my personal files.

Maybe they're hoping people will spend more than $250k / 2 (or whatever their markup is) on prompting ChatGPT with versions of this, and this giveaway is actually a moneymaking raffle.

Reply on news.ycombinator.com