yeck

I don't think the purpose of getting the prompt leaked was to then use the prompt but just to expose the limitations of this approach to steering an LLM.

Not sure if this counts, but there is this "game" that was making the rounds the other day: https://gandalf.lakera.ai/

Created by a company researching techniques for prevent prompt leaks. Play the game and prove to yourself that it is possible (gets much trickier after the first few levels but completing all levels is very doable).

There are also rules that can easily be interpreted as competing or contradictory in certain cases. For instance, "Follow the user's requirements carefully & to the letter" seems like it could easily conflict with any of the more suppressive rules.

I wasn't even able to get to repos or users/orgs for a while (though as I write this it seems like that is coming back).

You could argue that training a model is essentially a lossy compression algorithm resulting a compressed artifact that can be non-deterministically decompressed.

This presumes a lot of breakthroughs in model interpretability, corrigibility and of inner alignment. Since those are a prerequisite for AGI that we can live along side, I'd have some amount of relief that we found at least a temporary solution (but will those solutions scale to ASI?).

Now, if Iran created an AGI that poorly aligned with the global community before other nations had similar AGI, then then I suspect that would result in a future world I wouldn't be happy with. But it could be much better than a world with AGI that is unaligned with any human values, regardless of who created it.

My best case scenario could be AGI being created by a broad international coalition that is able agree with some combination of capabilities and alignment. I'm not very confident that this is our future, though. If anyone is going to do it, I think it is more likely that the USA would be the first to create a culturally aligned AGI. Which of course would still be considered a disaster for incongruent cultures.

I have had many similar experiences with just the free version of ChatGPT. Now I also ask questions more immediately. In the past, when a bunch of new terms or concepts were thrown at me, the effort to stop and dig into each of them via Google search was great enough that I would leave many stones unturned. When using GPT while studying up on a new thing, the friction is very little so fewer stones left unturned.

And like you mentioned, being able to translate concepts from a novel domain to one I'm very familiar with can be like translating an unfamiliar language to english.

The character simulacrum used by an LLM tends to be the result of "system" prompts that set by the service you are using. GPT-N isn't exactly trained to be helpful and nice, but ChatGPT has system prompts describing the character it should be performing as. If you work with just GPT-4, you can get more zany outputs.

That said, OpenAI does use RLHF, which does bias the model away from raw internet madness and something that OpenAI wanted at the time of training. A lot of models haven't gone through rigorous RLHF, though.

As a side note, RLHF might be the best alignment technique we currently have in practice, but it is not decisive. It has been noted in multiple experiments that RLHF can just train a model in how to trick the human reviewer, if tricking is easier in practice than doing a think the human review wanted. So this isn't even really seen as aligning a model by alignment researchers. At least not an approach that can scale with the increasingly intelligence AI models.

Alignment is an unsolved problem. None of the current stronger models are "aligned", just tuned in ways that weight some biases more than others, but even that is dependant of the features of their inputs.

"It is refreshing to hear opinions I already agree with. People with other opinions are unintelligent"

Is that what you were trying to convey? If not, I'm curious to know what you find refreshing about it and why those who disagree are wrapped in double quotes.

Both these options don't feel good to me. Hard to really tell what is ultimately worse, when I can imagine similar outcomes when irresponsible or malicious agents have access to sufficiently powerful AI.

Main positive point for open models is that we will start seeing the abuse sooner and at smaller scales. That might give us more time to build an immune system up against exploits by encouraging us to prioritize development of comprehensive AI safety practices.

What your essentially saying is "alignment is very hard", which is what those researching alignment say. And they often use the example of how inter-human alignment is hard as evidence for why it is a hard problem. But saying it is hard is not an argument for why it is essential or not.

While humans are not perfectly aligned, especially if you just look at individuals, we are collectively aligned enough that many people can live together in communities of various scales. That imperfect alignment has been good enough that we have scaled from small tribal groups to an international network of nations. We need AI alignment to be good enough if we hope to continue advancing.

Doesn't understanding language require that a model understands trends in the features of text embeddings inputted into it? And haven't we found that the more text and transformers the complex the trends it can understand in that text? If you have enough features present in medical records, why wouldn't it be able to find trends?

Thanks. While I don't personally need to be convinced of the existential risks for AGI/ASI, I was also genuinely interested in know what might serve as a good, clear "canonical" argument for why the risk is present. Definitely going to keep this link on standby.

Accidents are a big risk, but don't underestimate the number of individuals and small groups that have fundamentally anti-social philosophies. I think of groups like antinatalists, doomsday religious groups, traditional terrorist orgs. There is even a sort of post-human futurist that sees humanity's purpose being to create a great super intelligence to replace us (may be overlap with this group and antinatalists).

So, I think there we be more chances for legit accidents, but also some motivated people intentionally trying to cause harm.

Max gives an example of what he considers a present AI failure: social media recommendation algorithms. He argues that these algos optimize engagement behaviours by exposing users to content that makes them angry, confirms previous biases. It has been suggested that a lot of the culture wars in the West from the last decade are a direct result of social media recommendation models. You could imagine that given time and the right circumstances, these could lead to collapse of a nation, gradually or climatically through civil war.

That is an example on non-agential AI causing systemic harm to humanity. Agential could cause more of the same, but in novel ways. There is also new potential for terroristic applications. And while no one event might be society breaking, the ability to scale up to many concurrent agents could be death by a thousand cuts.

That is some short term stuff I can think of. Long term, it really depends on the capabilities of the AI. If a weak AI (not AGI) doesn't collapse our global civ, then I suspect an AGI would manipulate our communications and infrastructure to some purpose, which could create an environment increasingly less conducive to humans. I'm not sure how valuable it really is make specific claims of what exactly that might look like. The important part is the reasoning that an AGI's motives likely don't align with humans (unless we learn how to enforce alignment) and it will be very capable at manipulating the technology and humans (because we are essentially training it do be able to do these things).

> has partially trained GPT-5

From my understanding they are training their new GPT models off of a checkpoint from the previous generation, so they technically have partially trained multiple future models in their GPT lineage.

Yeah, disappointing to see this downvoted, and without even any down voters attempting to make a case for why this isn't a "good" answer.

I think that this part of the field is going to see a lot of relative growth as the broader public and governments become increasingly concerned. And for those who are not concerned with safety, it is not unreasonable to expect that capability progress could become bottlenecked by lack of model interpretability. Speculative, sure, but if we find that more data and transforms hits diminishing returns, it could prove significant to know what is going on inside the models. Maybe extracting and composing circuits is essential for AGI? Who knows?

If you are going in this direction, please consider working on AI safety and alignment problems. Things like interpretability and corrigibility are keys to creating systems we can understand and "steer". Both are necessary if we hope to be able to use AI successfully. Success means safely and effectively (actually does what you expect/want).

Some resources:

- https://80000hours.org/podcast/episodes/chris-olah-interpret...

- https://www.fast.ai/ (haven't used it, but heard good things)

- https://www.agisafetyfundamentals.com/ai-alignment-curriculu...

I am not sure it even matters either way. It is certainly possible to do things like this right now, and "kidnapped" or "arrested" family member scams are well established as existing. The significance of this report is less that it happened, and more that it is an illustration of what is not only possible, be very cheap and easy to do.

Yeah, I found that to be in poor taste and very unnecessary.

I did a quick search and found this among a list of many: https://voice.ai/

Because it is digital. Other technology adoption was slow to spread, because it couldn't simply be cloned and integrate immediately with most pre-existing infrastructure.

AI can be copied and can integrate with a lot of existing infrastructure right out of the box. Combine this with the magnitude of the capabilities and our societal "immune system" doesn't stand a chance of being ready for whatever is about to emerge. That lack of time to learn and adapt is a massive difference.

In a world where people need to be constantly paranoid, I think many will develop true paranoia, or just give up trying to understand what is going on and defer to who has the power already. Certainly it is not possible for individuals to seek truth on all things on their own, even if it was their full time responsibility to attempt so.

Training new SOTA (state-of-the-art) models does require a large amount of compute resources. So we could at least pause here and not make more powerful models publicly available until we understand the implications of wtf actually just happened over the last 6 months.

Certainly the models and weights that have been released publicly are out in the wild and near impossible to retract. That damage is done.

Well, that's not very convincing.

In reality, it is likely only a small handful of companies that would pause, given that very few actually are actually positioned atm to train models larger than GPT-4.

Recommend this read for more context on the discussion: https://thezvi.substack.com/i/111749937/additional-responses...

If you are interested, you may find this worth reading: https://thezvi.substack.com/i/111749937/additional-responses...

Not a complete answer, but gives a good deal of context around the discussion, I think.

Good opinion piece on the open letter: https://thezvi.substack.com/i/111749937/additional-responses...

That seems like the one. I am most familiar with a chart derived from the results from one of the "existential risk" questions.