Hacker News Reader

Comment by danpalmer | Hacker News Reader

I find it fascinating that emergent behaviour of these models is being used to wrangle them into doing what we want, but I find it worrying that this is considered the right approach for issues of safety. It seems obvious to me that an out-of-band approach to safety and security is necessary and that this method is never going to work in a generalised way.

View on HN

I spent a few years of my career in formal verification. As a result of that, I suspect that the desiderata of “technical alignment” research is most likely undecidable, and the existing RLHF and prompt engineering approaches are probably the best we can do.

A neural network is a large, nonlinear function. In many applications, such as those that release their source code and allow fine tuning, it is adaptive. Internally the weights evolve according to solutions of differential equations. All of these things smell of undecidability when trying to prove safety/alignment properties of them:

- The theory of differential equations (which governs the evolution of the weights during training and retraining) is undecidable - Nonlinear functions have decidable cases as long as we stick to polynomials, but even then the decision algorithms are doubly exponential—-much higher complexity than anything in NP-complete, so practically undecidable for very large functions - The neural network is a very large function

In summary, they are large, nonlinear, and adaptive, these three characteristics make the problem smell intractable and likely undecidable to me.

Isn’t prompt engineering a human way to explore the latent space of the model to create the mathematical interactions we want/deem useful? A human way of traversing the hyperdimensionalty. Given your background, would you say that prompt engineering is about finding the spaces in latent space the bound the model’s “knowledge” for our uses?

Yes, but I think the folks who work on technical alignment would like to have stronger guarantees—I think they would like to end up with formal guarantees that the AI will only be helpful, and not harmful.

I agree, prompt engineering is a fuzzy subset of the possible space that could provide the guarantees.

> Internally the weights evolve according to solutions of differential equations.

Is there a closed form representation for gradient descent?

Is it even possible to know the surface attack area for potential jailbreaks when models have unknown emergent behavior, the inner workings are blackbox and the input is anything that can be described by human language?

I don't perceive this as even being solvable and on the looming question of alignment going forward towards AGI I see the same. An unsolvable problem as the very premise to resolution is a paradox itself. Elaborated in further detail - https://dakara.substack.com/p/ai-singularity-the-hubris-trap

I've been wondering this too. How can the smart people working on these systems think that this is the right way to control them? Jailbreaks and other ways of getting them to do unintended things seem akin to social engineering to me. Even well trained humans can be convinced to break protocol sometimes, and I don't think anyone would say these things have a superhuman ability to detect social engineering attempts yet.

Reply on news.ycombinator.com