As a security person... oh, no no no no.
Glad i dont have to secure that. Black box we don't really understand executing shell scripts in response to untrusted user input.
Has a scarier sentence ever been spoken in the history of computer security?
Imagine what a world-class programmer could accomplish in this world if they thought 100 times faster than a human, and had no fear of going to jail. Our world is an insecure machine, and we're preparing to run untrusted code with root access.
And sure, maybe we can try to use less-intelligent AIs to secure things before then, but the weak point is still humans. Social engineering is typically way easier than straight up hacking. We've seen these lesser AIs threaten people, and while we can keep bonking them on the nose when they do that, we can't prove or tell that they won't ever do it in a different situation, when they judge that it's likely to be the most effective course of action.
I hope every day that this is all just hype and that another AI winter is coming, because we need time (who knows how long) for a way to align these things. But I really fear that it isn't.
I respect the rationalist argument if there's a 1% chance to achieve AGI, we should spend 1% of our resources to think about it. Makes sense! But having it come up in every discussion with just speculative opinions doesn't really advance the cause, and gets tiring soon (to me, at least).
You're not far off from one of my stories. Spoiler: it ends with the nations still using analogue pots telephone service becoming the new first world, while former technologically-advanced nations crumble.
I now don't know if I'm happy or disappointed that the idea is not so original!
And wire the other side to the open internet!
As a non-security person, I don’t understand how anyone with the most rudimentary understanding of computers can think that you can get a hardened security boundary by explaining the boundaries to the computer in English and then _concatenating_ that with untrusted input. I wouldn’t even know where to begin such an argument. “Have you heard of adversarial games? Social engineering? Are you aware that human language is ambiguous?”
Like the closest we have is maybe sandboxing or air gapping, neither is applicable to the: i want my ai to do arbitrary shit in response to emails, usecase
I think it's wolves by treating the LLMs assistants to the user, who should have no more permissions than himself. Not as a process system with permissions to do anything an admin does.
You have permission to empty your own bank account.
Permissions can of course be part of the solution, but they can't solve the problem of when you want to delegate a task requiring scary permissions.