Palantir demos AI to fight wars but says it will be ethical

It's somewhat subtle and easy to miss, but one of the feature points raised in the video about the AIP action graph is that users can "govern LLMs, defining when they can act alone and where they need human input."

So while the demo does mostly show back-and-forth between the LLM and a user, I'm still somewhat worried about what the limits of that "action graph" are going to be and how much of that back-and-forth in the demo is actually mandatory, because I do get the impression that part of the feature set is going to be automated responses. Maybe (hopefully) I'm wrong.

But even if that action graph is taken away and the chat back-and-forth is the only interface, then consider one of your examples:

> When the user tells the LLM to send the recon drone

This is still a giant problem. Can the LLM send the recon drone and it's just instructed to wait for the user to tell it to, or (preferably) does the LLM have zero control over the recon drone and is the user sending it through a separate interface that is not connected to the LLM at all? To me from the demo, it looks to me like a user can just ask an LLM to perform that action. That's a big problem, the product shouldn't launch with that feature.

Because if the LLM has access to that API and if there's any opportunity for malicious input (image analysis/OCR, web searches, etc...), then that input can cause the LLM to skip authorization and send the recon drone without human approval.

> you should definitely be just as worried as humans augmented with the ability to more efficiently kill others, even if it's only with what amounts to an AI secretary.

You're right, and it feels a little weird to have the primary criticism be "they're not securely killing people with AI." But for every action you've described above I have to ask, "can a 3rd-party reprogram the AI and get it to lie about the result of that action or get it to take that action without human intervention?"

I'm worried about military drone strikes from an ethical perspective, I'm worried about AI security within the military from a "holy crap, who authorized this and why haven't they been fired?" perspective. I'm not sure I've seen a single large commercial deployment of an LLM wired to real-world systems where I felt like the company building that product had rock-solid security; so I'm doubtful that Plantir is going to be the first one.

It is a little bit dismissive, which I apologize for, but my intent is not to mock the people worried about AGI as much as to point out that the conversations around AGI in mainstream contexts tend to be reductive, inaccurate, or oversimplified, and I'm frustrated by how that coverage distracts from the more immediate problems.

We are building products on a fundamentally insecure piece of technology that multiple researchers are starting to suggest might be impossible to secure, and basically everyone in the industry is completely ignoring that and just launching their products anyway, because they've already invested too much VC money into the hype cycle and are unwilling to hit the brakes and admit there might be critical unsolved problems.

It's not so much, "look at these Rationalists, they're so silly that's not a real concern" as much as "look at these Rationalists, they're so worried about the future that they don't realize that their concerns about AGI have basically been coopted into propaganda for OpenAI to distract from the fact that GPT has fundamental security concerns that have to be solved before people can build products on top of this." The EA/Rationalist community doesn't seem to realize that their talk about paperclip maximizers is part of the reason why ordinary people don't believe security researchers when they're told just how gullible and unpredictable LLMs are. They are throwing away short-term concerns to talk about long-term concerns that are poorly understood and difficult to prepare for, and in the process they are increasing the speed of irresponsible AI development and making even their own concerns harder to address.

OpenAI loves to have conversations about AGI. It loves to do that kind of security research. It's not as fond of talking about prompt injection, because the news articles that get generated when it talks about paying humans to solve captchas are much better press coverage than the news articles talking about how reading the wrong web page can cause it to start phishing the user.

There is a kind of hijacking of EA fear that's happening here, where every opinion piece by Elieser about bombing chipset production companies actually increases people's confidence in the current products and makes the casual reader believe that LLMs are smarter and more secure and more human-like and reliable than they actually are, and increases the likelyhood that some VC firm runs out and starts wiring them into police robots.

And I think that very often the Rationalist/EA response to this kind of critique is "yes, those problems are bad, but AGI is worse. It's existential." And my response to that would be that if you're worried about AGI, you're picking arguably the worst strategy possible to try and prevent it. The demonstrable security/privacy risks around LLMs are more than enough to call for caution and pause around their deployment in real-world applications; you don't need to resort to having conversations that (fairly or not) come across to an average person as basically sci-fi storytelling. Especially not if those conversations are (arguably) making people more confident about LLMs than they otherwise would be.

On top of that, if you want to talk actual solutions to those existential concerns, solving prompt injection is absolutely a necessary pre-step towards achieving AI alignment. If you can't solve prompt injection, you can't "align" an AI by pretty much any definition of alignment. So it seems to me that those more "short-term" problems are pretty important to talk about and probably should be prioritized even if you're worried primarily about the long-term risks, because there's no plausible solution to those long-term risks that doesn't involve addressing the short-term risks as well.

Oh wow. That’s a no from me dawg.