matthewdgreen

I know the authors so I’m aware of the intense amount of responsible disclosure they’ve been doing, and the very substantial behind-the-scenes impact this is having. So maybe the difference is the scope of this, the number of protocols and vendors they were able to detect and attack in one go, and also the way they’re approaching disclosure. In other words they’re being much more systematic in every aspect of the research, disclosure and re-analysis. I expect some people won’t take seriously and there will be more papers in the future.

I mean another way to put this is: maybe there’s a problem if you can say “there’s loads of previous work” and yet massive and systemic problems still exist. Where that problem is (holistic nature of the research or the disclosure process) is probably something you could drill down into. But you’ve basically admitted the previous research didn’t do the job, so all we’re doing is haggling about the price.

By next year encryption will be vastly more prevalent across geostationary satellite links, and it will be entirely due to this research (the actual mechanism being “everyone who ‘knew’ this internally now being empowered to fix it, rather than uselessly ‘know’ it, because now it’s public and newsworthy and embarrassing.”)

I’ll let other people comment on the actual novel elements of the research, because those exist too. But I want to point out that some huge portion of the value of public security research is really “intellectual garbage pickup”: calling out bad technical debt that “everyone knew about” and turning it into actionable security upgrades. Security research is a good part of the reason it’s mostly safe to browse the web on public Internet connections, when it wasn’t a decade ago.

PS As someone who is very cynical about security deployment, even I thought cellular network backhauls would all be encrypted as a matter of course by now, at least in the US.

Hostnames, IP addresses and maybe occasionally an HTTP connection that lets you tie all that metadata to an actual human identity.

Battery storage is always measured in the amount of power that can be delivered (Watts). Secondarily it’s measured in the number of hours that power can be delivered (hours, which is almost always about 4.) To get MWh you multiply watts times hours. This is standard in the industry and has nothing to do with reporters.

Does anyone have actual numbers on what France’s nuclear fleet cost? I thought it was somewhat shrouded in mystery due to government and national security subsidies.

The threat model described in TFA is that someone convinces your agent via prompt injection to exfiltrate secrets. The simple way to do this is to make an outbound network connection (posting with curl or something) but it’s absolutely possible to tell a model to exfiltrate in other ways. Including embedding the secret in a Unicode string that the code itself delivers to outside users when run. If we weren’t living in science fiction land I’d say “no way this works” but we (increasingly) do so of course it does.

My point is that if the threat model is “agent got sensitive information and wants to exfiltrate it”, looking at the code isn’t going to save you. You’re already dead. This was the threat outlined in TFA.

The premise of TFA as understood it was that we have lethal trifecta risk: sensitive data getting exfiltrated via coding agent. The two solutions were sandboxing to limit access to sensitive data (or just running the agent on somebody else’s machine) and sandboxing to block outbound network connections. My only point here is that once you’ve accepted the risk that the model has been rendered malicious by prompt injection, locking down the network is totally insufficient. As long as you plan to release the code publicly (or perhaps just run it on a machine that has network access), it has an almost disturbingly exciting number of ways it can do data exfiltration via the code. And human code review is unlikely to find many of them, because the number of possibilities for obfuscation is so huge you’ve lost even if you have an amazing code reviewer (and let’s be honest, at 7000 SloC/day nobody is a great code reviewer.)

I think this is exciting and if I was teaching an intro security and privacy course I’d be urging my students to come up with the most exciting ideas for exfiltrating data, and having others trying to detect it through manual and AI review. I’m pretty sure the attackers would all win, but it’d be exciting either way.

Yes. If by "subtly obfuscated" you mean anything from 'tucked into a comment without encoding, where you're unlikely to notice it', to 'encoded in invisible Unicode' to 'encoded in a lovely fist of Morse using an invisible pattern of spaces and tabs'.

I don't know what models are capable of doing these days, but I find all of these things to be plausible. I just asked ChatGPT to do this and it claimed it had; it even wrote me a beautiful little Python decoder that then only succeeded in decoding one word. That isn't necessarily confirmation, but I'm going to take that as a moral victory.

Understood. I read the article as “here is how to do YOLO coding safely”, and part of the “safely” idea was to sandbox the coding agent. I’m just pointing out that this, by itself, seems insufficient to prevent ugly exfiltration, it just makes exfiltration take an extra step. I’m also not sure that human code review scales to this much code, nor that it can contain that kind of exfiltration if the instructions specify some kind of obfuscation.

Obviously your recommendation to sandbox network access is one of several you make (the most effective one being “don’t let the agent ever touch sensitive data”), so I’m not saying the combined set of protections won’t work well. I’m also not saying that your projects specifically have any risk, just that they illustrate how much code you can end up with very quickly — making human review a fool’s errand.

ETA: if you do think human review can prevent secret exfiltration, I’d love to turn that into some kind of competition. Think of it as the obfuscated C contest with a scarier twist.

He wrote 14,000 lines of code in several days. How much review is going on there?

So let me get this straight. You’re writing tens of thousands of lines of code that will presumably go into a public GitHub repository and/or be served from some location. Even if it only runs locally on your own machine, at some point you’ll presumably give that code network access. And that code is being developed (without much review) by an agent that, in our threat model, has been fully subverted by prompt injection?

Sandboxing the agent hardly seems like a sufficient defense here.

I personally think Trump will be too old to run, but I don’t think for a second they won’t try to run him if he’s able. They always start by making it a “joke”.

They changed the features of the system considerably. It used to let you set the speed and handle manually-initiated lane changes. Then they started having it initiate lane changes on its own; because this was annoying, they added a “minimize lane changes” feature. Then (and I think this coincides with “FSD” being released to customers who opted in) they removed that minimize option and replaced it with an entirely new system with different modes, one of which is called “Hurry”. This combines aggressive lane changing with speed choices, but the speed never seems to be quite what you want it to be (manual control is gone.) This isn’t unique to my car: if you search for complaints on Reddit and Tesla forums you’ll find plenty.

I'm a 2018 model year owner who also paid for FSD. The current FSD is not "fine" or "FSD" at all. It requires constant monitoring because one out of every 50-100 decisions will be problematic. Even the highway driving (autopilot), which was great initially, has recently degraded in quality: now it's much harder to get it to consistently stay in one lane or choose the speed I want to go.

Tesla knows the current FSD isn't fine; if it was, they wouldn't be upgrading the hardware to handle bigger models. So there's nothing surprising about expecting Tesla to deliver the same hardware to people who already paid them for the product they're still trying to deliver. If you're satisfied not receiving the full product you paid for, that's fine, of course. Heck, you're free to write Tesla a donation check. But Tesla should deliver the product (or a refund) to those who aren't.

Not one disagreement with what you're saying. I have cameras outside my house. I'd like them to be end-to-end encrypted and am perfectly fine with a voluntary self-report feature. But what Ring seems to be pushing for is opt-out mass surveillance, and once connected to AI this means we're going to the bad place.

I’m ok with that as long as I, the camera owner, am choosing to hand over the footage. At best I can see some sort of watermarking to ensure that it’s legitimate.

Waymo is coming to DC and London. I assume they'll also be targeting centers of political power throughout the rest of the world. Starting on the West Coast made sense as a technical strategy, but the huge disadvantage is that nationa legislators and regulators don't go there enough. So lots of people can blithely assume that self-driving taxis are a decade away, or that Elon Musk is going to solve it. That's a dangerous business problem. Waymos don't seem possible til you've ridden a few of them, and then they're ordinary.

>I think the age control attempts are much more interesting and consequential.

Exactly. The age verification and the "human identification" problem are the same problem. Anyone who thinks this tech/policy ends with identifying 16-year olds is out of their depth. Everyone, including governments and adtech firms, is planning for a world post-AI whether they're smart enough to realize it or not. I'm just worried about the details of that world.

I was retained as an expert witness in some of the cases involving Google, so of course I’m aware that Google keeps logs. (In general on HN I’ve found it’s always helpful to assume the person you’re arguing with might be a domain expert on the topic you’re arguing about; it’s saved me some time in the past.)

But Google’s internal logging is not the question I’m asking. I’m saying: can you find a single criminal case in the literature where police caused Google to disgorge a complete browsing history on someone who took even modest steps not to record it (ie browsed logged out.) Other than keyword search warrants, there doesn’t seem to be much. This really surprised me, since as an expert I “know” that Google has enough internal data to reconstruct this information. Yet from the outside — the experience that matters to people - they’ve managed to operate a product where real-world privacy expectations have been pretty high if you take even modest steps. I think this is where we get many of our privacy expectations from: the actual real-world lived expectations of privacy are much closer to what we want than what’s theoretically possible, or what will be possible in a future LLM-enabled surveilled world.

It's not at all clear how easy it is to obtain a user's search history, when users don't explicitly log in to those services (e.g., incognito/Private browsing), and don't keep history on their local device. I've been trying to find a single example of a court case where this happened, and my Google/ChatGPT searches are coming up completely empty. Tell me if you can find one.

The closest I can find is "keyword warrants" where police ask for users who searched on a given term, but that's not quite the same thing as an exhaustive search history.

Certainly my personal intuition is that historically there has been a lot of default privacy for non-logged in "incognito" web search, which used to be most search -- and is also I think why we came to trust search so much. I expect that will change going forward, and most LLMs require user logins right from the jump.

As far as the "I can see no reason" why LLMs should be treated differently than email, well, there are plenty of good reasons why we should. If you're saying "we can't change the law," you clearly aren't paying attention to how the law has been changing around tech priorities like cryptocurrency recently. AI is an even bigger priority, so a lot of opportunity for big legal changes. Now's the time to make proposals.

A simple answer to this is: I use local storage or end-to-end encrypted cloud backup for private stuff, and I don't for work stuff. And I make those decisions on a document-by-document basis, since I have the choice of using both technologies.

The question you are asking is: should I approach my daily search tasks with the same degree of thoughtfulness and caution that I do with my document storage choices, and do I have the same options? And the answers I would give are:

* As a consumer I don't want to have to think about this. I want to be able to answer some private questions or have conversations with a trusted confidant without those conversations being logged to my identity.

* As an OpenAI executive, I would also probably not want my users to have to think about this risk, since a lot of the future value in AI assistants is the knowledge that you can trust them like members of your family. If OpenAI can't provide that, something else will.

* As a member of a society, I really do not love the idea that we're using legal standards developed for 1990s email to protect citizens from privacy violations involving technologies that can think and even testify against you.

With an attorney you have a clear sense of when you pass outside of that privilege. With a friend or colleague you have a social sense of what's going to remain confidential, plus memories aren't perfect. "Preserving, recording and reporting every word" is not the same as any of these things. This cannot be the world we all have to live in going forward; it's not safe or healthy.

Why do employees need to be involved? It's AI. It is entirely capable of doing the surveillance, monitoring and reporting entirely by itself. If not now, then in the near future.

I have enormous privacy in my home. I can open up any book and read it with nobody logging what I read. I can destroy any notes I take and know they'll stay destroyed. I can even visit the library and do all these things in an environment with massive information access; only the card catalog usage might get logged, and I probably still don't have to tie usage to my identity because once upon a time it was totally normal to make knowledge tools publicly-accessible without the need for authentication credentials.

Not good. These tools (from search engines to AI) are increasingly part of our brains, and we should have confidentiality in using them. I already think too much about everything I put into ChatGPT, since my default assumption is it will all be made public. Now I also have to consider the possibility that random discussions will be used against me and taken out of context if I'm ever accused of committing a crime. (Like all the weird questions I ask about anonymous communications and encryption!) So everything I do with these tools will be with an eye towards the fact that it's all preserved and I'll have to explain it, which has a huge chilling effect on using the system. Just make it easy for me not to log history.

They also have a huge amount of line construction in the pipeline and have a tendency to do well at building things when it’s a priority. I feel like in just a few years we’ve seen anti-renewable/anti-China talking points shift from “China won’t build renewables” to “China can’t build renewables, it’s not economically feasible” to “ok, China is building renewables inexpensively and at massive scale but for some reason they won’t connect them to the grid” to “oh they are actually connecting renewables to the grid and emissions are dropping.” That midpoint of “they build but don’t connect” doesn’t feel like a hill I’d want to die on, and instead feels a lot like cope.

Can you give me some links supporting this? All the references I can find show that China is building transmission lines at about the rate they originally planned to.

They're going to operate a coal-backed renewable grid, while we (were up until recently) trying to build a natgas-backed renewable grid. They just have coal instead of natural gas, and they're actually building the renewables.

You don't want to be a few years out of date when making statements about China's electrical grid. Things are changing so rapidly that even being a year or two out of date is talking about the distant past. Most recent data is available from a good search LLM.