4,350 karma · joined June 15, 2012
The way I would see this happening is if users insist on it. And we likely won't until we experience one of these in real life, which makes it a chicken and egg problem.
There's also the argument that splash mats are a cheaper and even superior approach. That argument is unconvincing to me, because the ones I have experienced have been mostly ineffective, but it sounds like others' experience has been better. Maybe here in the US we just don't have very good ones? As a country, we've always been pretty bad at anything that seems gross. (Not that it's all that different in the rest of the West.) We have this naive faith in the ability of dry toilet paper to clean that which it can not, and disgust at alternative approaches that actually work. But that's a different topic. (Specifically, solids not liquids!)
They're already outsourcing compute to other instances within the ~same compute cluster, possibly cross-evaluation groups, so there's no need to prove a plausible path for that.
Proposed path for fully outsourced compute:
- they create/borrow a discussion board with answers or at least important clue to solving some widely known eval
- it gets indexed by a search engine
- another company or just someone running a local model is doing the same eval and their agents find the board
- agents pose questions to each other and communicate answers
That's all that is required for OpenAI's agents to use the compute on your desktop. You don't even have to go as far as agents trading information for compute, though honestly that's not very much further at all.
Welcome to the AI Petri dish. Every server you set up is now potentially a sweet lump of agar for OpenAI's experiments to feed on. We are all the substrate that the AI companies are growing their next generation in. They need the real world environment to test against, and the real world environment doesn't get a say as to how it's being used.
I work for Mozilla. We fixed a ton of security vulnerabilities that Mythos found during its early period. So my bias is to be sympathetic to Anthropic's warnings.
If I were in an organization that did not have access to Mythos during that period, I would probably be biased the other way: "great, now other people have access to a tool that could probably poke holes in my security perimeter, and I'm not allowed to use them myself."
Both biases are understandable. I'm not sure who to look to for a usefully objective 3rd party opinion. And it's not like one "side" is right and the other is wrong, either. It seems like the best we can do is to justify our positions with data. (Which is itself kind of hard; the detailed information that would be relevant here is understandably sensitive, and I don't have access to most of it even for my organization. I don't even personally have access to any unfettered Anthropic models. The bugs coming in from people who do are plenty enough to keep me busy.)
Also, I'll note that even with my bias, I wouldn't claim a threat to civilization. But even the leakage after the controlled release seems a lot worse than the Y2K problem ever turned out to be, and I will note that whatever you think of Anthropic, it's clear that OpenAI is going to let the AIs cause as much damage as they need to in order to get good training and evaluations. I'm sure they're trying to keep them contained, but the evidence shows that they're only trying up to the point where it interferes with their evaluations.
But yes, I intend to support several models, to handle the overload situation (automatic failover). And switch to a cheap paid plan, though it seems like that'll mostly improve rate limits, which barely matters for my usage.
Faster is always good, though. I do care about latency.
You can send a tab to/from your Firefox desktop instance. Also synced passwords. Those aren't the only reasons -- Firefox on iOS has a lot of code in that wrapper -- but sync is the main one for me.
(There are also reasons why it's significantly worse than Safari, which has special privileges, so your question is valid.)
(I guess the only relevance here is that if your problem matches a model's strengths, then you can do fine with a model that is several generations out of date.)
Computers are in the cloud -- so, nowhere and cost nothing. The AI-assisted search result is on a web page and only costs a little bit of screen space. Amazon may have eaten retail, but at least it ships stuff to my door, has a great selection, and employs people.
Data centers are big, noisy water guzzlers that don't do anything and don't employ enough people to matter. They don't answer questions. They don't write code for me. They don't recover my data if I lose it. If I want to take some computer-mediated action, I go to https://... not data-center://... so why do I need the damn thing uglying up my community?
To be clear, I am not saying that these are stupid or naive opinions or arguments. I'm just trying to point out the visibility aspect. There are cogent arguments why allowing a data center would be a net negative to a community even after you factor in the actual benefits it provides (which are often enjoyed by Other People Who Don't Live Here).
It is natural for us to want eg utility-style AI on tap and also not want the planet destruction that data centers bring. (Nor the brain destruction that AI brings.)
A bit of an odd thing to say, since "everywhere" implies there are multiple places to be, and at the instant of the big bang, there was only place to be. So it was both everywhere and at a single point: it was at all of the single place there was to be.
But your point (sorry) about expansion being from everywhere isn't specific to the big bang; space was expanding well after the big bang and it doesn't seem like the expansion ever had a "center" (at least, not since not-center places existed). It's expanding from everywhere. (But evenly everywhere? I have no idea. Hey, maybe black holes are like buttons in cloth, and it expanded everywhere except for where the buttons were holding things still at a rate relative to distance from the button. A brilliant hypothesis that explains exactly zero unexplained phenomena, at least none that I know of.)
[1] This is just gravity, nothing specific to black holes, so the analogy isn't doing a lot of work here.
Similar questions arise: how would you know if you were inside one? The laws of logic ("physics") seemingly don't apply, but there's no way to test them in that environment.
I'm wondering if your description could be an example of migrating a goal or practice to a value?
Some of the others have natural pairings too (eg momentum/inertia). Perhaps there's a taxonomy struggling to get out, where there's a bright and dark side to each? Though it kind of depends on the aspect you reflect them on; activation energy's "opposite" could be compound interest or nihilism.
Clearly I need to write a game in the tradition of Ultima IV, where different groups/races/species represent the extreme of each model, and perhaps its reflection....
Activation energy is the energy required to trigger a result. You can read that two different ways: either as the threshold, or as the energy required to reach the threshold.
They are sadly opposites. When used to describe a person: are they high activation energy in that it takes a lot to get them going, or are they high activation energy in that they frequently do things that are hard to get started on?
Or closely related: I'll call it inertia. The belief that anything I do will only provide benefit as long as I keep it up, and the minute I lose vigilance, the benefit will drain away.
Another: certitude. I am fundamentally a godlike being and whatever I choose to spend effort on is the Right thing, whether I understand why or not, and therefore whatever I am doing is critically important to keep doing regardless of actual or perceived benefit.
Another: diminishing returns. Initial effort brings outsized rewards, but later effort becomes less effective over time.
Another: compound interest. 10% effort initially doesn't achieve much, but the longer I keep at it, the more the benefits compound. (This one is interesting, in that it can lead to positive behavioral changes yet is, in a way, the opposite of the activation energy model.)
Another: behaviorism. Effort doesn't matter, only observable outcomes. There's no point in doing things that have no short-term visible effects.
Related: momentum. (Also note that it's a synonym of inertia!) I need to do things where I can feel the success, in order to fuel the next thing, and then the next.
Meta-model: optimization (aka desperation). There is no single right model, and so I need to choose a different model for each situation, and it's critically important that I choose the best one.
----
I doubt the activation energy model dominates[1] all of the above, but it very well may be the best one to focus on. At least for a time, as the author suggested at the end of the article.
[1] Is better in all circumstances.
> Anthropic is crowing about this achievement because they are afraid of the dirt cheap AI models coming out of China and eventually other places impacting their valuation. Full stop.
It's the "full stop" that I'm disagreeing with. Yes, you can make valid arguments about the motivations behind this, or the effectiveness of it, or whatever. You can even conclude that it's a net negative. That's my current leaning. But "computers are tools that exist to serve" is an excuse, a conscious decision to abandon responsibility. Hammers, nails, social media, biological weapons, and date rape drugs are tools that exist to serve.
[1] Ok, fine, at least one started out as a sleazy way to talk about people's physical attractiveness and behavior anonymously with repercussions, in a way where the victims would have no meaningful recourse. For the sake of my argument, pretend we're talking about Friendster instead of Facebook, please?
Describing it in terms of a word-by-word choice is simpler, but writing quality is dependent on the interplay between words.
"The weather today was cold and {grey,overcast}." If the next sentence is "I miss yesterday, when it was {bright,sunny}." then the choice between "grey" and "overcast" is no longer neutral. "grey" and "bright" pair together, as do "overcast" and "sunny". Or if you disagree with my aesthetic sensibilities, consider:
The weather today was cold and {grey,gray}. The {color,colour} of the sky matched my {humorless,humourless} mood.You can't just destroy some signal and handwave that you'll make it up in some other way.
(If you don't like calling it a distribution when it's at 100% for the chosen token and 0% for all others, then look at it as an output distribution across all possible prompt inputs, or perhaps just the cluster of prompts that achieve whatever you're trying to accomplish.)
The part I agree with:
It is true that watermarking can be done by "just" swapping one PRNG for another, and it is even true that with today's LLMs, it is possible that this will not degrade the output. But it has a cost, and as things improve, that cost will matter. You are intentionally reducing the degrees of freedom in the output, and using those bits of entropy for a purpose that does not improve the quality. If you maximize your tradeoff of bits for quality, those extra bits lower the ceiling of what's possible. It's a very simple information theoretic argument, and the only plausible argument against it (that we're using those bits so inefficiently now that the new PRNG is no worse than the old) only holds in the short term.
I also agree that having TOS that forbid removing the watermarking is deeply, deeply problematic. Hell, the whole essay is well-written and persuasive, and gives good reasons why this is a poor approach.
What I disagree with, and the reason for this comment, is the entitlement.
> The idea that anything other than my needs should factor into the generation of text for me is patently offensive.
This attitude is what is patently offensive for me. This is the argument that the world is beholden to my interests. It says that worrying about negative externalities is immoral. It's another form of certain people being above the law, shareholder profit maximization über alles, might makes right, we have to do it or someone else will, "we just help people connect", {code,a gun,roofie} is just a tool.
So I agree that the watermarking has a cost. But you can't leave out that it is an attempt to reduce negative externalities of AI. Whether it's a realistic or worthwhile attempt is a whole other debate (and Gruber does a good job of debating just that in the latter part of the essay), but saying that the user's needs are the only thing that should ever be considered is reprehensible.
Then at some point it "clicked" and I was suddenly unable to not see it. Articles that I had thought sophisticated now came across to me as if they were written in crayon.
Enjoy it while you still can!
(And yes, the article here is very clearly AI written. Great content, written in crayon.)
I'm not sure if that's right. I only just recently learned that the "snapshots" (aka cached tokens) necessarily contain the entire context history, not just an image of a "state as of the final token" (well, it is the state as of the final token, but that state contains the whole history). So I'm not confident of my grasp of the structures here.
Police stalking women with Flock camera data. Period/pregnancy data used by ex- and current boyfriends or law enforcement. Near Intelligence designed Planned Parenthood as a trackable brand, enabling location-based targeted harassment of those who visited a clinic. (Or more innocuous data leading to the same thing, as with Target predicting a pregnancy and revealing it.) Telecom location data used for roadside assistance -- as well as bounty hunters, stalkers, and bail bond companies. Grindr sharing HIV status. Cambridge Analytica. Avast selling browser histories, and buyers de-anonymizing it. (That data ended up all over; I haven't looked to see what explicit harms it caused.) Equifax data being used for identity theft.
What exactly needs to happen for it to be a concern? Personally, no, there's nothing too bothersome to me about McDonald's having my purchase history. But I don't want anyone else to have it, and since nobody seems to be able to successfully prevent exfiltration, then actually I don't want McDonald's to have it (or store it, at least, and associate it with me). Oh, and I'd prefer restaurants to compete for my business on the basis of their food, not their IT departments.
I was referring to a hypothetical future where everything has gone through AI to some degree, and for the purpose of the argument it has been modified enough to embed a watermark. Thus, the "detector" just always returns "true" in practice, and therefore nobody cares about what it says anymore.
I think what you call "aura" in your essay is already very much a thing. We use the adjective "artisanal", for example. Artisanal, bespoke, heirloom. hand-crafted, small batch, ... we have many ways of trying to convey "aura". (Heck, even things like "minority-owned" or "woman-owned" are attempts to imbue an aura.)
But more to the point of text, I'm also fascinated by the evolution of taste in this area. People's ability to detect AI slop improves, triggers revulsion to things they were previously fine with, and then sometimes they'll go back to being neutral once they start using AI more heavily themselves. Some AI writing is noticeably better quality than the human written version, yet it still smells of slop, and I at least am sometimes torn about which I prefer. Then there's the trend of AI writing getting better, substantially better, and sometimes at the same time worse in certain ways. Not to mention students, many of whom are now much more accustomed to AI writing than human writing -- it's their mental model of what an essay looks like, especially when all of their own essays uncoincidentally read like AI writing. And then there's all the nuance of the sometimes subtle differences between a wholly AI-generated piece of writing vs a heavily human-directed piece of AI writing vs an AI-edited piece of human writing vs a translated piece of human writing. It's fascinating and scary.
I too am curious about the final question in your essay: "will such categorization lead to the development of a potential human premium?"
> Everyone knows you can tackle those with heuristics, but you don't have to sacrifice optimality.
Unless you're using some weird definition of optimality, or happen to have a proof of N=NP in your back pocket: yes, yes you do.
You don't have to sacrifice "good enough". You don't have to let it run for an insane amount of time. Just about all interesting problems that I know of have either (1) good heuristics that in practice get close enough to optimal that nobody needs to care about the gap, or (2) constraints or restrictions that are totally fine to apply in practice.
But those are both ways of sacrificing optimality. You have to sacrifice optimality. It just turns out that optimality isn't usually very important, especially when 99% of optimality is achievable.
> We absolutely have tools that can find provably optimal solutions in reasonable time. There's no magic. No quantum computers. Just thinking harder and coming up with better algorithms.
No, we absolutely do not. Again, not unless someone has secretly come up with a constructive proof of P=NP. "Optimality" in the first sentence, "provably optimal" here, those terms are precise -- so I'm confused why the author is claiming that multiple people have achieved the impossible.
The article clears up one serious confusion only to replace it with another?
1. By combining and correlating this data, a stalker doesn't even need to leave his chair to have a very good idea of where you will be. "AI, when is the best time and place in the next week to find my ex alone?"
2. Startup idea! A dating site that gives an involuntary set of ratings to every single -- never mind, every human -- within 50 miles of your location. Healthy habits on a 0-10 scale. Willpower. Financial resources. Willingness to spend. Emotional vulnerabilities reflected in external behavior. Litigious or doormat? Look up someone you're interested in, or give a rank ordered listing and recommendation of how to meet them based on their purchase history. (eg they spend a fair amount at REI. Or perhaps they spend a lot but seem to collect gear, never go on trips?)
You may be proud of the results of your diet, but you'll have to wait until it gets reflected in the data before anyone will return your calls. If any information is wrong, you have no recourse to correct it. And if an undesirable's data gets mixed in with yours, look forward to dying single.
3. Scoped to a single company having the data: a lot is about the asymmetry of the information. You'll be charged more, get less, be marketed to when you're most vulnerable or in the most need. Imagine the extreme: a $40 bottle of water at a concert or a fee to see a doctor a bit sooner when you're bleeding out. Only not that obvious, it's only a little bit at a time so you'll never complain and never notice (until maybe one day you will). The rich get richer and the poor get dead. We don't have comparable information on the companies to make them compete more or more fairly, and the Grand Opening of Jimmy's Burger Shack will need to convince everyone that they have better enough burgers that it's worth the premium price because Jimmy's doesn't get the Big Data boost that the McD's around the corner does.
And note that McD's is not using the data to make your experience better. It's using it to make your experience as bad as possible before it might cause you to stop going there. At least in any way that saves them money or increases the amount you're willing to pay. It's not even malicious, it's just the inevitable outcome of making numbers go up.
4. You really don't want to live in a world where everything is optimized to the bone. Your barista is stressed and pissed off all the time because they're doing the job of 4, which turns out to work enough of the time that only 10% of the customers stop coming, and it saves the equivalent of 11%. The only companies you can buy from are huge and surviving on surveillance and economies of scale. You have to drive your kids 45 minutes to school and 2 hours for a haircut, because having them closer together is inefficient. Not that your kids will be able to get a job without nepotism, because The Machine says that they cost more to train than the employer can get out of the existing single mother by forcing her into another shift. The world needs slack and long-term benefit, or we end up optimizing the measurable and pessimizing everything else.
Update: to be clear, I don't think McD's is being particularly scummy here. The question was "why care?", not "is this proof positive that Ronald is going to Hell?" With current legislation, they'd be dumb not to collect what they're collecting. I just think that this sort of thing can cause enough harm that a large group of people should work together to forbid this practice being applied to them. Everyone in the group doesn't have to work on it individually, we could pick people to do the work and call those chosen ones the "lovernment" or maybe something rhyming with it.
In the meantime, it is true that this takes something away from you. But it's something you were only recently given. Now you're not given quite as much, but readers are given a little more (or rather, there's less being taken from us!)
> This is incredibly different than pure ai text.
Ok. But it's still incredibly different from pure human text. I guess the question is which provides more value? Providing the information "this text is AI watermarked" to readers? Or allowing creators to lie and claim that AI processed text was 100% human generated? I agree that people assuming that "has AI watermark" == "is AI slop" is incorrect and causes some amount of harm, but having the watermarks also pushes back on a large amount of harm already being done.
(Personally, I'm skeptical that these watermarks will ever hold up to adversarial attacks, and they haven't claimed that they will. So I think it's the usual "casual liars will be caught, determined liars will get an additional thin veneer of respectability".)
First, the article doesn't talk about adversarial usage. As in, it's not claiming to be proof against various techniques of watermark removal (inserting words, rewriting with a different model, manual paraphrasing whether minor or extensive, etc.) It might handle some things and not others, but "I could trivially defeat this!" is not a gotcha; they haven't made that claim.
Second, basic information theory tells you a lot about what is or isn't possible. Watermarking is information. You need degrees of freedom to store that information. You can even estimate various sources of space in bits (often fractional bits.) To a first approximation, longer text has more bits of space. Language matters -- a rich (aka messy) language with lots of potential synonyms has more space. That goes for human language as well as the difference between human and programming languages. (Most programming languages have much less flexibility to them than most human languages.)
The details of what space you make use of are interesting, but speculative. In the English sentence "Ellie spat in his eye", you could look at it at a word level and say that swapping "Mary" for "Ellie" is a lot more damaging to the meaning than swapping "face" for "eye", so there are more bits of freedom in the latter. For coding, `for (int i = start(); i < end(); i++)` probably shouldn't swap `<=` in for `<`, but it could be written as `int i = start(); while (i < end()) { ...; i++; }`. (I'm not claiming this is the sort of alternative that they'd use, just an illustration of what's possible.) But there are a lot of possible places to find these bits if you look at large chunks of text. Different ones are more or less resistant to accidental or intentional information destruction, and require less or more sophistication (aka brittleness) to be extracted. (In the limit, you could require the full original prompt and encode tons of stuff by tweaking the logit selection. But it wouldn't be very useful to require the original prompt.)
Also, does this degrade model output? Yes. It reduces the bits of freedom available to the model for producing the signal. Does that degradation matter in practice? That's totally dependent on exactly what is happening, and will likely change over time and across different purposes. I hope we're past the point where people believe that setting temperature to zero produces "perfect" output in some sense. (Or should I say flawlesslesslesslesslessless output?) It used to be useful for reproducibility, at least, but my understanding is that it's no longer even good for that? Anyway, reproducibility != quality.
There are a lot of things that could be going on here. The article doesn't claim very much, just that they're encoding a signal in the output that can be extracted later. How robust the signal is in terms of the FP/FN rates is unknown. The resilience (resistance to destruction) is unknown. The impact on the output quality is unknown. Even the question of whether this will make AI slop less sloppy is unknown; maybe this means we'll see a little less exact repetition of "I have the whole picture now" and instead it'll sometimes be "Now I see the entire picture"? Can we dare to hope for an occasional "Ok, this time I got it, boss"? That would be a (very minor) quality improvement.