Hacker News Reader

A way to think of it is that any time the model faces a choice, it leaks some fractional bits of information. Instead of making the choice randomly, you can put information in those bits.

But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty. Exotic prompts ("Use no more than five E's in four consecutive words anywhere in the text") seem like they will confuse the hell out of attempts to extract the bits from the output text alone.

> But unless you know the prompt, you don't fully know which choice the model faces. Surely a lot of coding space is wasted compensating for that uncertainty.

When you only need to encode one bit, the signal to noise ratio can be very low. If I try to write my own human words under the policy of "try a little bit to avoid the letter 'e' in every fifth word," then a sufficiently long text would still be 'watermarked' even if I only succeed in this dictum (e.g.) 10% more often than the baseline.

Whereas if you fully avoid the letter e, everyone will know you are George Perec

Who is Gorg Prc?

And if you lowercase everything, you're e e cummings.

most people have at least two, but generally around four to six (or more, thanks internet) interactional styles (not "selves", just things which vary depending on context, subject, and people they are interacting with). probably more. some of it is due to simple physical comfort levels (right now I am in a physical position where capitalizing is more difficult); some might be due to talking to a peer group instead of a group of kids or a priest or boss at a job, etc. unsure how that will shake out with AI but it unnerves me.

There's more to it than encoding one bit. You also want to avoid false positives. You can encode a single bit by XORing all the bits in the UTF-8 encoding. But then you get a lot of incorrect hits. The lower your tolerance for false positives, the more it acts like you're actually requiring more bits in your payload.

In fact such ‘arbitrary’ constraints uniformly improve composition. Thus eg if I force a - largely arbitrary - technical glossary to be unrelentingly applied to a translation, every single sentence improves in quality.

Oulipo vindicated.

The scheme that Scott Aaronson describes essentially uses a specific prng, and you can then check a certain function with relatively few tokens to get a sense of whether or not a model using that scheme generated the text

A few caveats: you need to know the key to the function (used when generating the text) and you need to know the bias it would introduce

The point is that you do not need to know the full prefix, just a modest sample set of contiguous tokens

> The system is fragile, and small edits (or rewrites by a local model) can fool the detection.

Well no, small edits wouldn’t fool the detection as long as the seeding only uses a small run of previous tokens.

And yeah full rewrites breaking it is by design. The watermark is just meant to tell you whether the text was generated by a watermarked model, not whether the ideas came from AI or something like that.

In practice, it is theater. Are they going to do this with the code output too? This is nonsense security theater for the low thinkers to have a sense that someone is in charge. When we all know nobody is in charge, anywhere.

Their AI model tends to write a lot of lengthy comment blocks.

That's a fine place to put the watermark to track those users who accept the code blindly and don't delete/edit the comments.

The model doesn't particularly "know" which tokens within its output are "comments", it's not so easy

It knows when several tokens are about equal vs. times where one token is vastly preferred. In the latter case, that’s usually code or math or something similar and so it won’t alter those tokens.

EU regulations are going to force it, friend. The tsunami is coming and cannot be stopped and Anthropic has jack to do with it.

I wonder if that is entirely true. It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it ("Habsburg AI").

In the case of Anthropic it would also be entirely unsurprising if they've been lobbying the government to force everyone to do something in their (Anthropic's) own best interest.

> It is also in the AI companies' own best interest to be able to detect AI generated text so that they can avoid training on it

It’s also in their interest to demonstrate that they can be trusted and to show that they at least pay lip service to limit the obvious downsides of the tools they are selling. The use cases they sell to mainstream audiences are not affected by detection tools. The point of having a LLM do the work for you is that the work is done, and reliably. It does not matter if it is done by a LLM, and most of the time it is obvious anyway.

The EU mandate for AI watermarking is likely the first step in the direction of prohibiting AI for specific use cases. The pretext for outlawing (or at the very least controlling) the use of AI when the time comes will be something along the lines of data integrity or just general compliance legalese.

So, the use cases that are being sold to mainstream audiences actually will be affected by detection tools, especially if the output is intended to be monetized in some way. In the near future the EU will likely come down with heavy intervention to prevent AI from impacting employment rates across Europe. The number of legitimate use cases for costly frontier models drops significantly once eliminating professional jobs is off the table. This is all conjecture at this point though.

I’m sorry, this is going to be a bit long but you made good points.

> The EU mandate for AI watermarking is likely the first step in the direction of prohibiting AI for specific use cases.

I am not sure how practical that would be. The cat’s already out of the bag and they won’t prevent companies in the whole world from releasing open weight models. Playing catch up by distilling flagship models is also relatively cheap; we’d see smaller companies setting up shop in friendly regimes. And I don’t see any appetite to go full child porn and criminalise the possession of a LLM. So we’d end up with a similar situation as with illegal downloads, i.e., everyone will do it and nobody will care.

> The pretext for outlawing (or at the very least controlling) the use of AI when the time comes will be something along the lines of data integrity or just general compliance legalese.

They could forbid using LLM for hacking, but hacking is already illegal. They could make it a factor when determining punishment, but I don’t think that would work terribly well. Most of the dangerous stuff we can do with LLMs is already illegal, or should become so. Things like propaganda, identity theft, harassment, scams. We need enforcement with teeth on these, not pointless feel-good legislation. Again, there are parallels with cryptocurrencies and torrenting software. These things have illegal uses, but it’s also really difficult to make them illegal, at least in semi-functioning democracies.

> So, the use cases that are being sold to mainstream audiences actually will be affected by detection tools, especially if the output is intended to be monetized in some way.

I don’t know that mainstream audiences are really against LLMs. They are mostly against AI in a nebulous sense, but even non-technical people use ChatGPT or equivalent. I think that the critical mass is already there and the tools are convenient enough that they couldn’t outlaw them without a massive uproar.

Detection tools don’t seem all that relevant to mainstream audiences’ use of LLMs. AI companies will sell this as a safeguard against misuse, and everyone will be happy about it. The politicians will say they accomplished something, the AI companies will slowly turn public opinion, and the public will have shiny toys.

> The number of legitimate use cases for costly frontier models drops significantly once eliminating professional jobs is off the table.

I don’t know. They can open possibilities that we don’t necessarily consider.

One example I have is a friend who is getting his house refurbished. He’s not an engineer or a material scientist. He does not have enough free time to read thoroughly on the many subjects involved. With a decent LLM, he could untangle the technical documents sent by the architect and the contractors to really understand what was going on and be involved, rather than passively follow the architect’s advice. For starters, the LLM was very useful in finding issues in the quotes he received when he was looking for an architect. Those were long, technical documents, with no really standardised structure and full of jargon. I don’t think that person is going to want to stop using LLMs now. Many people are having this sort of moments right now.

> In the near future the EU will likely come down with heavy intervention to prevent AI from impacting employment rates across Europe.

Maybe. But i don’t believe the EU is well equipped for that. Labour laws are largely local and different in each member state. The EU regulations are basically the common denominator, ore or less, and it is easy to see why: for regulations to get adopted, they need a strong enough majority in the Commission, in the Parliament, and in the Council. It is very difficult to get anything controversial that affect the sovereignty of member states passed.

The angle of the current AI regulations is that they set the rules for the single market, which is where the EU is the most legitimate. It is difficult to see market angle for the effect of AI in labour, and I think enough member states would be keen to kill the project.

Also, there are many influences at play, but the EU is fundamentally an economically liberal institution. It very rarely goes in the direction that reduces economic activity. Look at how clumsy it is at fighting against cheap Chinese imports. I don’t think the institutions themselves would really want to make AI illegal. Companies have too much to lose.

Great points all around, lots to think about. It's totally possible that I am way off in my assessment of the future.

I think your comment about torrenting touches on an interestingly relevant case study, specifically media piracy. The seed of that technology was sown when the internet was still just clusters of machines passing files around and then exploded with the PC, Napster, and TPB. Napster tried to be a legitimate commercial enterprise with a business model of undermining the ability of copyright holders to rent-seek on the consumption of the material they 'owned'. At the core it was a novel technology (P2P) that revealed an economic arrangement to be out of date (if a distributor no longer has to manufacture a copy of the media for each individual consumer, their business model boils down to rent-seeking). Western legal systems were quick to rule on the matter (in favor of copyright holders), and I have no doubt that they were looking ahead to a future wherein media creation was totally disincentivized by said novel technology.

Now a novel technology (the cloud inference-backed LLM) is challenging another economic arrangement. This time around, the arrangement being challenged is the higher education->professional job pipeline. All advertising and media messaging aside, it really does seem like the frontier labs are only economically viable if they get massive enterprise deals across a broad spectrum of industry. There is fundamentally one chunk of capital organizations are going to spend either supporting their talent pipeline or padding it (to put it gently) with enterprise LLM deals. If the latter path is taken too far, consumer spending plummets (due to lack of middle-class incomes), assets backed by consumer debt/spending fail longterm, and we will have to deal with a deluge of socio-political issues stemming from the absence of real social mobility (we are in the early stages of this now, incidentally).

All this to say, I think parallel situations from the past can guide our thinking re: AI regulation and the forces shaping it. Reasoning based on regular political/economic incentive structures (e.g. economic liberalism not wanting to reduce economic activity) will fly out the window at lightspeed once fear becomes a factor. I personally think its great that LLMs empower individuals such as your friend to increase the control they have over real issues in their life; that is what technology should be doing for us. Cloud inference-backed LLMs are doing the opposite: drastically reducing the power that the everyman has over his socio-economic future by throwing high-paying career paths for a whirl and incentivizing powerful organizations to destabilize the labor market. Only time will tell how this plays out.

They say in the paper that they will not do this for code but it may happen for comments.

Does that also mean owner must re-run very model ever released by them to detect anything?

What happens if we train models (GPT or human students) using the outputs of a model with text havingbthose watermarks? Is there something preventing the watermark from being learnable?

no, and in fact they should be extremely easy to detect if they are substituting a token that is a least-probable synonym