It's clearest in the conclusion.
I feel like the “this is AI” crowd is getting ridiculous. Too perfect? Clearly AI. Too sloppy? That’s clearly AI too.
Rarely is there anything concrete that the person claiming AI can point to. It’s just “I can tell”. Same confident assurance that all the teachers trusting “AI detectors” have.
I have opened a wager r.e. detecting LLM/AI use in blogs: https://dkdc.dev/posts/llm-ai-blog-challenge/
These ai hunters are like the transvestigators who are certain they can always tell who’s trans.
Also, you do realize that writing is taught in an incredibly formulaic way? I can't speak to English as second language authors, but I imagine it doesn't make it easier.
Now, apparently, we have a generation of "this is AI slop!" "bots".
just one scenario, I write 100 rather short, very similar blog posts. run 50 through Claude Code with instructions “copy this file”. have fun distinguishing! of course that’s an extreme way to go about it, but I could use the AI more and end up at the same result trivially
obviously if a million dollars are on the line I’m going to do what I can to win. I’m just pointing out how that can be taken to the extreme, but again I can use the tools more in the spirit of the challenge and (very easily) end up with the same results
That's a laughable response.
this wager is a thought exercise to demonstrate that. want to wager $1,000,000 or think you’ll lose? if you’ll lose, why is it ok to go around writing “YoU uSeD aI” instead of actually assessing the quality of a post?
> I won't actually make this bet!
> But if I did make this bet, I would win!
???
see other comment though, the point is that assessing quality of content on whether AI was used is stupid (and getting really annoying)
If you could actually identify AI deterministically you would have a very profitable product.
I find it interesting that you believe this claim is wildly conspirational, or that you think the difficulty of reliably detecting AI generated text at scale is evidence that humans can't do pretty well at this much more limited task. Do you also find claims that AIs are frequently sycophantic in ways that humans are not, or that they will use phrases like "you're absolutely right!" far more than a human would unless prompted otherwise (which are the exact same type of narrow claim) similarly conspirational? i.e., is your assertion that people would have difficulty differentiating between a real human's response to a prompt and Claude's response to a prompt when there was no specific pre-prompt trying to control the writing style of the response?
> I find it interesting that you believe this claim is wildly conspirational
I don’t believe it’s wildly conspiratorial. I believe it’s foolishly conspiratorial. There’s some weird hubris in believing that you (and whatever group you identify as “us”) are able to deterministically identify AI text when experts can’t do it. If you could actually do it you’d probably sell it as a product.
I think you will find the OP said no such thing. They instead said they identified a mixture of writing styles consistent with a human author and an LLM. The OP says nothing about deterministically identifying LLMs, only that the style of specific sections is consistent with LLMs leading to the conclusion.
> Parts of it were 100% LLM written. Like it or not, people can recognize LLM-generated text pretty easily
I still think that's a far cry from deterministically recognizing LLM-generated text. At least the way I would understand that would be an algorithmic test with very low rates of both false positives and false negatives. Instead I understood the OP to be saying that people have an intuitive sense of LLM generated text with a relatively low false negative rate.
I am certain that the skill varies widely between individuals, but in principle there is no reason to suspect that with training humans could not become quite good at recognizing low effort (no attempt at altering style) LLM generated content from the major models. In principle it is no different than authorship analysis used in digital forensics, a field that shows fairly high accuracy under similar conditions.
I am making an even more limited claim than the article, which is only that it's possible for "experts" (i.e. people who frequently interact with LLMs as part of their day jobs) to identify AI generated text in long-form passages in a way that has very few false positives, not classify it perfectly. I've also introduced the caveat that this only applies to AI generated text that has received minimal or no prompting to "humanize" the writing style, not AI generated text in general.
If you would like to perform a higher-quality study with more recent models, feel free (it's only fair that I ask you to do an unreasonable amount of work here given that your argument appears to be that if I don't quit my lucrative programming job and go manually classify text for pennies on the dollar, it proves that it can't be done).
The reason this isn't offered as a service is because it makes no economic sense to do so using humans, not because it's impossible as you claim. This kind of "human" detection mechanism does not scale the way generation does. The cues that I rely on are also pretty easy to eliminate if you know someone is looking for them. This means that heuristics do not work reliably against someone actively trying to avoid human detection, or a human deliberately trying to sound like an LLM (I feel the need to reiterate this as many of the counterarguments to what I'm saying are to claims of this form).
> I’m not going to write another detailed explanation of why your “slop === AI” premise is flawed.
This isn't a claim that I made. I believe that text written with LLM assistance is not necessarily slop, and that slop is not necessarily AI generated. The only assertion I made regarding slop is that being written with LLM assistance with minimal prompting or editing is a strong predictor of slop, and that the heuristics I'm using (if present in large quantities) are a strong predictor of an article being written with LLM assistance with minimal prompting or editing. i.e. I, I am asserting that these kinds of heuristics work pretty well on articles generated by people who don't realize (or care) that there are LLM "tells" all over their work. The fact that many of the articles posted to HN are being accused of being LLM generated could certainly indicate that this is all just a massive witch hunt, but given the acknowledged popularity of ChatGPT among the general population and the fact that experts can pretty easily identify non-humanized articles, I think "a lot of people are using LLMs in the process of generating their blog posts, and some sizable fraction of those people didn't edit the output very much" is an equally compelling hypothesis.
This seems like the kind of thing to share when making a bold claim about being able to detect AI with high confidence. This is a lot more weighty than not so subtly asserting that I’m too dumb to recognize AI.
> a human deliberately trying to sound like an LLM (I feel the need to reiterate this as many of the counterarguments to what I'm saying are to claims of this form).
I assume this is a reference to me. To be clear, I was never referring to humans specifically attempting to sound like AI. I was saying that a lot of formulaic stuff people attribute to AI is simply following the same patterns humans started, and while it might be slop, it’s not necessarily AI slop. Hence the AITA rage bait example.