Hacker News Reader

> But for a lot of things I can with high probability discern between AI and human in every day appearance and I know many other people can as well.

It’s easy to say that when you already know the answer. But my point is people frequently misattribute stuff as being AI generated when it wasn’t. The GPs Twitter example is proof of this. And you only have to look at the number of times people accuse someone of being AI on HN because they happen to use em dash or an Oxford comma (or any other writing style that was part of the corpus models were trained on), and it’s pretty clear that “obvious tells” are just guesswork.

There was a study years ago where a group of individuals were given white wine with red food colouring, and they were asked to describe the wine. All the participants described plumb and other notes that are exclusive to red wines.

These weren’t dumb people. And the people who claim to spot “obvious AI” aren’t dumb either. It’s just very easy to identify tells if you believe you already know the answer. Unfortunately that’s just how fucked up our human psychology is.

> One of the things I look for is specifically certain types of extreme... not just flaws... just overall atrocity.

So, please forgive me, but this means that you are good at finding out some patterns in AI output. There are always a lot of false positive and negatives, and I haven’t found a convincing study showing that people are good at finding AI in general.

What helps is that the market is very concentrated so if you know GPT-isms and Claude-isms that’s basically the current situation covered. It does not mean that you can reliably identify AI in general.

How do you know you can discern AI vs human with high probability? You need some external source of truth to judge that, otherwise all you know is that you can discern bad, obvious AI. See also https://en.wiktionary.org/wiki/toupee_fallacy

I can say that my false positive rate is very low which is enough in this context.

How do you know that?

I trust Pangram's stats to be a good baseline.

Hey Claude, re word this until it passes pangram as not AI written.

As I explained in another comment the false negative rate is irrelevant in this context.

“I can tell because I judge that my judgement is correct.”

I’d expect false positives to be low. Bad AI has a distinctive style and is easy to pick up on. Just like bad toupees are easily recognized. The false negative rate would be far more interesting.

The false negative rate would be interesting but is irrelevant in this context. I was replying to a comment claiming that most people take issue with the ethics of GenAI rather than the quality and that people have a high false positive rate. I raise myself and my experience with people I know as a counter example.

There might be AI generated things in my every day life that I do not recognize as AI, but that's not what's being discussed.

It's not irrelevant in the context, you're just saying it doesn't interest you. Just because you chose to focus on one side of it, doesn't change the fact that you replied to a broader question of "How do you know you can discern AI vs human with high probability?"

And even if you had been right that the entire thread's context had been limited to just caring about false positives and not false negatives, it still wouldn't have been a problem for wat10000 to expand the discussion by talking about false negatives, too... they don't have a duty to only reply about the parts of a subject you chose to care about!

You could either just ignore the comment, or reply explaining why that side doesn't matter to you without trying to shut them down as if they did something wrong :)

Edit: Since you've mentioned Pangram a few times in other comments, FYI a lot of people (myself included) believe Pangram's accuracy & consistency to be bad enough that it shouldn't ever be treated as more than a gimmick.

How did you check? What was your oracle when you thought something was AI and when you did not?

I mean, this situation is a textbook case of ground for a lot of common fallacies, starting with confirmation bias. We need to remember that our brain wants to be tricked. If you think you are special, you are not. You’re just not aware of it and even more prone to falling into these traps.

They are pretty shit, but the main problem is that they are the generic style of the model. So that’s what everyone gets by default. If they were a one off they’d still be bad, but at least unique.