Hacker News Reader

Comment by jedberg | Hacker News Reader

Have you tried putting known human writing into pangram? I have. I've gotten 100% AI with multiple samples of my own human writing. It has also given me 50% on things I know were 100% AI written (from my prompts).

Pangram and everything like it is useless. The results are random on known samples.

View on HN

Pangram specifically (as opposed to most other detectors) publish internal audits, and seem to welcome external audits [0]. I'm not saying that you are necessarily wrong, just that in my opinion they have earned a higher bar of criticism than random one off anecdote.

[0] https://xcancel.com/JohnHolbein1/status/2059648132250570975#...

That's a fair criticism, I certainly didn't run a full benchmark. Just a few of my own pieces of writing. I also did it a few months ago, maybe it's gotten better since.

I wanted to test the service by having it check a few paragraphs from a known writer that lived before "AI".

I pasted the text into their form, pressed check... and only then they told me they want me to sign up first.

Good riddance.

GP is right. This paragraph is a major tell. If you have read enough ChatGPT output that hasn’t been humanized, you start to instantly notice it:

“ The attacker hadn’t broken into anything. They’d just noticed something I hadn’t: I had a verified email-sending domain attached to open, unverified signup, and that’s a useful primitive if you don’t care what you send.”

Yes, instead he should have written: "The attacka didn broke into nuffin bruv. They jus noticing sometin I aint, yeh?"

Instant betterness.

I would rather read an email the sender actually wrote even if it looked like your example, as opposed to AI-written. In that sense it is "better" to me.

Well I wouldn't. The whole point of communication is that something is adequately communicated, otherwise it makes for an extremely frustrating experience.

And the whole point about "slop" is exactly that the communication is not adequate (despite superficial eloquence), because it's "sloppy" thinking full of unnecessary verbiage that goes nowhere and disrespects the reader, making for an extremely frustrating communication experience.

But if AI (re-)phrasing helps communicate a message clearly then it's not slop. It's just AI assisted. Being able to spot common AI patterns isn't proof of slop, no-matter how sensitive to slop one might be.

And I'm saying this as a person who's VERY much against slop.

That's interesting! I have tried to get false positives from pangram and failed, so I trusted it a bit more than any of the others, although I generally just rely on my own intuition. I am curious what your false positive samples looked like, if you're willing to share.

(I'm less interested in false negatives; I have successfully produced those myself.)

I'll try to pull them up for you, I'd have to go back and find them on my computer.

It's obvious that existence of AI will hugely dissatisfy people and make them demand authenticity. Market dictates that there will be a satisfaction of that demand even if it is a smoke show.

Anyone with half a brain knows that people can write like that too. Sometimes people reinforce their point. Sometimes they even do it three times.

The problem is not in posts that fail to pass some half imaginary authenticity check. The problem is in technology that makes us want those checks.

Reply on news.ycombinator.com