Hacker News Reader

Comment by nkurz | Hacker News Reader

Is there any reason to trust Pangram's tool over Carlyle's reputation? I don't know whether this was AI generated or not, but an online tool claiming it's AI generated doesn't sway my opinion much. Are there any studies showing that Pangram's tool is well calibrated for this type of article? If not, what makes you trust it?

(I'm not sure why it was marked that way, but I vouched to bring your comment back from auto-dead. I glanced at your comment history, but don't see any clear reason for this.)

View on HN

I only went to Pangram because I read through the essay and could not help but notice the Claude-like sentence structure and AI-isms in it, which distracted me and completely made me distrust the thesis of the piece.

My understanding is that Pangram is the best out of all of the AI detectors and if there is a better one I'm happy to switch to it. And it's easier to point to them than to give explicit sentences and examples about why it reads so AI-generated (and since you want it, it's these sentences in particular: "The message was unambiguous: energy is finite, security is earned, and comfort has a cost.", "That template is being applied again today — and markets have accepted it.", "One country never made the West’s mistake.", etc.)

Reading through it again, there are so many emdashes. And don't get me wrong, I was a liberal user of emdashes before AI! But just like, if your job is to communicate your thoughts to a wide audience at least respect their intelligence enough and not rely on crutches just to get an article out against a deadline.

To me these look like professional writing, and I've seen worse LLM'isms. I am with some people in that I don't usually want to read pieces where GenAI was involved. But this is one of the cases where I wouldn't want the remedy to be a life of constant purity tests and suspicion. There has been enough of that in contemporary culture. I want to trust people and let them be by default. I mean, I don't want writers to specifically memorize current LLM'isms in order to avoid them: or worse, to prompt an LLM to strip them. I believe from experience than GenAI texts have secondary bad characteristics, mainly being bland in style and thought, and often going nowhere, like mandatory student papers. This should be enough to bury or criticize them on merits more often than not.

I do see myself treating slightly broken grammar, typos as a slightly positive signal in writing in the recent years. I also see that this is messed up - I'd like to respect someone in kind if they put care in writing - and easy to fake if anybody wants to, with an LLM. If you do strongly suspect GenAI writing, I mean it's fair if a sincere opinion, but I'm tired of having those as top comments with a whole response tree. Ironically contributing to that now.

It’s either heavily AI written or the authors style has been heavily influenced by LLM writing. That said, using LLMs when writing does not mean the article is not informative, thought provoking, or relevant. An idea fleshed out in prose by an AI is not the same as AI generated content. But it is lazy writing.

How do we create an environment that incentivizes human writers to focus on their craft instead of “productivity”?

Thats maybe one of the more pressing issues of our time.

Yes there are studies, for example last year Pangram's false positives were measured to be under 0.5%.

https://www.pangram.com/blog/third-party-pangram-evals

Personally, at first I thought these sorts of tools were dumb and wouldn't really work, but I think it works because it just isn't designed to be "adversarial". If you want your AI to trick Pangram, you can make an AI to trick Pangram. It just catches people who are cutting and pasting from the AIs without putting any more effort into hiding it.

Any binary classifier can have a FPR under 0.5% if you don't have any restriction on FNR...

While I am quite skeptical of the claims linked above, that link does indeed cover the FNR at the FPR of 0.005, and finds it broadly to be on the same order of magnitude, i.e. also below 0.005.

If the FPR is very low, the FNR rate doesn't really matter if you get a positive result (unless the pre-test probability is very low, which is not the case here)

> Is there any reason to trust Pangram's tool over Carlyle's reputation?

You have your own chatbot, right? Ask it. I've never had one disagree with GPTZero yet.

Reply on news.ycombinator.com