Hacker News Reader

Comment by kgeist | Hacker News Reader

Interesting that the article itself gives AI-generated vibes:

>It's important to remember that while Wikipedia is "The Free Encyclopedia that Anyone Can Edit,"

(disclaimer: I'm a casual user of ChatGPT and haven't gone too much into the how it works)00

Given its nature as an LMM and a complex next word predictor, the phrase "it is important to remember" could be a way that was inadvertently trained so that it keeps itself on track.

If it has "some points, it is important to remember {something}, some more things" it may be able to better generate text compared to "some points, weird tangent".

Since it doesn't have a hidden memory, everything that it "thinks" is out there in the text including its own cues for what it should do. It also can't go back an edit its previous text to remove the self hints or clarify earlier points without calling them out. That style of writing differs from natural human writing since we are able to keep on topic (or not) without needing to write messages to ourselves that others can read.

When we do, it's pointed out rather than trying to slip it in casually. "Note to self or reader" for things that are to be pointed out and break the flow of the text or "as an aside" for the tangents.

Given that, would a chain of thought + final answer be a better output then?

How would you recognise AI-generated content? Is it a particular writing style? As a dyslexic person I am having a hard-time recognising it

Stating the thesis then (repeatedly) re-stating it with a (very) slightly different wording was a red flag. Professional journalists don't write like that, but AI and schoolkids trying to pad out their essays do.

I'm guessing that a human did edit this AI output, just not very well.

I think you’re giving too much credit to “professional” journalists. This is also an excellent way to hit a certain word count if you are a few sentences short. Basically you bloat it up without needing to write anything new. Listicles and other pre LLM blogspam did this all the time.

We're all facing the need to build a mental list of which news sources are publishing AI-generated garbage, so that we can disregard them.

I'm still surprised to see that, as far as I can tell, no news outlets have made a public commitment to never use AI in their writing. Seems like it would be an easy way to promote the brand on commitment to quality.

I've griped about this before, but here we still are. We now know that MSN news, for instance, has no credibility due to their publishing AI-generated misinformation. https://news.ycombinator.com/item?id=39043135

ChatGPT loves to add "it's important to remember" to every second answer. Also "multifaceted" and a few other adjectives like that.

objection -- ChatGPT does not "love" anything.. a reason to repeat that phrase is that it is written often, for real communication reasons. Declaring that something written often by humans is now an indication of ChatGPT is wrong-direction IMHO and directly objectionable here, a place where reasoning humans think and communicate via writing.

ChatGPT was additionally finetuned after the initial training of the base model, so it's not completely clear if it truly represents the original distribution anymore. I use open weights models and they repeat such phrases less frequently in my experience.

That's completely untrue. Try using any base (non finetuned model) such as llama base, mistral base etc and you'll see that it does not write like that. That response format and writing style was added by OpenAI during the finetuning stage.

the post says "a reason to repeat that phrase"

not THE reason, not THE ONLY reason, so it is not COMPLETELY untrue. agitated?

I can visualize a little cartoon Einstein scribbling away on his chalkboard, but everything I'd actually project on the chalkboard would be nonsense. Relying on my cartoon output or thinking it has value would be madness.

Someone who's studied the subject quite a bit might visualize an actual Einstein chalkboard from memory. Their output could be verified and referenced, but looking for new Einstein-level math on the chalkboard would be madness.

Someone who Einstein himself would consider a peer might use this visualization method as a way to do actual work.

If we're assigning value to our chalkboards we'd be able to explain why we chose the numbers -1, 0, and 1. This would bias any math we'd do towards one of the chalkboards depending on our intent. At this point my chalkboard is useful as a filter.

Putting this all together, we'd see that the overall look of the cartoon responses would be a blend of 0 and 1 styles and would depend on our requests e.g. a reference request would look mostly like 0's art style. My own personal art style will be intentionally absent because it's only ever framing nonsense, by my own admission.

long before CNET was admitting to this it was obvious to me. The writing has the vocabulary and grammar of a accomplished academic but no ability to actually convey the important points concisely.

An easy tell is "level of effort given context"

Enjoy for now that you can still tell. It’s early times like the video games before Call of Duty etc.

The worst thing about AI is how it can easily betray you, manipulate you and have swarms execute long term sleeper plans at scale!

Sounds like all information we're given should be met with a healthy dose of skepticism. The ability to smell AI makes it a little easier right now to put it in the garbage pile.

We are moving toward a future where future robotics viruses will infect deployed humanoids and command them to go on a killing spree.

https://www.youtube.com/watch?v=WWAnJX889j0

Eh, ChatGPT uses phrases like "it's important to remember" because they were common in the training data. The rest of the sentence definitely isn't ChatGPT:

> It's important to remember that while Wikipedia is "The Free Encyclopedia that Anyone Can Edit," it's hardly The Wild West.

My reading is that the article is average human prose (not great, not unreadable), not LLM prose.

I wonder if ChatGPT was trained on too many high school essays. It’s not very good writing style, but it’s what most people start out writing.

>My reading is that the article is average human prose (not great, not unreadable), not LLM prose.

It's not average human prose either(for one thing expecting GPT to converge on some average doesn't really make sense in the first place).

Base models with no rlhf or fine-tuning don't talk like that at all. This is specifically an artifact of the post-training fine-tuning/RLHF process

I never said that I expected ChatGPT to converge on an average, I said that this particular phrase is not strong evidence by itself of GPT authorship.

"It's important to remember" is a phrase that plenty of people used before ChatGPT. I've included three examples below from a quick time-boxed search. Just because ChatGPT says a phrase doesn't mean that every time you see it it came from ChatGPT—someone wrote the training data that ChatGPT was trained on, and someone else wrote the data (or selected the responses) that it was fine tuned on. ChatGPT isn't inventing new phrases out of whole cloth, it has a stereotyped style that is pieced together out of many existing phrases.

A collection of such stereotyped phrases in a single piece would be stronger evidence of GPT authorship, but I see no evidence of that here.

https://old.reddit.com/r/gravityfalls/comments/bgtnez/while_...

https://twitter.com/AsteadWH/status/1050813462673264640

https://stackoverflow.blog/2019/12/19/what-senior-developers...

I think I misunderstood you then. Fair Enough.

> My reading is that the article is average human prose.

Which is, of course, exactly what ChatGPT is trained to produce. A lot of people's mental AI detector is actually a mediocrity detector.

No GPT is trained to spit out anything that falls into its training data distribution.

There's no average. Pre-training incentives being able to predict the smartest string of text in the corpus as readily as the dumbest. It doesn't converge on "average" and it doesn't really make sense that it would either.

Base models don't talk like GPT. This is strictly an artifact of post training fine-tuning/RLHF.

My theory is that this phrase was baked in during alignment training to ensure the model prints out any necessary warnings.

"It's important to remember to consult a mechanic" for example.

It's just some priming.

If I play GM-level chess moves from Rybka or AlphaZero randomly on average every 5 moves on average, could you tell?

Every 5 moves, yes.

But as most Chess Grandmasters say, you only really need to use it in one difficult spot to change the result of a game.

Nah. I play the top engine move like half the time and I'm not a particularly strong player.

Both can be true at the same time. A lot of the time the best move is obvious. However, if you play the top move 20% of the time, you will also play the top move in a lot of cases where it's not obvious. Given enough games, it's detectable.

Reply on news.ycombinator.com