Hacker News Reader

Comment by itronitron | Hacker News Reader

I'm not sure whether it's SEO or AI that is to blame but in the past few years it seems like almost every article on the internet repeats itself in some form at least three times. Most blog posts about a topic feel like they were written as a High School English project. It's actually quite astonishing how bad the internet has become at conveying information compared to what it could be doing.

So I'm tossing around an idea in my head, of implementing a search engine that computes a 'redundancy penalty' on every web page it indexed. The measure of redundancy would need to be multi-scale and consider repeated phrases, points made, and paragraphs as well, so not necessarily a simple problem.

If you think this is something worth pursuing just let me know by commenting below.

View on HN

Three thoughts:

First, effective writing will almost always restate the main idea somehow; you'll want your algorithm to be more sophisticated than "demote content that seems to repeat itself."

Second, your novel algorithm will be expensive to compute, so you'll need some other signals to filter out very low value pages before you pay to analyze others.

Finally, your technique will be valuable as a ranking signal until you (or someone else) begins to use your algorithm to help steer non trivial purchasing decisions. At that point, someone will dress up their low-value (or negative value) content to appear "high value" to your algorithm and your output will be filled with spam. You can delay this effect by keeping your user-base small, so consider selling private access to institutions who want a unique view of the internet for research or intelligence purposes.

Good luck!

> someone will dress up their low-value content to appear "high value" to your algorithm

You are falling into the trap of thinking that Google and its competitors are victims of this. They are not.

They are knowingly letting themselves get "exploited" because it turns out that those spam sites contain ads (that may be Google's) or analytics (that may be Google's), or make the search result page ads (from Google) look good in comparison and more likely to be clicked.

The heuristic to detect and block spam in general is very simple. Spam is there to get you to either buy something (the signal is the presence of a credit card form, "buy" call to action, or links that ultimately lead to the former) or view ads (signal there is presence of ads).

This would be very hard to game because those signals only work when they are visible (and obnoxious), which also makes them trivially detectable purely at the visual level by a crude classifier (trained on website screenshots with ads vs the same page with an ad blocker).

The resulting penalty score can be used as a ranking signal, so that all other signals being equal, a result without ads would rank higher than the one with ads for a given query.

This is not rocket science. The problem is that the mainstream search engines have the same business model as the spammers (and profit off each other), so there is no reason to suddenly slaughter the cash cow. The myth that search engines are "victims" to spammers/SEO provides very convenient plausible deniability so they have no reason to disprove that either.

This scoring is awful. Literally what you have done here is incentivise sites from well funded bad actors, or sites that push more elaborate scams on elderly people.

A plausible and sustainable funding model is a good sign for sites.

It's about incentivizing sites to do what the user wants them to do. As a user, 99% of the time I am not out to buy something, so all sites that do try to sell me something are just wasting both my time and their server resources.

This isn't a complete blanket ban on ads or the commercial web. If there isn't a profitable way to run a website that provides the content matching the search query, the ad-infested website will still come up first.

But now, a hobbyist, non-profit or even commercial enterprise that doesn't directly sell to you has a chance to outrank the ad-infested garbage.

> someone will dress up their low-value (or negative value) content to appear "high value" to your algorithm and your output will be filled with spam

That's where we are right now. I'm not sure that this is a solvable problem.

First, effective writing will almost always restate the main idea somehow

Effective corporate writing.

This goes back to the era 50 years ago when public speakers were taught "Tell me what you're going to tell me, then tell me what you're telling me, then tell me what you just told me."

It works fine for certain long-form speeches, but as the length of the document/talk decreases, so does the need for repetition.

For the length of 90% of what's on the internet, repetition becomes redundancy and nuisance filler.

Not just length, but interactivity too. That suggestion makes sense for a long monologue that is ephemeral, but in the modern age where the content consumer is in the driver's seat and very little is actually "live", they can scroll/seek back, rereading or relistening to earlier portion, and review the summary and comments to regain context.

Your message above contains a quote, and four sentences.

Sentence 1: hints that the quote is wrong, because only corporate writing is repetitive.

Sentence 2: expands on this: corporate speech has been repetitive on purpose since the 70s.

Sentence 3: contrasts corporate writing, which is repetitive, with shorter content, which does not need to be repetitive.

Sentence 4: concludes that the quote above is wrong, because most Internet content (which is not corporate writing) does not need to be repetitive.

Initially snarky/amusing but the post you critique is a well written paragraph in which there is a clear progression and elaboration on one major idea.

I do not think this is fair evaluation.

Sentence 1: specified that quote is about one specific kind of speech.

Sentence 2: explains origins of the quote - history. It is the history part that makes this sentence valuable and adds something to the reader (me).

Sentence 3: states that this is not effective to the kind of content we talk about - internet articles.

Sentence 4: concludes that repetition is irrelevant and wrong.

-----------------------

The way you tl;dr it, the poster would need to completely change the topic to "not repeat themselves". What they did was saying something new with each sentence, just keeping the "repetition" topic on.

Ok that's fair. Your summary is more accurate and gets the GP's point across better.

Thanks

> First, effective writing will almost always restate the main idea somehow; you'll want your algorithm to be more sophisticated than "demote content that seems to repeat itself."

You very much not want it repeated the way it repeats now. Effective writing does not involve writing three similar paragraphs, it makes readers skip those paragraphs.

I’ve read some books where there where some major developments/ twists that were not clearly stated and not repeated. I had to go back and reread paragraphs to understand. If you read the Bible as a fantasy novel you will come across this several times

I would argue that the "were not clearly stated" is the root cause. There is also difference between repeating some fact a reader could have forgotten and repeating whole paragraphs of fluff.

The internet articles we complain about are in the latter category.

> almost every article on the internet repeats itself in some form at least three times

Even videos like YouTube shorts, which are meant to be short, have this problem. They are seriously padded with repetition. Here's thing, have you seen this before? Few people have seen this. Do you know how it works? I'm going to show you. People are surprised when they find out how it works. But I explain it.

There must not be stats on choosing "never suggest this channel again", I select that pretty consistently when I'm looking for short form content and these videos waste all that time.

I used to write blog and stopped as google algoritms changed. Basically, if you do not do SEO with artificial constructions, you was as good as not existing. I could not find my own post in search even when googling the exact topic.

I suspect that not coincidentally, blogosphere as used to be got much much smaller. Writing is not rewarding when no one finds it, so people just dont do it.

I think that AI will make it even worst. When everyone fires ChatGPT to get compilation of stuff, there is even less reason to write. So you dont.

> seems like almost every article on the internet repeats itself in some form at least three times

Spaced repetition, apparently it 'aids retention'. It's the same reason TV ads now run with a second shorter ad for the same thing later in the ad break.

Maybe the solution is to have LLM read all that and summarize?

And of course LLM can detect duplicated content - even if it is rewritten, it should generate a similar meaning vector.

Reply on news.ycombinator.com