Hacker News Reader

Comment by bnralt | Hacker News Reader

An important thing to realize, too, is that this is a problem that keeps getting worse. The article talks about product reviews and recipes, but it's been spreading a lot further than that. Recently I was trying to look up a technical error, and found a lot of web pages that seemed to be auto-generated with "How to solve [error_scraped_from_the_web]", complete with a list of generic things unrelated to the error (IE, "Step one: try turning your computer off and turning it back on again. This is usually a good first step, and you'll be surprised at how often...").

Likewise, I wonder how long appending "Reddit" will work. As others have pointed out, Reddit shills are already relatively common, and it's becoming increasingly common for bot accounts to create lots of random comments to appear to be human (such as finding a thread with thousands of comments, then copying and pasting the comment to another place in the thread or to another thread, or auto-generating a simple sentence based on other comments in the thread).

Sometimes the advertising hordes move so fast they kill something before it even takes off, like what happened with Clubhouse.

View on HN

Free project idea for someone with more time on their hands than me:

Classical search engines determine trust automatically, based on various factors including "link neighborhoods" where trustworthy sites link to other trustworthy sites. These automated strategies are clearly breaking down; the spammers are winning the arms-race.

So maybe we need to go back to human-based trust.

People used to curate lists of websites, which partly solved this problem but didn't necessarily scale. I wonder if that idea could be supercharged.

Consider a browser extension that people install, which:

a) gives users a button to mark a site as trusted/favorited

b) tracks domains visited (and frequency)

Then, separately, you can manually add people you know personally to your "network". You trust them, so anything they trust is also something you might be able to trust. Manual favorites could be weighted higher than frequently-visited sites, and both could be displayed inline next to links on all pages you visit. You could also see which people the trust in a given link comes from, in case some of them consistently have bad judgement about these things and you want to remove them from your list. Then, finally, you could create a personalized search-engine that only indexes the sites determined to be trusted by your personal network.

Of course this would require placing a great amount of trust in the extension and service themselves, so maybe they would have to be open-sourced or self-hostable or something (a profit motive might create a huge amount of temptation to abuse the data). That's a stickier problem.

Edit: There was a little ambiguity left here about transitive trust; “friends of friends” type stuff. I think if this went on for unlimited hops, we’d be back at square one. So maybe it only uses direct contacts, or maybe some small N of hops (where longer ones are weighted lower?). Maybe this would be configurable, not sure.

Also re: privacy, maybe you could come up with a clever way to E2E encrypt the site visit data, even though it’s shared with many parties?

Interesting.

Another "sticky" problem is how to make a living out of this I guess...

Have the extension serve ads.

There's no need for an extension; plenty of websites are part of webrings or feature blogrolls. I'm in the process of adding one to mine.

Throw in some microformats2 and/or schema.org structured data and you're good to go.

Certain search engines specialize in this type of manually-curated content; I listed some in the "non-generalist search" section of my collection of indexing search engines: https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

I've just had the deja Vu reading this post! Took me a minute to realise, I did actually come across this post and your blog some time ago, and really enjoyed reading all the posts then and now again! Hoping to read more from you!

Someone will sooner or later mention the sybil attack.

(At some points I feel it is thrown out as haphazardly as "correlation does not imply causation".)

But I think you might be onto something. It won't necessarily be easy but I think it deserves more than a quick dismissal.

GP comment already explicitly addressed this:

> Then, separately, you can manually add people you know personally to your "network". You trust them, so anything they trust is also something you might be able to trust.

It doesn't matter how many sockpuppet accounts a marketing company creates if you don't click the "trust" button on any of them, and if the system is designed so that your local trust network only consists of what you trust directly and (weighted) transitively.

I've been wanting[0] more or less the same for a long time now. Ideally the dataset should be public (and distributed) so that you could find bad actors and eliminate them from your network (i.e. when you see a bad result, you can figure out where it came from into your feed), as well as run your own tweaked algorithms. WoT alone is great, but it gets better still when you can tweak the algorithm.

At that point, it doesn't need to be people you know personally; it just needs to be identities with a proven track record. If you can manage multiple identities, you could make ones e.g. for the explicit purpose of presenting a blacklist of bad sites you've found, or for providing a "front page" to a certain subject. Anyone is free to trust your identity or not, and choose exactly how to rank its actions.

This whole thing could be expanded to tagging and other metadata as well as comments, and so on.

[0] https://news.ycombinator.com/item?id=29691303

I'm so glad to see others talking about this.

A system of distributed trust like this is the makings of decentralized moderation. Decentralized curation. I've also had this idea on my mind for a while: https://news.ycombinator.com/item?id=21975294

I like this idea of incorporating trust and reputation. As for the curation of websites not scaling, some time ago I thought about the possibility of a search engine where the user supplies a list of trusted websites (for example, university websites, blogs of people they admire), and the search engine ranks pages based on link distance to these websites.

I think the Keybase project would have been great for providing the "authentication" part of this solution. To bad it died on the vine of the Zoom purchase.

The biggest problem, doing that, is that you increase the pressure from bad actors to websites that are trusted by lots of people.

So, for instance, the more weight you give to sites that are quoted by Wikipedia in your search rankings, the more content farms will have incentives to sneak edits that link to their sites.

There are ways to counter that (eg moderation), but in general, defense is more expensive than offense.

Right but the idea is that you don’t use sites as sources of trust, you use people. Content farms won’t be paying your friends and family to shill websites to you.

"Like us on Facebook and get this 10 cent coupon or even the chance of winning something real!"

Works suprisingly well already, sadly. So yes - at some point companies would pay people to vote for them.

But I still would prefer that system (with some differences) over the default. Because the people I would trust, would not fall for the common scams.

I mean, of course they will?

A large fraction of facebook content is MLMs and mobile game ads from one's own contacts.

Then you just stop trusting them for your search results, it's not like you are unfriending them.

I think you are applying a solution that would work in a cooperative game. The problem is that this situation, like spam, is an inherently adversarial one - and one where one's adversaries are very motivated and have substantial (substantially more?) resources than you at the outset.

Like the Web-of-Trust or similar?

This reminds me of Cory Doctorow's "whuffie" from Down and Out in the Magic Kingdom (man, that title takes me back!).

Whuffie is, roughly, money determined by your social interactions. More importantly, others also have a queryable score that's weighted according to who you esteem highly - this sounds like what you're proposing!

This was kind of the idea behind a side project I started a few years ago:

https://blog.digraph.app/2020-06-13-democratization-of-searc...

In that post, I don't address the reputation management aspect as much, but it's central to making the whole thing work, and I think crowd-sourcing and a well-conceived reputation management system that can influence results are good next areas for exploration.

You could also incorporate "reputation" of the author. Basically have a real person, the author stake their real identity on the quality of the blog post they wrote.

I was thinking something similar recently, and I also believe it's an idea worth exploring. Something else to add to this conversation. There's an obvious difference between two cases in which a trusted person trusts a url:

1) Single contributor website (blog, personal page...): It seems that we could spread the trust the whole website in the algorithm (at least more than for the next case)

2) Multi contributor website (forum, newspaper): It seems the trust should be given at an URL level

Something worth delving into if we are designing this trust based search engine in real-time here at HN ;)

This is very much in the spirit of what we were trying to do with trove.to [1] — give people an easy way to curate & annotate lists of websites, and layer a social graph and endorsement system on top of those lists.

The problem we encountered is that the vast majority of people are not hyper-organized list makers — the 1% rule of the internet [2]. To create a "human curated search engine" with any utility, you need a massive amount of manually-categorized data — data which most people are simply not interested in generating. This is why no social bookmarking site (e.g. delicious, pinboard, etc.) has ever taken off to hundreds of millions of users.

I still think there's something exciting to be built here, but it will likely need to take a more "automated" approach as you suggested.

[1] https://trove.to/

[2] https://en.wikipedia.org/wiki/1%25_rule_(Internet_culture)

Yeah- that’s the only reason I suggested the automatic tracking of visits by domain, despite the obvious privacy complications there

Reddit effectively has this information buried in its database. But they don’t really use it, preferencing issues of the day and recycled tik toks apparently.

Except Reddit is now a top-down company, so I wouldn't expect them to innovate beyond what a typical MBA can regurgitate.

I come across these sites so often it's not even funny.

Different website. Different title. Exact same content. 4 or 5 in the first page of search results.

I'm assuming they're all ran by the same person, throwing as much ** at the wall knowing some will stick.

Many of my searchers now include "reddit" or "forum" at the end to filter out all the spam/crap.

"Different website. Different title. Exact same content. 4 or 5 in the first page of search results."

If only google was smart enough to figure this out

This feels like the underlying issue. Google may have stayed the same, or even slightly improved.

But the web, in the sense of quality:crap ratio, has gotten substantially worse.

This flood seems like the ultimate manifestation of turnkey hosting solutions.

Imho, we could do worse than reviving an idea from email's early days vs spam: negligible per-use charging. The idea was to tax emails at $0.0001 (or somesuch). Insignificant for actual users, but financially decimates high-volume, low-value spammers.

The web is like that because content farms are optimizing the pages to be found by Google and Google doesn't know how to filter them out, so we really can't treat it as a problem independent of Google itself.

Fair. The fact that Google exists + the fact that Google serves a huge amount of traffic + the fact that Google is unable / unwilling to filter out content farms = incentive to content farm.

If there were no Google though, we'd likely have the same thing.

So I guess the only reality that avoids incentivizing them is one where (1) there is a massive traffic generator & (2) that massive traffic generator severely disincentizes content farms.

It isn't so much that 'google doesn't know how to filter them out' but there's nothing left after having filtered them out.

Nobody is producing real content that isn't behind a paywall.

There's nothing to find.

I know this isn't true because I've used google for nearly two decades with good results. That information hasn't evaporated, it's just buried.

There is real content being drowned out by autogenerated SEO crap. I tried looking for rice cookers which didn't have non stick coatings, they do exist and some blog posts which talk about them but the top results are all stores which have just a generic category for rice cooker but generate 200 duplicate pages with the title changed to exactly match whatever your search term is. So it says "Ceramic rice cooker" but shows their generic listing of PTFE rice cookers.

Google search is constantly improving but the SEO spammers are improving faster.

In theory proof-of-work with increasing cost based on subjective untrustworthyness might work.

If you put site:reddit.com it Google will only return results from reddit.com

Not even this is a guarantee. This happens to me regularly:

https://twitter.com/jdgoesmarching/status/149367886211437772...

Interesting! I get all Reddit results on my laptop (Safari), but on my Android (Edge and Chrome) I get Good Housekeeping and NYTimes before Reddit. Some kind of ad? Though when I click on the three dots next to the result, it says it's not an ad. Odd.

Any Googlers want to chime in?

Yes, there's a particular bug that sometimes happens on mobile with some queries. It's relatively recent; we're aware and working to resolve it, because site: really should only show content from the indicated site.

I’ve had it for at least six months, almost always around product-related queries.

Yup, just found this morning that an article my wife wrote on a very obscure legal topic was stolen, reformatted, and posted on some "life hacks" sort of site. It shows up #3 in the DDG results. At least her originals are still #1 and #2.

Meanwhile I have in my inbox in the last 24h at least a half-cozen emails looking to do SEO work for my company website.

Web = untrustworthy? YUP

I'd happily pay for a serious version of 1999 Google, but updated to filter out anything advert based, and search for exactly what I want.

Search is such a fundamental function, and we've done the experiment and the advert model fails - it needs to be just another utility.

This has been happening a lot with StackOverflow and GitHub pages lately. A lot of the times, the actual GitHub or SO link won't even be on the first page.

I'm surprised they haven't done some kind of manual pruning of junk like that, or maybe they have and it's not working... but on the surface it totally seems like they could implement something that says "GitHub has content X, and these other 10 sites are 99% the same, but we've flagged GitHub as an authoritative source so they'll always outrank the clones".

Maybe it's a fear of appearing unfair. Or maybe they secretly want to hurt Microsoft by turning a blind eye. Or maybe this is actually a much harder problem. If I had to guess it's probably #3. But as a user of search it's frustrating to find the clones ranked above the real stuff.

Can’t they just look at where they first encountered the copied content?

Just as bad as the auto-generated pages are company blog pages whose SEO rigged post pretends to give "help" for a problem where the main solution is of course, using their product.

Yeah, looking up technical stuff now for anything outside of very major tech is an absolute nightmare. It's nothing but auto-generated pages made of random parts of random forums posts smashed together under some weird url like tech-helb-4-yuodbajdasdasd99234029242.co.xyz.com.org.

I've clicked on a few out of curiosity, immediately recognizing they were garbage from the description text, and it's just endless SEO links and completely random text.

You'd think one of the richest companies on earth could make a freshman intro to CS-level spam filter. If they can't, then they truly do hire the most incompetent people on earth. If they won't, then they hire the evilest people on earth.

Yeah, those pages are definitely auto-generated. Static site generation makes it possible for those types of pages (I call them "shims") to jump to the top of the results list. I wrote about it here: https://zestyrx.com/blog/nextjs-ssg

I don't see what static site generation has to do with it. You can spin up a huge number of shims even more easily with a dynamic site and a DB with a list of all the messages you want shims for.

SSG gives you the best SEO because web crawlers can understand the structure of a static page with pre-populated content better than a dynamic site that relies on Javascript. The pages also load much quicker and score better on other metrics that Google uses. I would go so far as to say that SSG is a must for major publications or anyone serious about SEO. This sort of data-driven SSG is exactly what you are thinking of but it happens at build time so that the build output is static content that can be put behind a CDN.

Absolute nonsense. You don't need a static site generator to serve up an HTML page instead of rendering with JS. People have been doing dynamic pages since before JS even existed.

The difference in speed between running a simple script and reading a file is not large enough to noticeably affect SEO.

Most of these sites are probably just using PHP, not any static site generator.

I concur on the looking up technical errors bit directing you to auto-generated sites!

I was recently trying to troubleshoot a very basic error message for Linux and was getting results and webpages that would list the error message in the title in some way, but then give instructions on "First, open up device manager", "Click Win+R to open windows command prompt", etc. Lots of untrustworthy ads. Different URLs, almost line-for-line identical webpages.

This was something like the top four search results (that weren't sponsored ads).

I usually look for these hallmarks:

1) if this is from the project's own site, it's good.

2) if it looks like an archive of the project's mailing list, it's good.

3) if it looks like an internet forum, it might be good, or it might be just another poor soul asking the same question.

4) if it's on StackExchange, it's like on the forum, except your chances are slightly better. Karma must flow.

5) if it's on Reddit, it's like on the forum, except your chances of getting an answer are worse.

6) if it's a blog of some geek, sometimes it can be better than 1) and 2), or you might just get a straight answer.

7) in any other case it's most likely a SEO farm. Run.

If I have a linux problem these days, google usually gives me the relevant piece of source code on Github from which the error message originates. Like, you're a big boy now, go figure it out yourself.

It seems we're back to the early 2000s, when search engines not so much specialized in topics, but leaned heavily towards one type of content or the other. Holy hell, maybe one day Reddit's own search will be good enough so google can be ditched for good!

What happened to Clubhouse? I just realized it was quite popular but now I haven't heard about it in a while. Can you link to a postmortem?

It still exists, the last app update was uploaded yesterday, but it seems most of the users didn't stick around after the initial hype died down. I guess the early focus on monetization and trying to turn it into payment app didn't help.

I want Google to allow me to specifically include/exclude mirrors from my search results. "Only show my the original source of this content", or "only show me mirrors of this content".

I don't want to see the same result repeated 5 times across different stack overflow mirrors.

If it helps,there's an extension called ublacklist that you can blacklist results. I use this to permanently blacklist those sites with garbage.

Google has publicly admitted they cannot reliability determine the original source of a peice if content. This is a big problem for content creators.

What happened to clubhouse?

I have noticed that a lot of tech and startup clubs on Clubhouse were created by users from India and Iran. Today the clubs are still dominated by users from these countries.

Nothing wrong with that but that also indicates the usual clickfarm spammers from developing countries had unfiltered access to Clubhouse from day 1.

This might sound elitist but it's probably a good idea that C2C apps get their first batch of users and community leaders from high income countries before branching out to the rest of the world.

India is a young country with a 1.4B population, with a lot of engineers. I know the quality can vary from gemstones to tombstones but there is an active tech community.

>This might sound elitist but it's probably a good idea that C2C apps get their first batch of users and community leaders from high income countries before branching out to the rest of the world.

Why?

For the same reason some apps (used to?) launch on iphone first.

> it's probably a good idea that C2C apps get their first batch of users and community leaders from high income countries before branching out to the rest of the world.

Unless they're targetting local communities with some sort of culturally relevant format, its good for their bottom line to target the most profitable customers of advertising anyways.

this is elitist and misinformed

It almost immediately upon the real adoption curve (this seems like October November 2020 anecdotally to me) just absolutely filled to the brim with NFT shills, cryptocurrency pumpers, and “self actualization with this one weird trick” promoters of their MLM. But early 2021 it was impossible to find any rooms with anyone discussing anything but these things.

Yeah, and unfortunately the only rooms not run by scammers were filled with people talking about anti-semitic conspiracy theories. What a cesspool it became.

Twitter knocked them off with Twitter spaces.

Nah, they just kind of sucked.

Like, who really needs a voip protocol pretending to be a program?

Google used to be better at filtering out garbage content like this. They have resources for detecting low quality content (e.g. all pages on this domain follow the same content-free pattern). I suspect that doing that wouldn't drive ad revenue up, so they don't bother.

So when doing a search, we are searching in a place (Reddit) that’s being moderated by humans.

Because doing a search in a place that’s not moderated by humans would generate too much noise.

I think this kinda takes us back to the old times with Yahoo (and humans sorting the information) etc…

A giant step backwards, if you ask me.

Why do you consider it to be a step backwards?

How bad is Google search?

I had to search crates.io . Let me tell you. It's not the pinnacle of search.

What I searched was `fast bitset implementation`. My results consisted of drill bits a stack overflow questions and a Baeldung article on hashSet vs long[]

> "Step one: try turning your computer off and turning it back on again. This is usually a good first step, and you'll be surprised at how often..."

This seems like a natural result of optimizing each search for revenue. Think of a search to solve an error message on your computer. There's a very small number of vulnerable people who are going to spend money as a result of that search, so optimizing for ads would mean tailoring the results specifically for those people, pushing them to sleazy sites where they might spend money on some kind of antivirus scam. The results are worthless to you, but who cares? You're worthless to Google when you're doing that kind of search. Try searching for something that people in your demographic spend money on, and the results will likely look better to you.

You forgot how the titles include things like "[SOLVED]" when there was no solution or "[Updated: 2022]" when it's clear nothing has been updated.

in google you can do site:reddit.com or even site reddit.com/r/subreddit

I do this to search for items on craigslist across the country.

This will force a reddit search.

Might the inevitable arms race between bot writers and bot detectors be the missing accelerator for a general AI that has a predilection for top 10 white label brands of generic consumer products?

Post truth society has arrived, trump was a symptom, not the cause.

Combined with AI imitating speech and deepfakes, and technology of inplanting false memories, we will have the matrix, just not the wau we expected ^ ^

Reply on news.ycombinator.com