Comment by crazygringo | Hacker News Reader

Comment by crazygringo | Hacker News Reader

Parent Full thread

I may have found the answer, and I've seen this before (it happened to me once). It's when a different spam site copies your content wholesale, and a search engine decides they're the "original" site, and you're the spammy copycat.

Because if you put the headlines (in quotes) from two of his recent articles into Bring, e.g. either "Megan Smith explaining the General Magic prototyping process" or "Denialists, Alarmists, and Doomists", both point as their first result to a URL starting with "https://www.scien.cx" which seems to be the spam site with a copy of each article. (The URL isn't loading right now, however, when I try to visit.)

How to fix it really depends on what techniques they're using to mirror your site, of which there are many.

Example search and resulting URL:

https://www.bing.com/search?q=%22Megan+Smith+explaining+the+...

https://www.scien.cx/2022/12/25/megan-smith-explaining-the-g...

Compare with Google getting it right:

https://www.google.com/search?q=%22Megan+Smith+explaining+th...

https://daverupert.com/2022/12/megan-smith-general-magic-pro...

Recently on HN there was "Someone is proxy-mirroring my website, can I do anything?": https://news.ycombinator.com/item?id=33952114

It feels like the Internet is a more hostile place than ever for small-time websites. You get squeezed from below by wily criminals, and crushed from above by careless megacorps who want to filter out anything that doesn't make them money.

Nowadays, when I search for things, the results are often clearly pages that have come from a program scrapping sites and then merging them into one page. You can tell because the pages are not really coherent and quickly start to repeat themselves. I assume they are getting money through ads on the pages, though I never actually see the ads because of my blockers. I wish there was a button in the browser that I could click to report the page as spam to all search engines.

> I wish there was a button in the browser that I could click to report the page as spam to all search engines.

The spammers would use that offensively to destroy the original content creators with that system

The way it should work is that the spam reports are used only to filter the results you see.

That's precisely why I started to use and pay for Kagi - it fetches search results from Google and Bing, but allows to prioritize, deprioritize, pin or block specific domains in your search results.

I'm still surprised no one else seems to be offering this feature.

Seems reasonable. If enough human-like users (with gmail accounts, yt activity, and other indicators) ban a particular site enough times that should offer some evidence that it's low quality too.

uBlacklist is really good for that. Just click "block this site" in the search results.

https://chrome.google.com/webstore/detail/ublacklist/pncfbmi...

I don't think it reports to any other search engines, and am not sure it affects any but your own subsequent results, but Kagi has a feedback scale on each result: "block", "lower", "normal", "raise", and "pin".

> It feels like the Internet is a more hostile place than ever for small-time websites. You get squeezed from below by wily criminals, and crushed from above by careless megacorps who want to filter out anything that doesn't make them money.

The problem is that the two work hand-in-hand, thanks to the advertising driven search model, and the search engines owning the main advertising platforms.

It should be easy for search engines to identify an original site from the SEO spammer rip-offs - the original site is going to have no adverts (or certainly fewer) while the SEO spammer copies are going to be covered in adverts. The problem is that the search engines have no incentive to do so, in fact if anything they have the incentive to send people to the sites with more adverts.

And of course the whole problem has been created by the search engines in the first place - there would be no point in SEO spammers making advert-laden ripoff sites if it wasn't to rake in advertising revenue.

No more hostile than real world, we are just finding out it is a reflection of our world, of course the difference being the global interconnectedness which magnifies the celebrities but also the crooks.

Well, it seems to me that none electronic outcomes tends to be slower and harder to copy at scale like digital material, doesn't it?

It's not clear that this is happening with all of the (many) sites that are mysteriously deindexed by Bing. See my comment: https://news.ycombinator.com/item?id=34389279

What should you do when another site copies your content like this then?

> What should you do when another site copies your content like this then?

Have we gotten to the point where websites (and their content) need to be verified like Twitter, Instagram, Facebook, and TikTok do for personal accounts?

If so, will search engines be the ones verifying - using this as a new revenue scheme (with the dangers inherent in this... ie; pay to be listed or ranked higher)?

I actually did add a Bing authentication XML file to my site, but Bing deindexed me again anyway.

Bing is the problem. It's broken.

Can confirm, authentication files don't help. Bing is seriously broken, and their support is all but helpful. This is an excerpt from one of their replies:

> "Thank you for your patience during our investigation. After further review, it appears that your site did not meet the standards set by Bing to remain indexed the last time it was crawled. To ensure that this was not a false flag, I also escalated the issue to our Product Team and they manually reviewed your site and confirmed that it is in violation of our Webmaster Guidelines detailed here:

https://www.bing.com/webmaster/help/webmaster-guidelines-30f....

We are not able to provide specifics for these types of issues but we recommend that you review our Webmaster Guidelines, especially the section Things to Avoid, and thoroughly check your site for any deliberately or accidentally employed SEO techniques that may have adversely affected your standing in Bing and Bing-powered search results."

Before snarking, please check that link and the long lists of things - I did not find to find my website https://linmob.net to be offending their "things to avoid list".

That was a reply to my first ticket requesting re-indexation, later tickets only got what I would call "non-replies".

No we’ve gotten to the point where Bing and DDG need disrupted. The answer is that these companies are ruining things not that DNS and simple search is wrong

I’ve moved from DDG to Kagi.com. It’s paid, but no ads and no incentives to remove legit sites that I’ve seen.

I have as well and have gotten at least one coworker to do the same.

No the problem here are Bing & co., their algorithm is clearly not good enough.

Various things from addressing the problem directly if possible (block the IP address range they use to scrape your content with, insert JavaScript to strip the content client-side depending on the domain it's being served from), to changing their search engine behavior (canonical meta tags, contact the search engine to let them know, build up links on the web to make your site higher ranked).

The more sophisticated and popular the copycat site is (scraping from a distributed network, stripping most HTML tags, etc.), the harder it becomes, and the only thing is to contact the search and hope they can manually mark your domain as the authoritative one. Your success may vary according to your popularity/importance.

> strip the content client-side depending on the domain it's being served from

That would almost certainly be regarded as a "doorway page"[1], resulting in you getting manually ranked downward or even deindexed entirely.

[1] https://developers.google.com/search/docs/essentials/spam-po...

You may have misunderstood, it doesn't have anything to do with doorway pages which are about presenting extra highly redundant content.

I'm talking about, if content on legitimatesite.com includes JavaScript that detects if it's being loaded on any other domain, then erase the entire article's HTML from the DOM.

Obviously this is easily defeated by stripping out JavaScript, so it's useful only for very primitive mirroring.

Hmmm, yeah "doorway page" probably wasn't the term I was looking for. However, that would almost definitely be regarded as some sort of spam or SEO tactic by crawlers and lead to further penalizations, which was my point.

You send a DMCA.

Chances are that the site will be hosted outside the US and the DMCA is a US only copyright law afaik. I don't think it's applicable outside the US.

In those cases, you can send one to DDG and Bing, which are in the US. It won’t affect the actual website or other search engines, but it’s better than nothing.

Indeed you can. I've had similar issues before (a site scraping content, though apparently from dev.to instead of my own domain and a YouTube channel making a 'video' with text and TTS from a post).

In the first case, I sent a DMCA to Google & Bing as well as to Cloudflare. Cloudflare responds by giving the name of the actual host, and I sent another DMCA to that host (they were US based, otherwise YMMV). The content was delisted (not the site, even though it was made up entirely of verbatim scraped content) from search engines and from the site.

Bottom line is you can send a DMCA notice to search engines and it appears to be effective. Actually, in case search engines demote sites like this in some way, I would send the DMCA notice to search engines _first_, because if the content gets removed from the original site they may not be able to verify the duplicate content.

Ianal, but I think ddg could come back and claim that their service doesn't copy the content or that their abstract is fair use, so a dmca wouldn't apply to them.

If that happens I'd use language about the exclusive "public display" right that you have over your work.

I'm not aware of this claim being used for a dmca, but I'd like to see how such a claim turned out.

IANAL either, but the DMCA has been used (by the RIAA, MPAA, and friends) to takedown pirate content on Google. I’d assume the same arguments would apply here. It can’t hurt to try.

Linking to a site that infringes copyrights may make the linker liable on the grounds that they are committing contributory copyright infringement. It's cheaper for search engines to delist sites than fight that battle--particularly if their corporate masters also rely on licensing media for distribution themselves.

IANAL but could this be what the spam site has done to get OP delisted?

This is exactly a perfect use case for a blockchain. In fact if people are interested we should create a POC.

How do all of those Stack Overflow mirrors stay up if there is a mechanism to pull the copy-cat?

Maybe there is a retrocommission program to which they are affiliated?

Reply on news.ycombinator.com