Comment by dopa42365 | Hacker News Reader

Comment by dopa42365 | Hacker News Reader

Parent Full thread

Some basic filters for stuff that pops up all the time and is never what I'm looking for:

/.pinterest.\../

/.torrent.\../

/.spankbang.\../

/.mp3.\../

/.proxy.\../

/.block.\../

/.addiction.\../

/.recovery.\../

/.detox.\../

/.clinic.\../

/.rehab.\../

/.rarbg.\../

/.hydra.\../

/.123.\../

/.stock.\../

/.linkedin.\../

/.subtitle.\../

/.shopify.\../

/.dreamstime.\../

/.depositphotos.\../

/^(m|mobile|it|es|ja|fr|pl|at|ru)\.\w./

Also recommend TLD filters:

/.\.(tk|photo|dev|page|science|ninja|monster|st|cricket|store|website|help|bg|party|work|xyz|link|camera|computer|click|cc|club|pw|space|info|rocks|review|faith|date|win|racing|site|online|tech|news|life|design|nyc|guru|photography|global|today|solutions|media|world|biz|gq|rest|ga|casa|buzz|cam|ml|fit|ly|it|vip|php|pl|ru|app|name|tr|asia|surf|top|cf|tw|uz|am|krd|is|cz|pt|pk|cl|il|in|hu|hr|si|ee|bid|vn|kw|ro|mx|me|market|nu|how|red|guide|mobi|cafe|tn|xxx|icu|fun|pro|desi|id|live|one|ir|forsale)\/(.*)/

(modify for your personal needs, it'll save you many, many hours in the long term)

Ah yes the notoriously spammy .dev TLD.

I guarantee you that .com, .net, and .org have far more spam and low quality sites than this stupid list.

The original generic TLDs have many highly-ranked legitimate sites.

The problem with extended generic TLDs is that the spam sites have managed to become highly ranked. Yes, there's good stuff there, undoubtedly. But search engines can't find it. At which point it may as well not exist.

I'm not a fan of outright blocking at this level without good cause, but it could well be worthwhile to block an entire TLD in cases.

I'll add that if you're getting unwanted ads for that many drug addiction recovery-related terms constantly, perhaps your habits do merit examination.

You're losing lots of quality content by removing those TLDs. I'm using 3 of the listed ones just because they're cheaper and found similar ones to be a good indicator of "this is less likely to be blogspam, and more likely a personal space".

Ya, I use them for personal space. My last name is five letters ending in "st" and I managed to snag that three-letter st domain which I was pretty happy about :) I know a lot of people seem to hate them--and a lot of non-tech-types don't always seem to understand that they are domains--but I think they're fun, and I love finding peoples' personal homepages (mine is currently borked, of course, but not because of the st domain).

There is also a lot of junk on those TLDs, especially the inexpensive ones, outweighing the ratio of good-to-crap found on the big few.

Blocking them in search lists doesn't stop you finding them indirectly by them being mentioned in content elsewhere, or from even more direct referrals, so the non-junk ones can be found those ways. I tend to find useful content on .com et al as much those ways as by direct search these days anyway.

I'm not sure how spankbang got on that list but it's hilarious. Is it really a common problem like pinterest and similar stuff on that list?

If there is such thing as being "too good" at SEO, Pinterest is "too good" at SEO. It's a common complaint with image search that Pinterest will cannibalize the results. What will often happen is someone will repost an image from a website or blog to Pinterest and then when you look for images of that thing, you get the image that was reposted to Pinterest instead of the OG image.

It's supposed to be an adversarial system. Why does Google still provide Pinterest search results, given that if you follow them, you're not allowed to see the image?

I doubt Google wants to, but their users keep clicking on Pinterest's SEO'd results for "cute cake pop" instead of whatever non-Pinterest result Google serves up. Basically Google being outdone at their own game in this niche area.

Google special-cases their own stuff all the time. Why haven't they done that for Pinterest?

Who says they haven't? As to why they haven't totally blocked Pinterest, that would be pretty controversial given Pinterest paints inside the lines.

edit: oh, I see the confusion. People misunderstand what Google image search is, how it works, and are blaming Pinterest for Google's ineptitude.

People will correct me if I’m wrong, but I believe the practice of showing an image in search results and then hiding that image behind a login wall when users click the link would qualify as cloaking [1], a practice Google specifically prohibits [2].

[1] https://en.wikipedia.org/wiki/Cloaking

[2] https://developers.google.com/search/docs/advanced/guideline...

Unfortunately this and similar rules are not well enforced. The only high-profile case I can remember was when they delisted BMW for a short while for blatant keyword stuffing.

> given Pinterest paints inside the lines.

Do they? They advertise images in their search results that you aren't allowed to see when you follow the link. It's some of the most paradigmatic black-hat behavior there is.

You can blame Google for that, how on earth is that Pinterest's fault? Google is the one choosing an arbitrary thumbnail out of hundreds of images to represent "pinterest.com/cake-pops". Then Google uses that thumbnail to refer people to Pinterest, but doesn't tell Pinterest what thumbnail they used. Pinterest's hands are tied here. This is Google's fault, 100%.

Well, considering that a user can't view the image, Pinterest is showing images to Google differently than to the average user. What Pinterest does by hiding the images is specifically against Google's policy. There's no question that Pinterest deliberately went out of their way to create the currently existing paradigm, to suggest their hands are tied and they are somehow a passive victim of Google rather than an actively malicious actor is ludicrous.

It may well be the case that this is Google's fault and they are a malicious actor here as well as Pinterest, but Pinterest is blatantly malicious and does not draw within the lines.

You're seeing the same images that Google does, it's just that these are dynamic pages that change over time. Google might only crawler pinterest.com/cute-cupcakes/ every day, for example. Meanwhile, Pinterest refreshes the content of that page every day, but at a different time than Google crawls it. Boom, you click on the image google showed for "/cute-cupcakes/" and what Pinterest was showing is no longer there to be found. Both Google and Pinterest are misbehaving here. Google is using Pinterest to beef up its image search. Pinterest is using Google to get traffic. But Google won't share any of its precious data with other companies unless they pay for it, so Google doesn't pass anything like a hash or X-ORIGINAL-IMAGE-URL in the referrer headers when you click on a link, so Pinterest's hands are tied, sort of.

Anyway plenty of bad behavior to go around. Both Pinterest or Google could probably solve the issue with better engineering. I put a bit more responsibility on Pinterest, but I'm not giving Google a total pass here either.

This happens because Pinterest is better at categorizing images than Google is. Google should just buy Pinterest and integrate them directly into their image search. Or partner with them so this doesn't happen. Or they could do what Bing did and just copy Pinterest feature by feature.

interesting, does that mean that there is a part of 'active' pinterest users who take time to add tags to pics, letting pinterest to categorize a lot of pictures and let the google ranking higher?

At the moment it's one of my favorite websites of "this type", because they haven't done the content purge yet like the main sites did.

That's a whole lot of TLDs to block. .me is very common for personal pages/blogs/emails. .app is used for a lot of legitimate sites as well.

dot NYC is full of spam? A pity. Hard to get voting or ferry info here without it. :)

Then edit the list instead of blindly using it? And surely you know the local addresses for such things, or do you search for them using Google every single time you need to travel or vote?

God damn it. I don't want heroin detox; I want heroin.

Careful what you wish for

you really don't...

Is there a word or expression for when people fail to spot fairly obvious sarcasm?

.dev blocks a lot of personal blogs

Almost all personal blogs fall under that tld filter.

This "recommended" TLD list would block my bank site, government tax service and most of news sited since my country TLD is blocked here. Hilarious.

It is only a suggested list. Edit as needed depending on locality. I don't expect it would inconvenience you if it blocked my local bank or government tax service.

Always review blocklists rather than applying them automatically as your needs may vary from their creators'. No one list is going to be right for everyone, so criticising one for not being right for you is a bit "me me me me".

You need to escape your * characters, by prepending them with \ like \*, to stop HN interpreting them as markup intended to control italic/not.

I'm using .xyz domains for my side projects because they are cheap. Guess they are because they look "spammy" to some.

Interesting thing about .XYZ is that if you try to send someone a link with a .xyz url via SMS it usually just gets quietly spam filtered out by the carriers and never received.

Finally, death to pinterest!

Reply on news.ycombinator.com