Hacker News Reader

Comment by nonrandomstring | Hacker News Reader

> but that is detectable so you end up renting IP's from mobile number farms or trying to build your own 4G network.

Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found.

The established idea of a "search engine" seems stuck, limited and based on some 90's technology that worked on a 90's web that no longer exists. Surely after 30 years we can build some kind of content discovery layer on top of what's out there?

View on HN

https://blogsurf.io/ is an example of a small search engine that just stuck to a directory of known blogs instead of indexing the big sites or randomly crawling the web and ending up with mostly gibberish pages from all the spam sites.

thank you for sharing this! I read through the site's about and I really enjoy how the creator wanted to stick to a specific area for quality over quantity.

> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found.

I work on a small - medium ecommerce website and my code just... sucks. I kind of don't want to admit it but it is true. When there is some Chinese search engine that tries to crawl all the product detail pages during the day (presumably at night for them?), it slows down the site to a crawl. I mean technically I should have the pages set up so they can't pierce through the cloudflare cache but it is easier to just ask cloudflare to challenge user (captcha?) if there are more than n (I think currently set to something small like ten) requests per second from any single source.

I don't understand all the business decisions but yeah, I'd suspect the biggest reason is we simply have poor codebases and can't spend too much time fixing this while we have so many backlog items from marketing to work on...

Why are page loads so slow or demanding? I can't imagine how a web crawler could be DoS'ing you if it's in good faith. What is the TPS? What caching are you doing? What's your stack like?

Not GP, but from having run a small/niche search engine that got hammered by a crawler in the past:

Webserver was a single VM running a Java + Spring webserver in Tomcat, connecting to an overworked Solr cluster to do the actual faceted searching.

Caches kept most page loads for organic traffic within respectable bounds, but the crawler destroyed our cache hit rate when it was scraping our site and at one point did exhaust a concurrent connection limit of some kind because there were so many slow/timing-out requests in progress at the same time.

I would expect that a small to medium e-commerce site would cache all their pages.

That approach depends on what's in the pages.

Does the served HTML contain everything: the product, related products, comments, reviews, etc? If so, caching the entire page might be counterproductive.

But if the page is designed in such a way where it's broken up into fragments, then caching heavily is your friend.

Those fragments can either be loaded client-side (Javascript) or server-side (ESI, SSR, etc) -- but page fragments are key to increasing how much and how long you can cache.

And with this, you've added a layer of complexity that a small to medium e-commerce site may not be able to fulfill with their in-house talent.

> If sites don't want to be found and indexed why go to any effort to include them?

The news sites etc. want to be indexed by Google, and they whitelist Google so they show up in search results. They don't want to be crawled by random aggregators and scrapers which don't bring them ad views and subscriptions. Indexing for me and not for thee.

Other places like Slack channels are private or semi-private and shouldn't be indexed, I agree. There was some kind of scheme a while back (before Freenode went down the tubes in a completely different and unrelated way) to index Freenode. People rightfully freaked out about it and the scheme was shot down.

I think that is not what they mean. I think what they meant is the site will detect your headless robot and serve it good content, while serving spam and malware to everyone else. The crawlers need their own distributed networks of unrelated addresses to prevent or detect this behavior.

Maybe we need a categorized, hand curated directory of sites that users can submit their own sites to for inclusion and categorization. Maybe like an open directory. Perhaps Mozilla could operate it, or maybe Yahoo!

I know, right? Imagine if you went to the front page of Yahoo! and it was like a curated directory of websites. Like... a portal.

It could look something like this: https://web.archive.org/web/20000302042007/http://www1.yahoo...

Wow this gave me such an overwhelming feeling of nostalgia. I really miss the early years of the web.

We could also make a website where people can submit links to great websites they find, and also allow then to vote on the submissions of other users. That way you have a page filled with the best links, as determined by users. Maybe call it "the homepage of the internet".

You could even add the ability to discuss these links, and add a similar voting system to those discussions.

And dang could moderate it!

hell yeah. something like that without [sane] moderators would turn into fucking reddit.

I would love to see a combo of a search engine and a portal. Basically at any step drilling down into links for categories and subcategories, I could type in a search bar and every site at that level and below would be searched.

I've been sort of considering going this path with my own search engine, but I feel whenever there is any sort of interactivity, there is the serious problem of spam and abuse; I'm not quite sure how to solve that.

Feels like accepting user content no longer flies on the web in 2022 due to the effectiveness of bot spam and lack of viable countermeasures :-/

I think non-crowdsourced manual curation (by you/your trusted volunteers) would be a feature. Sure, search would have more limited results, but really with how much cruft there is I don't see that as a bad thing. Start with the "known good/useful" domains and branch out from there.

Maybe. The upside of integrating it with a search engine is that you kinda can get dead link detection for free. Otherwise link rot is a real bitch to deal with for web directories maintained by few people.

Could probably back it off some human-readable format in a git repo or something, pull requests might help with moderation, and recruiting volunteers might be easier if the data is open and available/forkable for other projects.

With Goggles[0] (goggles/googles/potato/potato) you can get them. Curated lists by topic.

[0] https://search.brave.com/help/goggles

This is EXACTLY how world wide web has operated before google.

>> If sites don't want to be found and indexed why go to any effort to include them? On the other hand there are millions of small sites out there keen to be found.

Then they should treat all bots equally and block Google as well. If they block Google as well, then yes, we should leave them alone.

Why give unfair treatment to Google? That's anti-competitive behavior and it just prevents new search engines from being created.

I think I understand, combined with jeffbee's answer, that these sites are behaving selectively according to who you are. So we're back to "No Blacks or Irish" on the 2022 Internet?

What do you think they have against smaller search engines? I can't quite fathom the motives.

There are a lot of crawlers out there, and many of them are ill-behaved. When GoogleBot crawls your site, you get more visitors. When FizzBuzzBot/0.1.3 comes along, you’re more likely to get an overloaded server, weird URLs in your crash logs, spam, or any other manner of mess.

Small search engines getting blocked is just collateral damage from websites trying to solve this problem with a blunt ban-hammer.

This is probably because, as I have learned here on HN, 1 in 4 programmers can't write a decent FizzBuzzBot and should never be hired.

> So we're back to "No Blacks or Irish" on the 2022 Internet?

I think this is the inevitable result of surveillance capitalism. Once you can reliably be identified as holding certain views or meeting some arbitrary criteria pages can be dynamically altered to restrict what you have access to. A racist social platform can look completely benign to anyone who isn't part of the in-group, forums for teens can automatically lock out anyone suspected of being over a certain age, companies can show products or services only to people within a certain class, etc.

We already restrict a lot of access to content based on guesses about location. Youtube doesn't even let you see that certain content on their platform exists on their site unless you're logged in from an IP in specific counties (as opposed to showing you the videos exist and throwing up an error saying it's not available for you when you try to view it) and netflix blocks VPNs just to try to keep a bunch of their content away from the wrong kinds of people.

These are the kinds of barriers the internet should have freed us from, but instead it's being used to put up more gates and to force people into them.

> Something is deeply wrong with such an adversarial ecosystem. If sites don't want to be found and indexed why go to any effort to include them?

I think it is not about being found. It is more about being copied.

These sites are afraid their content is stolen, so they only allow Google to crawl them.

They don’t want to be indexed unless you are Google

Sounds like federation might be making a comeback eventually

Reply on news.ycombinator.com