Hacker News Reader

Comment by ddevault | Hacker News Reader

Good morning, HN. Please note that SearchHut is not done or in a presentable state right now, and those who were in the know were asked not to share it. Alas. I had planned to announce this next week, after we had more time to build up a bigger index, add more features, fix up the 404's and stub pages, do more testing, and so on, so if you notice any rough edges, this is why.

I went ahead and polished up the announcement for an early release:

https://sourcehut.org/blog/2022-07-15-searchhut/

Let me know if you have any questions!

View on HN

Just a warning from a fellow search engine developer.

If you happen to be cloud hosting this, and if you do not have a global rate limit, implement one ASAP!

Several independent search engines have been hit hard by a botnet soon after they got attention, both mine and wiby.me, and I think a few others. I've had 10-12 QPS, sustained load for weeks after weeks from a rotating set of mostly eastern european IPs.

It's fine if this is on your own infrastructure, but on the cloud, you'll be racking up bills like crazy from something like that :-/

Could you clarify domain submition rules?

e.g. "Any websites engaging in SEO spam are rejected from the index" - how is determined whether something is SEO spam or not? More clarification of whats allowed/not allowed would be nice!

The criteria are documented here:

https://searchhut.org/docs/docs/webadmins/requirements/

And there's some advice for web masters on ways to improve your site's ranking without running afoul of this rule:

https://searchhut.org/docs/docs/webadmins/recommendations/

But ultimately, it's subjective, and a judgement call will be made. If it's minor you might get a warning, if it's blatant then you'll just get de-listed.

I think it would be great if we had Code Forge index to search uniquely. In this index are only the myriad of code hosting sites around the internet - shared hosting like gitlab, github, sourcehut, sourceforge, codeberg, and all the project instances like the kernel.org, GNU Savannah, GNOME, KDE, BSD, etc. Probably hundreds of them out there, and allow people to submit their own self-hosted Gitea/Gitlab/sr.ht/etc. instances to be crawled - maybe even suggest a robots.txt entry your crawler could key in on as "yes please index me, hutbot".

Long ago -- 2006 to 2011 -- google had a functional source code search engine: https://en.wikipedia.org/wiki/Google_Code_Search

I don't recall if it supported SourceForge and GitHub (2008) but it certainly included gzipped tarballs which were popular and prevalent at the time.

Do you intend any of this to merge/cooperate with other similar initiative?

e.g opencrawl, internet-archive, archiveteam

It strikes me the resources to crawl, update, and manage/index data is a common problem.

I intend to at least support other search engines by adding !bangs for them and recommending them in the UI if you didn't find the results you're looking for. I don't think that crawling is something that is easily distributed across independent orgs, though.

Reply on news.ycombinator.com