Hacker News Reader

Comment by marginalia_nu | Hacker News Reader

I guess hypothetically you could go for a wikimedia-style model and set up a foundation that operates the thing, but that would require a lot of funding, which would require decent utility up front. Vicious circle there.

I'm struggling with how to run even a small scale search engine as an open source project. I haven't really found any good projects to model for how to go about an open development process for a system that actually requires decent hardware to run, so right now it's just me developing in public, which is fine I guess, but not much different from how it was before I open sourced it.

View on HN

Wikimedia foundation actually did try to launch such a thing:

https://en.wikipedia.org/wiki/Knowledge_Engine_%28Wikimedia_...

https://www.newsweek.com/wikipedia-takes-google-new-transpar...

Spoiler: it failed.

From what I understand, they basically built a search function for wikipedia (and related projects). A noble goal since their own search function is kinda shit, but it was never really an internet search engine.

Well, what you could try is an engine that's designed to crawl specific niches and then allowing people to run their own instances. So you don't run any hardware yourself. You just let other people provide it - they might find a way to make money from it, or their community might find ways to pay for it anyway. It's not necessarily true that every search engine has to search the entire web.

The problem with this is that you get significant benefits from a larger crawling corpus, even if you index just a small portion of it, the rest will inform that portion.

That's a real problem that is ultimately hard to get around. Like my index is fairly small as it is, but even so it requires more hardware than say a student could afford. Like it's not enough that I'm dependent on external funding, but it's still not something you slap together for fun and then get bored of.

Reply on news.ycombinator.com