Uncertain Future for Marginalia Search

Dunno. In general I don't have a lot of faith in the profitability of search engines. Ads can work if you're Google-scale, the other option is subscriptions, but in that case, you need to be really good and my search engine just isn't, outside of some areas. That's actually one of my bigger design problems, how to let people understand which queries are likely to be useful. It looks like Google, and people assume it has the affordances of Google. It doesn't, and if you go in with those assumptions, you'll be disappointed.

My model, as far as I've planned one, is just to keep the operation as cheap as possible and subsist on donations and maybe partnerships with other search engines. A big part of what I'm exploring is ways of doing as much as possible with low power hardware. I think rather than indexing 1 billion documents, 90% of which are garbage that will never be a good search result for any query ever, if I can index 100 million 50% of which are potentially good hits, then maybe that goes a decent way.

> 2. Have you tried asking for sponsorships, either from companies or individuals? You should have an easy way for people to donate. I'm sure you'd have some support there, especially if your day job situation is unstable.

I haven't really been fishing for this. I honestly didn't see having to change jobs as I am right now. I do have a donations page from before, but all of this was fairly sudden, so I haven't really gone over that whole process all too much yet.

> 3. Is it just you working on it right now? Have you considered open sourcing it to get community contributions, or hiring more devs (once donations pick up or maybe someone would be willing to work on it on their free time as you do)? I can imagine that writing a search engine is a gargantuan effort, and doing it alone must be close to impossible.

It's been just me up until now. Solo work can be ridiculously efficient when beginning a new project, especially when doing the sort of exploratory programming this has been. I also haven't felt I have had enough bandwidth to manage an open source project. But I am approaching a point where it's becoming a bit much to do all by myself, especially given this isn't my only project.

So I am considering open sourcing it or bringing more people in, just need to think a bit about a good format for such a collaboration. It's relatively high maintenance and requires manual operations to keep going. As it stands, a lot of the code isn't trivially testable, running it (even with few documents) requires large language models and so on.

This might be a good time to start linking that in obvious places.

Fwiw it was very easy to find it through Google, but ironically not through marginalia.

I hope you the best in your endeavors.

Sorry, could you point out the bit where the wiki page says anything like that?

The problem is that an empty search box does nothing to tell you what could fruitfully be typed into it. Good type-ahead suggestions can help assist the query formulation a bit, but that's a band-aid at best.

You could reformulate basically any incidence of a search engine returning bad results as the visitor entering a bad query. The emphasis has been almost entirely on improving the capabilities of the search engine to interpret bad queries, rather than working with the interface design to offer the user an intuition for what is a good one.

Realistically you need to approach it from both ends, but I think the work has been fairly lopsided in search and discovery.

Some WARCs that go into IA get published on archive.org, not all of them, but some: https://archive.org/search.php?query=warc

It's also an all-around useful format as you can produce it from wget and other common tools. But the big reason I'm moving toward something relatively homomorphic to WARCs is to be able to (in the future) publish my own crawls.