Marginalia.nu API

Regarding the abuse, have you tried hashcash style challenges to mitigate them? Basically, you serve a random, signed string S; have the client brute force x such that prefix(hash(S+x),0,2)==“00” and submit that along with the search query. The problem with this approach is the client can offload the calculation to another machine which is part of a botnet. There are more advanced passive and active fingerprinting methods that you could use, but they require non-trivial infrastructure to collect said fingerprints, not making it worthwhile for a side project like this.

> Dunno why this is on HN but sure.

Maybe the poster needed to summon you. I’ve noticed that whenever the website is mentioned, you appear. No need to do it thrice, either, like other legends.¹

Jokes aside, Marginalia (and your interactions) seem to be well liked on HN, so no surprise someone submits them.

On the design front, I’ll recommend you don’t justify text, especially on the web. Unless carefully done it leads to rivers² and huge whitespace gaps which make text hard to read. Try that page on a phone to see what I mean.

¹ https://en.wikipedia.org/wiki/Bloody_Mary_(folklore)

² https://en.wikipedia.org/wiki/River_(typography)

Thanks for the cool project and extra details :)

I submitted this entry because your search engine has been interesting to follow. And it’s hacker friendly to have a simple and straightforward API like this!

"... it's constantly under siege by metasearch engines."

How does one know it's metasearch engines. Which ones are they. Have there been any discussions with them.

If a web search engine is truly noncommercial and one wants as may folks to to use it as possible, is it feasible to let others copy the crawler, the index and front end. It's like a fisherman who gives way his catch. As word spreads, he will be overwhelmed. Unless he teaches others how to do what he does, how to fish.

The problem with Google and all the Google wannabes is that they want all traffic to pass through their servers, for commercial reasons. In the beginning Google claimed their search engine would be in the academic realm. Presumably others could copy it. This turned out to be a lie. Now people cannot not seem to separate the idea of web search engine from potential money printing machine. The fisherman does not want to help anyone else to learn how to fish.

We need a noncommercial web search engine that can be replicated. It does not have to be a Google clone. A great way to stop anyone from working on web search engines is to suggest any new search engine must be as good as or better than Google, i.e., it must be huge, it must be commercial, and it must be a black box. On the contrary, a new noncommercial search engine can have humble beginnings and does not have to be anywhere near Google quality. IMO, there should be noncommercial web search engines with low traffic. They could all be instances of the same crawler, same index and same front end, they could be variations, they could be totally different. But IMO there should not be one set of servers controlled by one entity hoarding 90% of noncommercial search traffic.