I think that Search Engines have been mostly commoditized, and that the company with the biggest budget pretty much wins the game.
However, maybe there's space for a different kind of search engine.
Would you build a new one today? If so, what would make it different from anything else?
A new search engine would just drop the top x sites from its index, and be aggressive with dropping sites. Blog spam? Link farm? Quora? Pinterest? Wikihow? Don't index them at all. A Shitty wordpress site that's just copying stuff verbatim from other sites? Drop them. Over engineered SEO? News sites? Fake news? Satire sites? Don't bother with them etc. (or maybe put a clear disclaimer in the case of satire sites)
Make search about what it's supposed to be, discoverability.
There are so many golden sites out there to be discovered under the absolute cruft Google and the like bury us in. And the new 'you.com' isn't any better because they're just getting their results from Bing, and they've been caught taking their results from Google!
For one, these top 200 to 500 are always changing. Suppose you keep them static. The sort of things like "spam", "fake news" and so forth are highly opinionated categories. Even if you know objectively that a site is spam, it's possible that it contains the information the user is looking for.
Ultimately it would devolve to full curation, which has been tried and doesn't really work to the extent necessary to justify its own existence.
"Unpinterested!", if you use Firefox - https://addons.mozilla.org/en-GB/firefox/addon/unpinterested...
Google keeps competition away because it is able to generate much more revenue per search than competitors. Worse results translate to worse revenue (e.g. if the #1 result was always the best, why look at the ads?)
If somebody else could make the money work they could definitely give better search results.
I think 60% of my searches are now done with "site:" on google and the results are way better than most of the onsite searches.
For your second point what I know is that there has been 4 decades of research into search relevance ranking and about 0% of that knowledge has made it into real life products.
I went through a phase where I interviewed people at 15 or so search vendors (e.g. Open Text) and not one of them told me that they or their customers thought relevance was a priority. One of them participated in the TREC competition because they thought it would give them some credibility but then they decided it didn't.
If you look at any of those products you find they have hundreds of low-quality "integrations" with various data sources because they believe that's what sells the product.
I'd design it the same way I designed the text editor I'm making ( https://twitter.com/LeapJosh/status/1455511983512436737 )
Hit the minimums. Be irreverent, elevate the ordinary.
Manifest intrigue. If you can't do that, at least make your users feel weird.
Curating content for by subject matter and then aggregation and spidering the resulting content would yield better quality rather than quantity.
Decentralization with cooperative subscriptions would provide a good search population rather than the unfiltered firehose of the current web.
Would help if it was smart enough to show only latest information for your search. For example, no more XML configuration results for your question about modern Spring Boot applications.
Honestly I’d be happy with a google that just let me quickly hide results from certain sites.
What makes it different is business model that aligns incentives and obsession with the user experience.
Turns out they are cheap to build and operate, and and an endless font of interesting engineering challenges.
One side of the problem is to extract keywords from the documents, probably a good idea to start with something along the lines of OpenNLP and use that for TextRank. Well that's roughly what I'm doing now anyway, after having tried a few other approaches. It depends a bit on the nature of the documents I guess. Proper NLP is slow, but dumb TF-IDF goes a long way. (just google these things and you'll figure it out).
If you want full text search, you'll probably want to look into constructing a suffix tree as well. Or just fake it using N-grams, but you absolutely need some rules limiting which N-grams are valid search terms, or their numbers grow astronomically. POS-tagging is useful here if you can afford it.
As for the index: First you need a component that maps keywords to integers. This only needs to be one way, I use a custom memory mapped hashtable, but depending on how much data you've got, there are probably easier alternatives.
Next you need to set up a reverse index that maps words-id to document-id. Again, this is a bunch of memory mapped binary data, I use a mix of hash tables and sorted arrays for binary searches.
That's sort of it.
Then you draw the rest of the owl.
The repo for deploying Solr on Google Cloud is here: https://gitHub.com/kordless/grub-2.0
- HTML parser.
- Crawler.
- Index.
- Constraint solver.
- Query language.
- Ranker.
- Front end.
You can use libraries for as many portions as you like to bootstrap. It's a great personal project.
Niche search engines or metasearch engines like Tripadvisor and Skyscanner are welcome because they come in handy where Bing or Google do not or in another words they are more specialized for certain niches than general internet search engines.