What unsolved problems would a next-gen search engine potentially address?
What if you could have a conversation with a search engine, asking follow-up questions within the context of the previous ones?
All search engines need to make concessions. The tricky part is identifying the sorts of content that humans want to find.
* Confidence ranks you can interrogate. "Why did you rank high?"
* on the fly refinement of specific hit terms. "Find more like this"
I think uBlacklist is a good implementation.
Braves proposed “goggles” is better, but as yet I don’t believe there are any implementations of it. [2]
[1] https://github.com/iorate/ublacklist
[2] https://brave.com/wp-content/uploads/2021/03/goggles.pdf
If you think about it, search is an emergent sector that developed because there was no standard for sites to index themselves. Compiling that index and providing that would be far more efficient than trying to crawl billions (or is it trillions) of pages independently by multiple individual search engines, and would remove most of the incumbent advantage of such platforms. Instead we ended up with a few pieces of this --- robots.txt and sitemaps --- but not a full solution.
Yes, you'd need to check for cheats (sample pages based on the index and look for signs of stuffing or misrepresentation, penalise accordingly).
Reputation systems were based on inbound and outbound links, so those might also be offered.
I'm not entirely sure this would work, but I'd be interested in seeing the concept fleshed out further.
Rather than transferring the entire website, it would only be necessary to transfer the index itself. Mind that a sufficiently-large ngram-tuple index is itself large, I suspect limits to 3-word ngrams might be necessary, and should be reasonably sufficient.
I'm kicking around the prospect of a DNS-like query mechanism, with cache-and-forward capabilities. That could both distribute search and indexing load.
Reputation assessment and management remains the hard piece. Early computer science focused on the relatively easy problem of string search. At Web scale, the problem is of idnetifying reliable and useful sources. That itself isn't a static problem with a single solution as the landscape evolves in response to methods.
Search engines that filter out websites that don’t follow a certain code of conduct.
Search engines that are local to a certain area.
Search engines that are completely open-source.
It should still feature search that finds websites or categories in the directory, but also allows to explore from these points and discover related content.
User-definable ranking and exclusion criteria would be great.
In the past few weeks on HN we've seen articles on:
- How universally reviled autoplay video is. Excluding sites which do this from SERPs would be a strong incentive.
- Paywalls and privacy invasions, similarly.
- Known SEO-baiting sites, e.g., Pinterest and Quora.
- Excessive JS.
- User-hostile designs.
Simple site quality and reputation would be a huge factor for me. I've increasingly taken to searching specific sites rather than general Web search just to be able to cut through the crap.
Suggesting filters to apply might also be of interest --- say, "X filter returns / removes Y results".
As noted in an earlier comment: establishing a self-indexing search standard, which would allow websites to create their own indices, and for those to be distributed to multiple search platforms. (Yes, cheats would need to be identified, and mechanisms for establishing reputation determined.)
Better metadata search, and inclusion / exclusion by category, would be great. Search exclusive to scientific, technical, or academic sites, or exclusive of commercial, paywall, erotic, gaming sites, for example. (Such filters could of course be reversed if that was your kink.) Inclusion/exclusion of social media is probably another big one.
Tools for leveraging site-specific search more cleanly, inspired by DDG's Bang! searches, might also be interesting.
Really good date-ranged search. Google still beats DDG at this, though DDG can at least filter by past day/week/month/year, which is useful.
Search inclusive of the Internet Archive's Web and other holdings would also be great. A search not just of Web space but of Web time.
A true news / magazine archive search.
Governmetnt records searches.
> Better metadata search, and inclusion / exclusion by category, would be great. Search exclusive to scientific, technical, or academic sites, or exclusive of commercial, paywall, erotic, gaming sites, for example. (Such filters could of course be reversed if that was your kink.) Inclusion/exclusion of social media is probably another big one.
I would like to do this, but I'm not quite sure how I'd implement the labeling. .edu and .ac.xx works for academia for the most part, but the rest is a lot harder. Could possibly be done approximately using graph adjacency, but I don't know how you'd get a true-false value out of it...
Example: Wired apparently can't properly distinguish authors of additional articles and the author(s) of the principle displayed article in a manner that Pocket could reliably parse, or at least wasn't doing this at some point in time. So my saved Wired articles were listing a dozen or two authors, most of whom had nothing to do with the article in question.
My experience is that quality data isn't provided unless it becomes critical for function and/or revenue, and most likely the latter. If bad metadata earns a search penalty and whatever site is penalising has a sufficient impact on search results, then you'll see shifts.
A minority search platform won't be able to do that initially, though if its results are useful, it could easily capture a large share of the high-value search community, which would give it outsized impact even with small usage.
(Think of Apple's position as a hardware provider to a small but affluent market.)
For example, only about 25% of the sites I index use HTTPS. If you, like Google say "I want to promote HTTPS adoption by only indexing HTTPS sites", you are throwing an awful number of babies out with that bathwater.
So: make it hurt a bit to do stupid stuff.
Fighting spam through standards compliance and elegant site design is not the principle account. There should be other indicia of spam which are readily apparent, and used.
But any site, spammy or otherwise, which flagrantly abuses dark patterns, should pay a penalty for doing so.
Of course, this inversion only works if you're an underdog. Big search engines are fucked by their own success.
I suspect that the pressure will be more effective than you're portraying.
Few people hand-tool webites. Most use tools for generation and management, with Wordpress accounting for a tremendous portion of the Web itself.
Tool builders tend to aim for utility, particularly with encouragement.