Hacker News Reader

"next big idea" is probably "selectable algorithms".

Create 3-5 different search algorithms, with names and understandable characteristics, then let me choose which one when searching. You may have some algorithm that prioritizes recency, or rapid changes, or more interaction, and others that prioritize longevity, citations/links, others that prioritize that type of information - prioritize regional or noteworthy events over historical info, etc. Have some algorithms that penalize spam/linkfarms more heavily than others. The git/so spam farms that come up when I search for an error message are simply useless, and would love ways to switch away and try other algorithms, without necessarily having to switch entire search systems.

Have algorithms that explicitly take in to account the search/click behaviour of others. Let me choose to have the search behaviour of friends/colleagues/famous-folks explicitly included or excluded in search results ranking.

Let people easily switch between multiple algorithms, and let 1000 gardens bloom (or something like that).

Also, determining better ways to 'search' non-text stuff. If virtual worlds become a thing, finding ways to 'search' those (whatever that might mean) will be big.

Image search still has a long way to go, IME.

This could be a niche search engine for advanced users, but I feel that to average users this would be a strictly worse experience because they have no interest in figuring out how to "select an algorithm." They expect to type a query and the result they want comes up at the top. The search engine should be able to infer whether the user wants results that are more recent, more interactive, more authoritative, etc. based on the query, past search behavior, and anything else that is known about the user. If that isn't happening, the search engine needs to be made "smarter."

The way it's likely seen by many tech companies is that the more decisions you have to offload to the user, the less advanced your system is. An old car with a lot of knobs and dials and levers is not more advanced than a self-driving car with a single control, select destination. Some people will still prefer the old car, but it will be a niche market.

I agree, it’s the classic example of “people don’t really like choice”. When you are presented with the option of a search algo, there’s always this feeling that you chose wrong. It’s an additional thing to think about, and makes things more complicated.

it doesn't have to be an irreversible choice. clicking between 'filters' should be something that takes an instant, and is one click away.

3 big buttons at the top after search results

"You're viewing through FOOBAR lens. Click here to view results from another lens".

Doesn't have to be that complicated.

To some extent that's already implemented in the form of the various tabs - if you want very recent results, go to the News tab or click the Tools button and select a time range. If you want academic results with citations, go to Google Scholar. If you want information from books, go to the Books tab. If you don't know/care, the main tab will mix results from all tabs.

You can add three more buttons on top of the tabs that are already there, but would their value for more advanced users be worth the extra friction/confusion for average users? And how would it scale if someone wants a fourth, fifth, or tenth "lens"?

> Image search still has a long way to go, IME.

Try images.yandex.com: it's vastly better than most others. It's not too good at identifying specific people but it will find prior exact uses of an image (like TinEye) but is especially very good at abstracting what's the abstract subject, color, pose, setting etc in a photo and show you similar.

Also Bellingcat recommends Yandex: https://www.bellingcat.com/resources/how-tos/2019/12/26/guid...

I agree with you on this. I search for tech topics a lot and google offers an option to search "scholarly articles". I like that option as sometimes a tech term gets confused with something else. It would be great if there was a way to select from some check boxed, like an advanced search, "search only for semiconductor datasheets" or "search all Jazz music" or upload a picture and "identify an insect" (agree that does not work well yet) or whatever someone might be interested in.

The question is if such an approach would lure a lot of people away, I could see an entry angle might be a front end to google with an app that does that.

This is what Brave Search calls “Goggles” [1].

[1] https://brave.com/static-assets/files/goggles.pdf

EDIT: adding on to previous post.

We now have a couple digital generations of folks who've grown up with 'web', and we've generated billions of 'documents'.

TIME ACCURATE QUERIES need to be mainstream.

"Let me search for FOOBAR and get results that would have been accurate/returned for Dec 11, 2006. Now search for FOOBAR and get results from Dec 11, 2014", etc.

It seems the problem of "search" is mostly well solved. What isn't well solved is the problem of "SEO gaming". Perhaps the advancement in search isn't by having an AI that chooses a better page, but rather an AI that is really good at identifying a good result from a gamed result.

However as for Google, I feel like even merely surfacing a range of relevant filter options can help their search engine, since it's what informed users are already doing by structuring a query when the first one produces crappy results. E.g. These results mostly contain links to Pinterest.com, would you like to repeat the search with these omitted? (or even just being able to collapse these site-by-site.)

Additionally I think modern search could be broken down into a range of sub-tools that exist beyond google or even online. One such example is searching our web history, perhaps browsers could produce a privacy-preserving way of storing information about each webpage and its images that the user can then search later (by either terms or images.)

Read an article which mentioned that Google degraded the search service because good results no longer made financial sense, perverse incentive is that it benefits them better for a customer to pay money for ads than provide a good search (meaning that competition for keywords is now the algorithm). The next innovation in search then necessarily must be to find a different funding model for the engine besides ads, otherwise the same perverse incentive will arise.

This hits the nail on the head. I feel like Google results have gotten worse not because technical problems need to be solved, but because the financial incentives means that I'm now scrolling through tons of ads, little "tidbits" that have just gotten more annoying over time, AMP carousels, and god forbid I'm searching for anything that happened over 20 years ago that has any slight resemblance to a current event - all I get are pages and pages about the current event, and Google has "memory-holed" the past.

IMO, all startups have a sweet-spot where they've figured out what the market needs, and they can be profitable, but they haven't yet sucked that bone dry to where they're forced to extend more tentacles to grow their revenue. For Google, for me, that was about the 2007-2012 time frame.

Any other competitor would have the same financial pressures eventually. Long term, it's impossible to "not be evil".

That's ridiculous. Google's search dies if people stop using it. They stop using it if the results are poor. There are no perverse incentives here unless you believe Google is hoping to cash out tomorrow and close its doors.

It's not ridiculous at all.

> They stop using it if the results are poor.

These are not laws of nature. There are so many reasons why most people would still use $X even if the performance of $X was underwhelming.

The quality would have to degrade dramatically for people to stop using Google em masse for that reason. It can still degrade a bit to increase revenue.

There’s a potential technical improvement with Vector Search. Specially for the type of queries that are more “human” (“how do I…”, “where can I…”.

At least for me, I find myself adding “site:Reddit.com” all the time I need a good answer and not just a SEO-tricked ad page. Vector Search could be a technical solution for this. Although it’s fairly new and there is still a lot of research to do.

I like the idea of "crowd sourcing". When the SEO-tricked ad pages come up and I either scroll past them or, if I do click on them but come right back to the search results and continue scrolling/clicking ... rank downward those brief or passed-over links.

Seems like with enough actual humans I should never have to see another geeksforgeeks page come up in my search results.

Heh, Geeks for geeks wasn't that bad when I frequented it a few years back.

Yeah Google does this already though

> What's the next breakthrough idea that extracts the good signal from all the noise of a trillion web pages?

Here's another recent Show HN that demoed a new search engine:

Show HN: Goopt – Search Engine for a Procedural Simulation of the Web with GPT-3

https://news.ycombinator.com/item?id=30443738

I know this was offered as a bit of joke. But it prompted a really thought-provoking discussion and seemed to me to suggest a genuine paradigm shift in search or information organization. It is a novel way to extract a useful signal from all the noise. It brought to my mind "Dr. Know" from Spielberg's AI movie:

https://www.youtube.com/watch?v=x0QkgAuEPbk

Will the signal be a good one? It will need to get better and inevitably will. The thing that worries me: will it even matter?

It's not going to be 'search' that displaces search, it'll be impossible. It will be made irrelevant through something else.

For example, someone comes up with a 'Finder' thing that's barely a comprehensive directory of the internet, and that, along with 'site search' aka for FB, Amazon, does better for the use case.

Most people are not 'searching' the web, they're often typing URLs or searchers for things they want answers to.

When I type a technical thing I'm really just searching 'stack exchange'.

On mobile, we use apps, which are different than pages.

Apple could come up with something that renders Google less than necessary.

Google will not be replaced directly.

Whatever Algolia is doing, it is amazing. Would it be possible for them to scale as a general search engine for the web?

> One thing a competitor needs is a new and innovative technical algorithm.

Not really, because Google can do algorithms. For a new competitor to really succeed they have to start with something Google can't do. Perhaps:

- device/platform related. Google owns chrome & android but a competitor could emerge somewhere out of their reach, like only on iPhone, only on airpods, alexa, oculus, etc.

- being worthless to 99% of the market to make it 10X better to a 1% audience (e.g. developers, nurses, or even more niche like college athletes, jewish moms, etc)

- privacy/ad related

The only viable alternative to PageRank I know of is social search: show me results that my friends (and friends of friends and so on) liked/bookmarked.

Delicious sort of tried it but never executed well on the search side.

In a way it's a personalized PageRank algo with users linking to pages they like and to other users they follow. This always made more sense to me than a global authority ranking.

When looking for a babysitter you want recommendations from friends you trust, not some global ranking of popular babysitters.

I've heard this a lot. I think there are two problems with this. First, you need massive adoption for the "social" part to be at all useful, hence only the few big social platforms can even try it. Second, even at maximum possible adoption, you'll probably run into the problem that actual human social networks have nowhere near enough branching breadth to help rank the behemoth that the web is today, i.e. for most queries and most possible hits, there's not enough signal coming from the social network.

The benefit of this approach is that it's an addition to PageRank, not a replacement. You don't need massive adoption of the social part I think cause you can link to eg HN, which already links to other sites. It's simply extending the backlinking concept to add personal pages.

Most long tail queries work well enough with Google and indeed most of the benefit of this approach would be in the head where ranking matters more. Furthermore nothing prohibits users from liking domains, not just individual URLs.

> What's the next breakthrough idea that extracts the good signal from all the noise of a trillion web pages?

To me this is going to be manual ranking, but the biggest issue with manual ranking is not that you have to do it manually, but how do you know if you can trust users who rank pages. The Wikipedia system is a good start but not enough, and the second issue is how do you prevent its business model to kill its quality

You let the rank sources be selectable. Allow me to connect/select individuals or groups to include/exclude.

> counting back links

Do the algorithms weight links based on visibility or relevance? A link at the top of a sorted content site HN should probably be worth more than one at the bottom, and one buried in invisible metadata somewhere should be worth 0.

PageRank is a neat algorithm that recursively weights incoming links based on their own incoming links. So a link from HN has more weight than from a small blog. And how does it know that HN should have more weight? Well, because there are many important sites pointing back to HN. And how does it know that those sites are important? And so on...

But AFAIK Google no longer uses PageRank, or at least not the same version as we know it.

DDG also lost the plot when they started censoring results based on geopolitical opinion.

In practice, it used to say that 2+2=5 and they blocked that. How is that wrong?

Evidence?

https://www.nytimes.com/2022/03/11/technology/duckduckgo-rus...