Comment by slimsag | Hacker News Reader

Comment by slimsag | Hacker News Reader

I think 'general purpose' search engines are often doomed to fail, and this extends to 'web search engines' (which, in reality, do not search the full web and are NOT general purpose.)

This list makes a great list of attempts at general purpose search engines (I would include DDG and Bing) which ultimately fail and are not what the masses want.

There's also precedence for how only domain-specific search engines are valuable / make sense these days: Google itself is shifting to ML-based answers (attempting to build the holy-grail of general-purpose search engines), and whatever you do isn't going to beat Google at that - even Google hasn't been successful at it yet.

Examples of real-world domain-specific search engines today:

* Image, YouTube, Maps search

* Amazon search

* Code search (Sourcegraph, cs.github.com, cs.opensource.google, etc.)

* Facebook, Twitter, Reddit, TikTok, LinkedIn search

* Documents search (iCloud, Drive, etc.)

* Messenger search (Discord, Slack, Messenger, etc.)

It's really telling how Google has largely only achieved general-purpose search in their domains of data (a Google search turns up YouTube videos, Maps locations) but isn't even complete in that area (Drive, images, etc. don't show up in Google results)

General-purpose search is simply non-viable these days due to (a) data silos and (b) you need to have domain specific search to provide an edge (e.g. regex for code search, or drag-n-drop an image for image similarity search)

So, I'm an electrical engineer (these days). I work with parts. Parts have part numbers. I need information about parts. Some of my parts are weird and rareish.

It is shocking how many searches I do which are well described by "show me every single thing on the entire Internet which matches this string". (I'd love to match a simple regex, since part numbers have stupid trivial variations, but I'll take what's actually feasible.) Or, I often want to match only PDFs, but that hurdle seems well solved these days. There aren't that many results for a few of these things (and those are often the ones I need information about the most), so I sometimes really do want to see every result.

I don't see any other way to do that than a full-text, full-web index. I just don't. And my job gets a lot harder if that class of search isn't possible anymore. So I'm desperately hoping these indices remain at least somewhat viable.

And yes, I am noticing that Google is lovingly correcting my long part numbers that do actually match a fair bit of results to crap that is useless. Fuck that bullshit, Google.

Google is running behind NLP(Natural Language Processing) causing problems like this. I feel we should learn to use search engines and not search-engines trying to guess what we want to find using some weird Black Box AI

I have had this problem for years in modern user interfaces. Most of the times, when a system is trying to guess what I want to do or find, it guesses wrong.

Indeed. Search got along just fine before NLP. You should at least have the option of a "raw" text search.

I'm guessing the following: to index a long tail of infrequently-used keywords (such as part numbers), it takes a lot of storage costs, with little if any financial return.

This is exactly the problem of "general purpose" search engines as mentioned by GP. In classic information retrieval theory, a search query is a textual representation of the a user's mental search intent. A search engine's task is to interpret that textual representation in order to serve the user's need.

Basic NLP starts with simple techniques like stemming, synonyms, typo correction etc. But many users expect more, e.g. contextual disambiguation or broader synonym/similarity resolution. These users would see a search engine as dysfunctional if it could not even find pages in which a plural form or a spelling variation of a query term occurs.

However, there is an inherent contradiction to those users who want to find a specific string. Google, Bing and others apply complex algorithms ("weird Black Box AI") in order to interpret the user intent and the meaning of web pages in order to optimize towards the majority of internet users. To some degree, they do try to also serve users of (large enough) minority use cases in which the user feels like they knew exactly what words (strings) they were looking for. But again, the different use cases pose contradictory interests that are probably not resolvable in a "general purpose" approach.

I follow that, but I still have to ask: Why does it often ignore the quote marks it suggests to me? It's supposed to search for that exact string, but it never does and shows the same rubbish regardless. The quote marks should resolve the contradition between synonym searching and specific strings and they don't for some bizzare reason

Because their products are a complete fucking MESS.

Quotes used to be an exact search, but it now overrides that basically everywhere and continues to search for what it thinks you mean (and has for like a decade now). Even though the "Missing: ___ ‎| Must include: ___" popup just wraps that word in quotes in the search. Which is infuriating, since they know damn well adding quotes won't actually search for that word.

In reality, you need to click "tools" -> "all results" and change it to "verbatim" which does at least still appear to work... for now.

Today I learned... Thank you!

>Basic NLP starts with simple techniques like stemming, synonyms, typo correction etc. But many users expect more, e.g. contextual disambiguation or broader synonym/similarity resolution. These users would see a search engine as dysfunctional if it could not even find pages in which a plural form or a spelling variation of a query term occurs.

Any morphological variation would already be covered with stemming alone, so this particular point doesn’t speak in favor of more advanced interpretations, does it?

Having an path which uses advanced AI powered tools doesn’t necessarily imply one should drop any ability to search with more technical query interface for advanced users. And it’s doubtful big companies wouldn’t have the budget to maintain even two completely distinct tools, would it be required.

pose contradictory interests that are probably not resolvable in a "general purpose" approach.

How are they not resolvable? If Google respected quotation marks (as it used to do, in the old times) that would be enough to accommodate both kinds of users (or rather both kind of search needs - I'm the same person and do both).

Came here to say this. Quotation marks were hugely useful.

> actually match a fair bit of results to crap

Do you use advanced search features? Like quote things that MUST match. ext:pdf etc.

You can have this interface to help you: https://www.google.com/advanced_search

I mean, I'm able to get to where I'm going, between Google and other engines.

It's just annoying as fuck to see "Did you mean 'apple orchards in Siberia'?" for querying "ADS124S06IPBSR". Because that shit IS NOT the same.

(Note for the pedantic: I made up that example because I can't remember a real one. But it gives you an idea.)

Full text search on the entire web index is not feasible in terms of latency and compute cost

Is it feasible to roughly categorize websites (or parts of websites, e.g. subreddits) and then perform full text search on them only, or at least establish priorities? For instance, it's extremely unlikely OP would find anything useful about part numbers on *.reddit.com/r/funny.

Does anything like that exist today in usable form?

It's interesting you ask that, because the now-paid Kagi Search has a feature like this called Lenses: https://blog.kagi.com/kagi-features#:~:text=and%20Google.-,L...

I've found them somewhat helpful in getting rid of some of the garbage that pops up when I'm searching for a specific topic. But often, I find myself leaving them off since I sometimes want all the results on anything even slightly related because it might be useful.

This looks awesome. Unfortunately the pricing is like an order of magnitude higher than what my broke ass can pay for, but I'm truly rooting for them.

Sure -- you can restrict the search to websites that seem to be relevant to the query. Maybe you do it by hand at first, then you switch over to an ML model. You optimize the ML model for a few years, adding in more signals, and allowing it to guide other parts of the query to keep costs in check and raise quality (as measured by your preferred metrics)... and now you have reinvented fuzzy search that behaves in unintuitive ways, and people on HN complain that your service was better when it did raw string matching.

This is what I like so much about you.com They show results from specific, popular sites/apps (I listed some examples below) plus allow you to set preferred sources to see sites/apps you find useful (for me it's reddit) higher/on top of your search results. I find it especially nice for coding cause sites like GitHub and StackOverflow are supported.

https://you.com/search?q=python+pandas+concatenate+two+dataf...

https://you.com/search?q=How+to+care+for+orchids

some supported sites:

- GitHub - Reddit - StackOverflow - Arxiv.org - TikTok - LinkedIn - W3 Schools - Twitter

Is this the one where you had to install an extension first to search ? And the CTO/Owner had a fight with everyone on HN ??

link or didn't happen

I do a lot of those specific searches with hash bangs through DuckDuckGo.

I would love a general purpose search engine. Most websites / web applications have bad search, and behave inconsistently. So I use site specific searches through DuckDuckGo.

The issue is that some of these domain-specific search engines kinda suck on their own, for example reddit search is notoriously bad. This is often overcome by searching what you want in a general-purpose search engine and appending "reddit" to your search query.

> whatever you do isn't going to beat Google at that - even Google hasn't been successful at it yet

Why not? This seems exactly like the sort of situation where an upstart could be very successful: an industry transitioning to a new approach.

Reply on news.ycombinator.com