The High Cost of Keyword Search | Hacker News Reader

The High Cost of Keyword Search

blog.zir-ai.com

56 pointssvcrunch45 comments

Elasticsearch, and probably Solr too, has some support for vector searches. https://www.elastic.co/blog/text-similarity-search-with-vect...

I looked into using that last year. In the end we went with a simple more like this query, which proved good enough relative to the model our machine learning people came up with (not so great because real world data is just hard). The nice thing with this is that you can combine vector search clauses with additional clauses and other features such as aggregations, highlighting, etc.

Another way to use machine learning with Elasticsearch would be the more like this plugin which re-ranks results matching a particular query. It's a simple linear regression (usually) based model that you train using features in the form of elasticsearch query clauses. Basically it learns the boost factors for each of the features. E.g. Soundcloud uses this in their search.

Either way, the hard part of stuff like this is testing your system. Going from a "this looks ballpark right" to "our users click on more results by 0.5% after change X", is a long journey. Not a lot of companies are able to do this and I've seen a few teams fall into the trap of having a not so great machine learned model with a flawed tests suite proving not much at all about its alleged properties. A well crafted query can do a lot of the same things. A benefit is that you can fix things easily whereas most machine learning is a bit more like a black box (either works or does not).

Vector search is where the next leap in search and recommender system improvements will come from.

* Is already coming from. Google, Bing, Spotify, Pinterest, Facebook, etc, already use it in production.

All flavors of ES have some kind of vector search available, albeit with some limitations. There are also stand-alone open-source and managed solutions popping up left and right (I work for one, Pinecone.io).

From what I’m seeing, the hard parts people encounter are:

- Finding or developing good embedding models. I consider KPI testing a part of this process, since it tells you whether or not your models are actually “good.”

- Finding and tuning vector search libraries that can meet speed and accuracy requirements. (Easy if you use ES although you’ll run into latency issues with >10M docs. Even easier with a managed vector search solution.)

- Scaling the above to handle millions (or billions!) of items and high query throughput. Again, less of an issue if you’re using managed ES on AWS or Elastic.co, or a managed vector search solution like Pinecone, or you have a team of distributed systems engineers sitting idle.

Amazon search is rather lame. Last week I typed in the exact title of a book. The book way way down in the search results. Before it was what Amazon thought were "better" results.

Please, Amazon, verbatim matches should go first. Then do the AI thing.

Google, too.

This is really important when looking for something obscure.

I've found that this new wave of 'smart' search systems are actually making search results progressively worse. People complain that they can't find anything on Google alternatives, my experience is that search results are equally bad. These new systems start making things up "I bet he meant to search THIS" and there's no way to escape it.

All you get is ads and generic results any time you try to find anything out of the ordinary. I remember that if you were good at formulating your search query you could almost always find what you want. Instead they assume the user is stupid or just want to serve up an ad.

I like the imdb.com search. If I type in the exact movie title, there it is. If I only have a vague idea of what the title is, it'll often suggest it to me.

Ahh but they are also very lucky in that they MOSTLY only search movies,actors and tv-series. 3 data-types.

Yeah, well, when I select "Books" in Amazon's search selector, I darn well expect it to give me results that are "Books". Is this too much to ask? :-/

You will be surprised at how difficult this is. Book titles can be all over the place and mimic product names. For example there is a book called "Apple".

Search is definitely a case of "the devil in the details", each person got their own very "simple" test or search improvement but to make search for everyone "generally" valuable is a very hard tuning problem.

In my past job as head of search for a large price-comparison service, what we build and found work best is to handle search-tuning as the same as test-driven-development.

We will actually write test in the form of "When someone types apple, there should be at least xyz products from category cellphones and their pricing should be between $XYZ--$ABC", whenever we had a manager or a person just walk in with oh search should work like this, we will put it in a test and thus as we tune, weights and vectors we can always see how it impacted previous cases of "Managers walking in telling us, his fav search query is bogus"

I did click on the Books section first. I expect the Books section to restrict its search to Books.

Does it?

Yes, at least for the example of "apple" it does restrict to just books if you choose "Books" from the pulldown next to the search field: https://www.amazon.com/s?k=apple&i=stripbooks

Before it was what Amazon thought were "better" results.

The mistake you're making is that you think Amazon should order results in the way that's best for you to find what you want. Amazon puts the results that are better for Amazon first, based on millions of people's clicks and subsequent purchases. They have no reason to put something first if they know that putting something else first will result in an increased probability of a more profitable sale.

Amazon search is a tool for Amazon to promote things to sell.

They sold me the book I typed in the exact title of. But I nearly gave up and Amazon nearly lost the sale before I found it.

So despite not giving you the results you wanted they still got the sale. That's probably very common. So why would they ever give you the results you want over the result they want to show you?

It’s also probably very common that people give up and Amazon loses the sale.

I suspect it isn't, especially for books. If someone goes to the lengths of looking for a specific book on Amazon I imagine they'll end up buying it there 9 times out of 10 at least. The amount of effort necessary to trawl through the search results for a few minutes is always going to be much less than the effort it takes to go to a different website, or to a retail store.

There's usually some lead time between becoming an annoying nightmare and another service getting traction and then before the avalanche hits.

But when it hits nobody should believe it when someone said "it all happened so fast".

Because this is a long time coming for both Google and Amazon.

Which will prove that with enough effort even the mountains of goodwill that Google and Amazon has kept piling up for it's first decades can be burnt through with effective (ab)use of bigdata, ai and "UX".

Amazon and Google have replaced mountains of goodwill with mountains of cold, hard cash that they can use to either buy the competition or pay engineers a lot to build just-about-good-enough versions of things that will keep people from leaving to go to the competition. Either way, competing with Amazon or Google in their main profit centers (retail and cloud in the case of Amazon, or ads and search in the case of Google) is virtually impossible unless you're an already exceptionally big company like Microsoft.

I don't believe any company that doesn't already exist will ever compete at scale with Amazon or Google. Competition in those areas is essentially finished. I would love to be proved wrong though.

Straight up competition - a marketing company built on search or ecommerce for everything - probably wouldn't. The company that takes them down will come from a side angle and might not even be recognized as a competitor. Amazon itself is an example of this. It wasn't too terribly long ago that Walmart ruled retail and Amazon was a niche little company selling books (online!). Whatever takes down Google and Amazon will be something similar.

> I don't believe any company that doesn't already exist will ever compete at scale with Amazon or Google.

People said the exact same thing, with the same conviction, about IBM, Microsoft, Walmart, etc.

it means it is better for Amazon to lose some sales to promote something else.

For all I know, you could be right that Amazon search is garbage, but the anecdote you've presented doesn't prove it at all because it suffers from selection bias: If better (or "better") than literal search protected you from poor results 100 times as often as they frustrated you, by definition you'd never know about all those other times.

Point is it used to be better.

Same with Google.

Today there has been multiple breakthroughs in a number of relevant fields but yet we see quality declining in obvious ways.

I understand that sometimes one needs to take a step back to take two forwarded, but in Googles case they are now in their tenth year after taking 5 steps back and at some point the new solution has to prove itself.

Sadly I guess the old solution is "lost to time", that noone is able to replicate it anymore. It is the only explanation I have why it hasn't been reverted.

>Amazon search is rather lame. [...] Please, Amazon, verbatim matches should go first.

To see an illustration of how bad Amazon search for exact products is, here's an example.

I want the newer 2020 Milwaukee 2161-21 flashlight and not the older 2017 model 2110-21. This is the manufacturer's product page: https://www.milwaukeetool.com/Products/Lighting/Personal-Lig...

Screenshots of searches:

- amazon.com[1] search for "milwaukee 2161-21" gives the wrong item 2110-21: https://imgur.com/a/2lrHb0I

... but I can go to Google and append "amazon" to find what I want...

- google.com[2] search for "milwaukee 2161-21 amazon" : https://imgur.com/a/afpJSgF

So Google's scraping of Amazon's shopping pages returns better results than Amazon! It's ridiculous.

Lest one thinks that there's an Amazon profit motive to show an older 2017 model that deliberately doesn't match the customer's search query, I'm not sure about that. In this specific case, both products are sold by 3rd-party resellers which means Amazon gets the same 30% fee either way.

And as more trivia, both HomeDepot.com and Walmart.com return the correct result for "milwaukee 2161-21".

[1] https://www.amazon.com/s?k=milwaukee+2161-21

[2] https://www.google.com/search?q=milwaukee+2161-21+amazon

> Last week I typed in the exact title of a book. The book way way down in the search results.

I've had good results searching for the ISBN rather than the title.

> I've had good results searching for the ISBN rather than the title.

Apparently you're not one of the people who finds the ISBN by looking it up on Amazon. (joke)

> This is really important when looking for something obscure.

Sadly I do not think that Amazon goal is to optimize searches so you get what you look for, but so as many people as possible find something that they purchase.

There is many complains in this threat about how bad the "smart" searches are becoming. But, the reality is that "smart" searches are becoming better and better at their goal: to maximize profit finding something for you to buy/click/etc. regardless of what was your initial search.

e.g. If you look for a 3$ book that gives Amazon a profit of 50 cents but they know that they can show you another that you are going to buy for 50$ with 2$ profit Amazon will show you what is right, for their own profit. If the book is just in a similar topic or not useful at all to a minority of people, that does not matter meanwhile it maximizes profit.

Specifically, using these numbers (50 cents and 200 cents), as long as only 5 % buy the more expensive item instead, they still make a profit even if this type of profit engineering turns away you and 15 % of users like you leading to no purchase at all.

In exchange for a small chance of selling a more expensive item, it can actually be worth running an even greater risk of losing out on a purchase of a cheaper item. (It sounds obvious when put like this, but people I've talked to have been surprised by it.)

Of course, this is all about short term effects. There's a cost for your brand and reputation if you become the place where it's hard to find what you're really looking for.

----

This is is hard. I work with providing product search for online stores and our service is one of the better in the business, going by a solid track record of controlled experiments against competitors.

But like -- how do you know you're better? The user making a purchase is a really strong signal. Many people would be envious to have such a good signal as "person opening their wallet." It's about the gold standard when it comes to assessing the opinions of people without asking them directly.

Except... as stated here -- what if they didn't actually find what they wanted, only we managed to figure out something else they might as well buy while we were at it? Is that success? Partial success? How can one tell the difference?

You should try the search on Amazon Prime. It might as well be a random title generator. Or they made default configuration to invert search results. It's just so so so so bad.

I stopped searching on Amazon for this reason. It's optimized for selling me crap based on some probabilistic machine-learning thingy that's trying to guess what items have the highest likelihood to make me spend money.

Amazon no longer had the edge in terms of cost nor the exclusivity in terms of inventory. I still occasionally buy stuff on there when they do have a price advantage, but that's becoming rarer and rarer.

eBay still has one of the best searches if you know exactly what you want as they support more complex queries, eg.

    (adidas,puma,nike) high top 11 -womens (black, red)

But they don't have everything of course.

More and more I find myself using Google Shopping then buying at some specialized site. It may take 3 days instead of 2 to come to be and it may require I type in my payment info but many support Apple Pay and for those who don't, I memorized my credit card number making checkout very fast.

>Please, Amazon, verbatim matches should go first. Then do the AI thing.

I don't think so. If I search for "animal books" I probably want popular books about animals, not a low rated product with the exact name "Animal Books." Besides, that makes it highly game-able, no? What if I title my book Horror or Sci-fi?

Its not as simple as you say.

Want to see the worst search anywhere on the web? Try this site for movies:

https://letterboxd.com

Here’s a search for “Parasite” with a one-letter misspelling:

https://letterboxd.com/search/Parisite/

The results are so comically bad for anything that isn’t an exact match that I think they are just random.

It’s such a great site. I don’t know how its search is so awful. Letterboxd users joke about its search awfulness. I think the developers must just be punking us. Really.

> Whether your objective is to improve user engagement, or you simply want to help your users quickly find the most relevant information, you pay the high cost of keyword search every day in the form of missing and irrelevant search results.

At the moment my biggest frustration is that both Google and DuckDuckGo keeps ignoring my keywords and keep jamming my results with all kinds of obviously irrelevant data.

Google having a "verbatim" option that they ignore on top of already ignoring doublequotes after having butchered the plus operator adds insult to injury..!

There's probably some UX research that indicates that in complex scenarios it is better to give the user the impression of control while silently doing tje right thing.

Well, guess what: that depends on you doing the right thing, and the right thing is not to stuff my results with pages that doesn't contain the error messages I was looking for!

At least on DDG, if you use quotes it will give you a "no results" page before it includes anything that doesn't have it.

It used to also place the results that match all of your keywords first. They stopped doing this a few years ago, but they didn't mess with the quotes (yet?).

> At least on DDG, if you use quotes it will give you a "no results" page before it includes anything that doesn't have it.

For some reason I don't get that except for extremely obvious cases. Example: https://duckduckgo.com/?q=%22lego+sauce%22&t=fpas&ia=web

I recently make a vector search engine demo[0] that uses neural networks (CLIP) to encode text and image into vector. It runs surprisingly fast consider its hosted on a 2 core virtual cloud running both the text encoder and doing the vector similarity search over a 500k index results for both image and website links.

[0] https://unisearch.cloud

Suppose I would want to run something like this on corpus of documents I have and I'm not really a machine learning expert, any recommendation for a library/boilerplate/starting point?

This tutorial uses an off-the-shelf ML model (Average Word Embeddings) and a managed vector index for document search: https://www.pinecone.io/docs/examples/semantic-text-search/

Even if you choose another model or method for storing and searching through the vector embeddings, the order of operations will be approximately the same.

I haven't used it yet, but https://github.com/semi-technologies/weaviate is a vector search engine that does embeddings and search. I don't know if they have succeeded, but the goal seems to be to make embedding and searching vectors easy enough to not need to be an ML expert.

The algorithm reminds me of Soundex.

https://en.wikipedia.org/wiki/Soundex

> A great deal of research has established the importance of relevance in engaging and satisfying users.

Too bad then that this is not the goal of web search engines. Their goal is instead to sell ad space.

This is only to get worse if more people switch to voice-based interfaces (Alexa, anyone?). The way people search when they type vs the way it's done with voice commands can be quite different.

Good point. If the actual result you’re after is the 5th, you’d have to _listen_ to 4 bad ones which costs a lot of time and will quickly lead to frustration - a bad UX. The beauty of text is that you can quickly scan. The beauty of voice is that you can provide input faster. You’d need a voice / visual (glasses?) combo to overcome that and combine the best of both worlds.

Yep.

On the other side, voice based interface makes it easier to supply context - for example, am I searching for a book called "Apple", the fruit "apple" or the company "Apple"?

A (future) AGI-based search engine can go even further than that, and generate the answer for any problem based on the knowledge it has ingested.

Reply on news.ycombinator.com