Hacker News Reader

Comment by kristopolous | Hacker News Reader

Biggest advice I can give is you probably don't need search if you're indexable by search bots.

No really. Look over people's shoulders sometime.

They'll just go to Google and type in their search followed by terms such as Wikipedia, imdb, Stackoverflow, YouTube, Bandcamp, Amazon, eBay, Yelp... all sites that spent a lot of time on their search and have done quite a decent job. Oh well.

So unless you really need it for some critical reason where you know your users aren't going to do their regular patterns of going through one of the general engines, close the ticket as out of scope and go home early.

Don't bother implementing hard to do and expensive to maintain features that nobody will use. It'll become more headache than fun real quick.

View on HN

  > Biggest advice I can give is you probably don't need search if you're indexable by search bots.

I don't know how relevant it would be today, but ~2005 or so I had a site that used Google search for the internal "search" functionality. It was a webmaster feature of Google at the time, the google results would be displayed with my site's branding, colours, etc. Only results from my site were displayed, but there was a clear Google logo and link at the top. I think that ads were not shown, but I'm not 100% sure.

A/B testing showed significant (I don't remember the numbers) loss of visitors on that page. When presented with our own internal search, visitors would stay on the site the vast majority of the time. But when presented with the google search, visitors would often leave. I would say that the results that google returned were no less relevant than those our internal search provided, in most cases near identical except for ordering.

So as a casual web browser, I have been guilty of this. IMHO it just breaks the users experience when they feel like they have been swept out to google. Almost like when someone asks a question, and someone replies with a link from lmgtfy

I wish that we had spoken a decade and a half ago ))

This is true in principle, but in practice, the use of 3rd party search has died down over the last decade. This is not the phenomenon of a superior way winning out.

Either search is not an important feature, and a suboptimal, DIY implementation that looks OK is good enough. Or, search is a primary feature and then you need control over it. IE, if you have an online store, travel site or dating app with a search based UI, then you'll probably roll your own.

There are cases where google really is the best way. As you say, stackoverflow, wikipedia & such. Even so, you'll eventually roll your own. Spolsky original UI concept totally leaned on Google for search, but SO still has its own.

At some point, you'll need results to take inventory into account or make autocomplete smarter about tags... and now the headache is yours anyway.

It would have been cool if the web had really developed into the hopeful, "semantic web era" where this kind of approach works. That didn't happen. Half the game is over control, and controlling UIs matters the most.

Also, the power of pagerank has dwindled as the web itself changed. Links aren't what they used to be. That makes Google relatively worse at what it was once best at. Google search as a whole is much, much richer but most of what makes Google good today has less to do with your use case anymore.

TLDR, Maybe google search works for searching wikipedia, which is perfectly loyal to the original WWW concept. Even they have a DIY search. If you work at Reddit though, and your search sucks, google will not fix this. Your search will just suck, and your app will be less usable.

> Either search is not an important feature, and a suboptimal, DIY implementation that looks OK is good enough. Or, search is a primary feature and then you need control over it. IE, if you have an online store, travel site or dating app with a search based UI, then you'll probably roll your own.

Most stores have garbage search and even worse filtering. >98 % of stores have only categories for filtering and only trash like "price" for sorting. In the 2 % of stores that can actually do faceted search the number of facets is too small (e.g. shoe stores which only have a "size" facet plus categories) or the data in the facets is garbage. This is likely a big contributor to sites like Geizhals existing, whose sole purpose is to offer decent search and filtering.

I don't disagree.

If you have bad search, and search is a primary feature, then your ux sucks. I just don't think "just use google" is a viable alternative to making search good.

If you're making an online store, search is probably important, and a major determinant of how well the software works. The best way to handle that is with search that doesn't suck. It may be hard, but that's the job. Fair point that search is not trivial to do well, but that doesn't mean it isn't the job.

Ninety percent of ecommerce sites have terrible search because ninety percent of everything is crap [1].

That doesn't mean that having a great search can't help your shop get repeat customers, just like good service, fast shipping or a good checkout flow can.

1: https://en.wikipedia.org/wiki/Sturgeon%27s_law

> controlling UIs matters the most

Or as former Google design ethicist Tristan Harris wrote, "Whoever controls the menu controls the choices."

https://observer.com/2016/06/how-technology-hijacks-peoples-...

This only counts for public sites with semi-static information.. I've been on plenty of internal/LOB projects where the filtering/searching/slicing of data and exporting to various formats are one of the core features of the application, where the data will also be different as time marches forward (non-static sites). Thousands of non-tech people rely on those kinds of features to do their job. Oh, and some of them are behind firewalls/proxies/vpn and never see the light of day on the public internet, so no search crawlers can see them - and even if they could, they are not optimized for crawlers and/or too dynamic.

Well they said in the very first sentence it only applies if you're indexed by search bots. Internal apps (typically) aren't accessible at all, and quickly-changing data isn't indexed in time to be relevant.

It can still be public and indexable but you will still miss finer grade filters.

I will very quickly stop using a given website if it doesn't provide a good, on-site search. If I'm looking for information that I know exists on a particular website, I see no reason to involve a third-party search engine in the equation at all.

Granted, a lot of sites out there that provide a search function do not necessarily need one, and would instead benefit from an organized, hierarchial index. With these types of websites, using the search function only adds unnecessary friction.

Would it bother you if foo.com took your query and send "query site:foo.com" to google? The results would be foo.com specific.

I often find this gives better results. Eg I'm much likely to find what I'm looking for if I search for "(query) site:reddit.com" than if I use reddit's own search feature.

Much more likely*

I agree with you but many sites use a third-party search engine to provide on-site search functionality. e.g. Algolia's DocSearch

Given the state of search functionality built into websites I use often I can only imagine you have a very heavily curated list.

In my experience so many sites have searches that return bad results, restrict your ability to see a list of results due to some poorly thought through typeahead functionality, or simply do not work without third party scripts and cookies that I often resort to "site:<domain>" in ddg or worse, google.

I find hierarchical indexes only of marginal use when looking for information. I can only know where in the hierarchy something is if I'm very familiar with the website - because many things could logically be in more than one place.

> organized, hierarchial index

Yes!!

So many blogs without a list of posts in chronological order. So many sites without a list of pages. So annoying.

Yes, but a good embeded search makes all the diff.

VueJS and Tailwind CSS both are indexed, but on those particular site I use the web site search _when I'm looking back for the reference of something I know_, because it's faster and more accurate than googling.

Granted, it's rare, but if you manage it, it's great.

Maybe. The point in those examples was that they all solved search in a specific contextualized manner impressively well and despite that people are creatures of habit and they'll want to port the same generalized patterns over that they do for everything else.

Not consciously. They'll just do it and expect it to work which is why effective indexing and "SEO" (as in, the engine can scrape the content and crawl around without getting confused) is likely the actual work to do to implement "search".

From my experience yesterday I can tell you that Zenodo is struggling with reliably repeatably returning results basic boolean search queries. I think I’ll try parent’s search-the-site-via-Google approach.

Agree with your sentiment.

I cofounded ZIR AI to provide ML (vector)-based search as a PaaS solution, similar to what Algolia or Elasticsearch do for keyword searching.

We have a demo (https://zir-ai.com/demo) running over Quanta Magazine articles. Not only can it outperform the keyword search embedded on quantamagazine.org, but one some queries, it even outperforms Google with a site restrict (e.g. how old is the universe site:quantamagazine.org).

> Granted, it's rare, but if you manage it, it's great.

ML-powered search will make this commonplace, once the tech goes mainstream in the next 2-3 years.

I don't see it anymore, but people used to put a google search bar on their website that would use the sitesearch parameter to restrict results to their domain.

It looks like there is a modern alernative - seems a little more complex to get going with which might be why it's not so common.

https://developers.google.com/custom-search

We do it at https://dlang.org

It's not the greatest, as it seems to prefer to return references in D forums in preference to the manual pages, but it works well enough.

There used to be a big yellow blackbox server called Google search appliance that you could put in your data center and get internally indexed white label google search.

Before Google figured out what its business model was.

I was going to say something similar.

It’s not common, but it happens. For small sites it should possibly happen more. If you’re working hard to achieve mediocrity, you should put that energy somewhere else where you can at least get to good.

That's not always true.

I run https://volt.fm and search is one of the most used features. If it didn't have built-in search, I doubt any of the users would use Google instead to find the artists/songs they were looking for.

I'm sure you're right and you've done the analysis. What's the inbound search engine versus your search endpoint as a fraction of each other?

Even further, any ! on DDG is using the site's search so even if we go to a search engine (e.g. `foo !w`), it's possible that we are using the website's search anyway.

I don’t want users to leave the site and use google. That’s bad UX and DX at the same time as it only increases friction and uncertainty.

I want them to believe that if it is there, they’ll find it with search. But that depends on how good it is and how helpful the feedback is.

Yes, when I'm browsing documentation, e.g. for Python, I don't want to leave the context and type "X Python", where X is what I'm looking for. Also Google searches in that case might give all kinds of irrelevant results, like StackOverflow pages (sometimes useful, but not if I want to read just the documentation), ads, and such. Typing "site:docs.python.org" is also out of the question.

I think this very much depends on what kind of service your site provides, but in many cases I think we can argue that the Google fallback is good UX. It means the user doesn't need to figure out and navigate a new/different search system for each service.

If the GP's right and people really do just drop out and use Google every time (even though the target site has a search function), then I think it's hard to argue that it's bad UX. Unless you want to argue that the specific sites' search functions are poorly made.

> Unless you want to argue that the specific sites' search functions are poorly made.

You lose trust in the search function very quickly, if you gain it at all. Whether yours is effective and helpful does not only depend on the search implementation but also on whether your site actually has the information (that it should or is assumed to have) to be effective and helpful. A lot of factors come into this, representation, content, design both for search results and for the actual target pages.

Now if you have the trust, are helpful and effective, then the UX is drastically improved.

For one, search engine results have been degrading. There is _so_much_spam_ out there. Countless SEO sites that just crawl the web and generate crap output to show you ads, some seem to be handwritten as well. It is increasingly probable to get top results that are just presenting "stolen" content in some form or another. You get low effort blog-like posts that are just restating things they read in a discussion that is actually not the primary source, but a generated site, which is referring to the actual information somewhere else.

Secondly if you have in-built, decent search people will use it and they will be happy for it, because you'll be presenting them much better suggestions in a better way and you'll do it faster.

Think of some of the _best_ sites like MDN. Sure, you might enter the site via DDG or Google (in the former case you'll be searching directly like so: "!mdn [search_term]"). But when you're on the site you'll be happily navigating it via links and search, because it's just a very good site.

Other examples of this are: wikipedia, tailwind, clojuredocs, reactjs, hn.algolia...

They all have rich content, good search, useful links. It's a very effective combination.

but if the search isn't good enough I'll just go to Google anyway but now you've annoyed me with your own useless site search, which is what always happens

How is it bad developer experience (DX)?

We don't want to be even more reliant on optimizing for something we have little to no control over right? Maybe this is more of a mindset/approach thing. But I like making things where I can communicate the guarantees and assumptions with confidence.

Note I'm not saying we shouldn't care about search engines. They are extremely useful and important. I'm saying if you have the type of content that benefits from being navigated via search, then consider direct control over this.

Yeah, I suppose there are two ways to think about it: maintaining an outside dependency or offloading that dependency completely.

Ironically I use hackernews's search engine a lot

Also Google search is very "fluid". Sometimes, for example for searching documentation, you need something more advanced.

Is there any actual evidence that might be true? I encountered this argument many times, but only from programmers (who thought it was to difficult because of their tech stack), never from users.

As a user, there is no site that does not have search, no matter how good it is, that does not feel quite horrible because of that. Even the search by Google feel horrible -- romhacking comes to mind. It is also frustrating because some instances of title searches are notoriously unindexable, while a nice site search lets you narrow the context through tags or model specific attributes (system, genre, ...).

The statement is intentionally hedging and qualitative. Some people do the usage pattern, some don't.

The advocacy is to push back on search as a difficult and often unnecessary problem to solve.

The caveat is my increasingly toxic pattern to push back on almost everything as unnecessary. It's probably overly antagonistic.

I'm a techno pessimist programmer. I didn't understand this attitude when I was younger but then again people often become that which they fail most to understand

Formulated like that, I'm very sympathetic to your point of view. Having a small set of well-refined features rather than a growing spaghetti of half done ones. I guess my own point of contention is that programmers would say "task X is hard" when it isn't that much, it is that their technical choices make it hard.

For instance, one Front-End engineer was really proud of using a very recent and trendy framework to rewrite the whole site from scratch. After deployment, people are ordering the wrong products. His reply was, "keeping the query filters in sync is too hard of a problem". It's really not.

An interesting example is the search query “<question about anything> reddit”.

Without the ‘reddit’ qualifier, the results are nearly always spammy and useless (as much as modern Reddit tries to compete here notwithstanding)

Its pretty often that I do a search, get bad results, then add reddit and get good results, so I hope some data analyst in Google sees those as opportunities to improve.

Reddit itself has an annoying dark-patterns UI. You have to edit the URL into old.reddit.com to get a reasonable page to read.

Or as a slight refinement implement it as feeding ("site:mysite.com " + user_query) through to Google, to spare your users 2000 Pinterest hits and 1000 scrapes of a very old Stack Overflow question that somehow matches.

Indeed, there's above board white label options for embedded search from most of the indexers, usually for a nominal fee.

Really depends on what kind of content your dealing with. Search can be as hard as you want it to be.

"Critical" is subjective but:

* Are Google indexes updated frequently enough to capture important changes?

* If you consider certain search terms synonymous, would Google do so too?

* Are Google result rankings compatible with yours?

>They'll just go to Google and type in their search followed by terms such as Wikipedia, imdb, Stackoverflow, YouTube, Bandcamp, Amazon, eBay, Yelp... all sites that spent a lot of time on their search and have done quite a decent job. Oh well.

I'm someone who does this, and I do it because I've found that it either works better than a site's specific search and/or it's so much faster to go to Google and search than to try to find the search functionality of the specific site.

It depends. If your main UI involves search, you might want to make sure that it is actually usable, competitive and not embarrassing.

If your main business is selling stuff that users find via your search features, then doubly so of course since you are literally losing cash every time your search ends up not doing its job. Easy to measure too and most eCommerce companies that survive long enough do that obsessively because it shows in their recurring revenue if they don't.

Also, if your competitors have awesome search and you don't, your users might realize and jump ship. If you have content that is great but nobody is finding it, you might want to fix it by allowing them to find it via better search functionality.

If search is actually not critical to your UX or product, then by all means, cut corners. Google will happily redirect users to your competitors as well. Make sure to give them plenty of money for keywords. If you don't, somebody will. Either way, your analytics will tell you how people come to your site and what they do once they get there.

Either way, it's not that hard to build a decent search experience if you know what you are doing. The key point of this article is that many engineers kind of don't know what they are getting into and mess it up.

I really do not agree. There are many technical blogs that I like and navigating the content on their site is a pain. They will have interesting multipart blog posts and finding all the connected parts requires clicking through multiple pages of 10 entry blog posts to find what I want. Sometimes they will have links between related posts in the blog post if you are lucky. If they are writing about a specific concept, good luck you need to start clicking.

I have never ever been ale to find anything on Wikipedia with search, except if I know the title of the article I'm looking for. But as I often do want the information from Wikipedia, I just search <search term + wiki>, and it takes me straight to where I want to be, whether I'm using DDG or Google. Works so much better than !w

Lately I had to scroll down on Google search results a lot to find the relevant Wikipedia article, it often being somewhere below some irrelevant images, followed by a completely unasked for and irrelevant map (why in gods name would I care for where the nearest factory for a product is?), some random blogspam, and ads.

It used to be that you reliably had the Wikipedia article at the top of your results to provide context and basic information in case you didn't know what your search term means, you could expect it to be there if it exists. Now you have to waste mental energy hunting it down, which is a waste if there's no article at all.

I've lost faith in the ability of the Google approach.

Their results seem to have been turning to trash but then again, so has everyone else's.

There's a few explanations. Easiest one is me, I'm getting dumber with age or have changed my standards. Second one is they are all using similar approaches and SEO'ing has ruined search. Third is Google sets "the standard" and the other engines tweak themselves to follow the goog, regardless of results.

Reality is it's probably all 3 and a few more that I haven't thought of

That does make you dependent on Google not starting to suck. For the first time last week I had the experience of Googling a programming question and realizing the results were so bad I should probably try searching StackOverflow directly. I did and got twice as many hits.

I agree. But I do think there is some room to look in to the details. It is possible that these search features targeted at someone other than the general audience.

And for something like EBay, or Amazon, they'd be mad as hatters to send any search traffic on their website out to Google where competitors are buying ad space. They don't really have a choice, they have to try and implement their own search even if only a small number of people use it. That is likely free money as far as they view the world.

A free text search on your content via a search input is the killer app of a search engine but it is not the only app of a search engine.

A search engine (and its backing index), just like a relational database, is a technology that allows you to build all sorts of solutions on top of it. Nobody says don't use a graph db because Facebook already has the best friend of a friend solution you could hope to find.

Yeah. Absolutely. There's plenty of merits in the pursuit.

But products are constrained by requirements and most of the time the manager thought search was in there, after a little talk, search was not in there any more.

There's resources, deadlines, budgets, contracts to satisfy, blah blah blah. The insight is this costly feature can usually be shaved from the requirements with a little conversation.

A big secret of "10x" programming is realizing "90%" of the work isn't truly necessary.

But yes, search is totally a fantastic thing to do if you have the space and expertise to do it.

Are you saying a customer who came from a search engine should go back to it when in need of searching another product on your site?

It's such a bad practice to let the users leave for whatever reason.

Sites are putting every effort not to leave their site and this is anti pattern.

Also I'd think the site doesn't even enough budget to put a search function and the business or the manager has some problem.

I use DuckDuckGo's bangs[0] to search, say Wikipedia (!w, !wf), Word Reference (!wref, !wrfe), Hacker News (!hn). They are incredibly useful and using Google would be a waste of time for such queries. I'm glad these sites implemented search.

[0]: https://duckduckgo.com/bang?q=

If this is your situation there's a nice site search builder for DDG here: https://ddg.patdryburgh.com/

I use that for my static site as it means I don't need to introduce any moving parts. Probably not the best search experience but it gets the job done

If I think of myself as a user, that is spot on for information/entertainment, but far from the truth for products. The most extreme examples being everyone using "$THING Wikipedia" but nobody I know using "$THING Amazon" in Google search

You're totally right. Especially forum search engines are terrible. This is the best bang-for-buck solution.

If someone has any idea why forum search experience is so bad and know how to improve it, please chime in. I have my hunches, but let's not get ahead of myself.

You might be right in most cases (for example blogs etc) but it's just disturbing on apps like spotify if you just can't find the music you want to listen inside the app. Or when Netflix gives unavailable movies as search suggestions.

Third-party search comes at a huge price for me. Crawler bots kick the shit out of my websites, introducing tons of unnecessary noise and load.

I've chosen to block them all and only spread the word organically instead.

I have yet to begin tackling the search problem :)

Wikipedia actually doesn't want to be indexed by third party search engines. Mediawiki is a heap of underoptimized PHP garbage so pages are (expensively) rerendered every time you visit them.

> underoptimized PHP garbage

rather uncharitable description - that PHP implements dozens of interesting features and recall. It is true that it is a mess, however.

or do it like the w3c - make a search form with action https://www.w3.org/Help/search?q=xslt.

Privacy may be a thing, however.

Yeah, the key part is IF, you are indexable. In the project I'm working on everything is behind a paywall.

Many sites with paywalls still open up their content for search bots.

Reply on news.ycombinator.com