Hacker News Reader

Comment by moralestapia | Hacker News Reader

How to build a good search engine:

1. Actually return the results matching the words that people typed in your search box. The more they match, the more they go up.

More and more, it has become extremely disappointing what you get back from Google (and others as well). Verbatim search (where you surround exact terms in quotes), seems to have vanished in the last year or so. More often than not I have clicked on a result, only to find out that my particular query is nowhere to be found on the site, but some "related stuff" does.

View on HN

>1. Actually return the results matching the words that people typed in your search box. The more they match, the more they go up.

As virtually all early search engine developers have found out the hard way, it's extremely easy for website owners to cheat such a basic algorithm by spamming keywords.

But the answer to that shouldn't be to return results that don't even contain the words searched for.

Typically, turning on verbatim helps with that.

Where does one find that in DDG and Google?

Not sure about DDG. In google, it's under tools. Click the "all results" dropdown.

One of the reasons I finally gave up Google a couple of years ago was that verbatim hasn't worked reliably since maybe 2015 or something.

Of course, around that time Duckduckgo started going downhill as well in this regard.

> the answer to that shouldn't be to return results that don't even contain the words searched for.

This is everything that's wrong with search in a single sentence.

First, verbatim mode should actually work when you're sure this is what you want. But as a default I think this would produce bad results.

"What's the name of the plant that that eats flies?"

This should, and does, produce results for Venus Fly Traps but I doubt any useful page contains all of those words.

But why should that return results for Venus Fly Traps? You are search verbattim for the sentence "What's the name of the plant that that eats flies?". I'd say it's a bug to return any result that doesn't have that specific sentence.

If you were searching that sentence without the quotations I could get behind it return Venus Fly Traps results, but if you are using the double quotes you are trying to search for that EXACT string.

Exactly. Results should first satisfy at least the "contains all words in query" property. Then filter and rank that set of pages.

How do you handle a query like

"What's the movie where the guy turns into a pidgin?"

This should return results for Spies in Disguise, and indeed, every search engine I tried does. I doubt any relevant page actually contains all the words in my query.

Just recently I was googling nftables (A replacment for iptables in Linux). Google decided iptables was a synonym and all the results were for iptables. (As it's older and more common)

Try “nftables” in quotes. It’ll help prevent query expansion.

FYI there’s a search mode called verbatim, and it’s not the same as putting the individual words in quotes. It’s hidden under the search tools menu. In my experience the results it provides always contain the search terms.

Why on earth is that not the default?

Because it's wrong for basically everyone. Spelling errors, stemming, etc. Most people want pages that match what they mean not what they typed and for natural language (vs programming topics) that's right.

I share your frustration, but the obvious rejoinder is that most people don't agree with you. You can certainly target a relatively niche group of power-searchers, but most Google users probably think that the search engine guessing their intent is part of what makes it "good".

Ha. My parents actually do google.com just to type in something like facebook.com which searches and they find facebook.

I doubt getting them to switch would be very easy unless you did some deal with browsers. But they like most others else uses Chrome, doubt Google would do that.

It's not just your parents.

I was sitting down with a very experienced C++ engineer a few months ago to work on a problem. There was something we needed to do a web search on.

He opened Chrome, clicked in the address bar (which already had the keyboard focus, but never mind that), and typed "google". This did a Google search for Google, and the first search result was www.google.com. Then he clicked www.google.com which took him to the Google home page, and there he typed in the search terms.

Yes, he Googled Google to do a Google search.

That’s… That’s concerning

Well it's possible to be so focused on something that you know nothing about related subjects. Possible though not likely.

Perhaps Google should serve its home page, instead of returning a SERP for "google", if the search URL indicates a search from a browser address bar. If a user really wants to google for "Google", they can then search on the google.com home page.

Most of the time when I Google the term Google, I want search results like Wikipedia, or pages discussing Google's business or history.

Serious question: if he had done that during his job interview, all other things being equal, would that have been a deal breaker?

I don't hire or interview people (and also don't google google to open google), but just want to comment that we are all humans and not robots. We all posses certain quirks and behaviors that are suboptimal. Seriously considering such episode in a hiring is like measuring during interview how far from the paper he put down a pen, because putting it too far is suboptimal and he wasted several microseconds reaching for it to pick it up. It is ridiculous metric and HRs need to be self aware is they are using such metrics in real life, as opposed to real metrics like ability to solve work problems.

Of course we all agree that technical interviews are often broken and interviewers use ridiculous metrics.

The distance from their pen to their paper is an example of a ridiculous metric. And expecting humans to be robots is a straw man to what I'm asking.

The candidate's ability (or effort) to understand and use the tools they use 100 times a day to do their work is not an irrelevant metric.

Doctors need to have a basic understanding of how stethoscopes work. If they start by listening to your elbow, and only then proceed to listen to your chest and back, something is amiss.

It's not a matter of suboptimal behavior or poor efficiency.

It is debatable whether web browser is a work tool, in the context of specialized tools. I'm using DDG for 3 years already but I still don't use bangs, even for often used websites like wikipedia, instead I always do a normal search and then click on a wikipedia result. It is obviously suboptimal and I'm doing it in the web browser, am I unfit for my job because of this? The question to the hypothetical person who will select candidates based on googling google - did you yourself optimized ALL that can be optimized in your workflow? (this is a rhetorical question, answer to which is 100% "no", regardless of who is that person)

i would attribute this kind of thing more to acting on auto pilot than a lack of understanding.

I'd agree to some degree if you could be confident that this was his actual routine, instead of a brainfart caused by nerves or distraction or any of a thousand other things that makes an interview environment different from a day-to-day work environment. I remember my dad getting annoyed at me when I was in high school because he asked me what time my soccer game was and I looked at my watch: he thought I was about to make a smart-ass joke[1] in the middle of us trying to figure out a schedule.

That doesn't suggest that I don't know how calendars or timekeeping work: I was just distracted and glanced at my watch on autopilot.

[1] I did make a lot of smart-ass jokes...

I'd say it's muscle memory. Back 20 years or so before the address bar became a search bar it was common for browsers to just successively add '.com', '.org', '.net' to an invalid address and take you to the first one that came up.

It's a coincidence that modern behaviour is the same for certain domain names.

If he's an experienced C++ engineer he's probably of that vintage, and has probably been in demand enough in the meantime that he hasn't had to address this "niggling" behaviour.

Perhaps it was muscle memory? I could see that happening to me if I had recently installed the browser and it won't auto-complete the searches based on past history. Also the fact that my IQ drops by 50 points when someone is watching my screen and I look like an idiot.

> Also the fact that my IQ drops by 50 points when someone is watching my screen and I look like an idiot.

I've heard this a lot, and have basically trained myself by rote not to hover over people's computers when we're looking at something together, but I still can't say I understand it. I don't know if it's an inarticulable phenomenon, but do you have some sense of what drives this and/or what it feels like?

I am not sure what causes it. Maybe it's the anxiety? I've dealt with it most of my life. Even basic things like going outside, talking to a clerk at a store, talking on a phone would make me a nervous wreck, where I could barely function.

Perhaps it makes me really uncomfortable when what I am doing is the center of attention. Like the nervousness normal people feel while performing something on a stage in front of hundreds of people, but on a micro scale, even doing something in front of a single person elicits the same response from me.

Surely we're all seeing this on Zoom calls these days?

People sharing their desktop/app/window, then breaking out of that sharing selection to bring something else up, while they're talking.. and searching, and things aren't working exactly as expected, so they go into rabbit-hole mode, etc...

Everyone's got a million things going on in their brains, and an audience changes things, and "presenting to an audience" is different than sharing with an audience.

I switch between my Mac and a Windows machine, between Safari and Chrome, between ctrl- and Command-, etc etc. Half the time things I'm connected to are broken (VPNs, endpoints, services...) so I find myself half-stabbing my way through little time windows during the day. And if I'm talking to someone, while trying to do something I've done 1000x before, I'm probably covering my bases by stabbing at the keyboard even more.

I do presentations a lot too, it's a totally different mode.

Try doing the following queries from your address bar:

view.java

main.rs

main.cc

libc.so

winrar.zip

I trained myself to use ctrl+k for searches vs ctrl+l for the url bar, from back in the day when I used Firefox and they had separate search/URL bars. Chrome respects these shortcuts, but it distinguishes searches from urls by pre-pending a question mark.

>he Googled Google to do a Google search.

The HN circle is complete

"Google’s Top Search Result? Surprise, It’s Google" https://news.ycombinator.com/item?id=23975001 https://themarkup.org/google-the-giant/2020/07/28/google-sea...

Yahoo Search used to (not any more) return a second search box as the first result if you searched for Google, because a lot of people with their homepage set to Yahoo would enter google in the Yahoo search box to get a link to Google to go there to search.

I don't remember the stats, but when I worked for Yahoo (2003-2005) they got a fairly substantial number of daily users to stay on Yahoo with that trick.

> most people don't agree with you. You can certainly target a relatively niche group of power-searchers, but most Google users probably think that the search engine guessing their intent is part of what makes it "good".

Translate: Feedback from catering to the least common denominator boosts techie self-esteem.

PSA: Similar reasoning is responsible for political ads.

I don't rely on Google anymore when I'm trying to find good information. I actually have much better results searching Reddit and HN for specific information.

I was not aware that HN had a search feature. Link?

See the footer.

The search box is at the bottom of this very page.

Type "[search term] news.ycombinator.com" or "[search term] reddit.com" into Google. If that doesn't work, you can reverse it "news.ycombinator.com [search term]" if you really want to limit to just hackernews. The former seems to work better for me most of the time.

https://support.google.com/websearch/answer/2466433 you actually want "site:".

No need to downvote. I'm familiar with the keywords. My point was that omitting them sometimes produces better results. I suspect that putting hackernews in the query tricks it into realizing I'm looking for technical topics.

It's at the bottom of the page dude.

After using HN for 6 years I just realized that it is there. Instead I had a separate bookmark to HN search for all these years :) . Apparently I just ignore footer completely as useless part of website.

There's room for slight improvements to the hn layout I guess ;) While we're at it, we could maybe place the logout link a bit further away from the profile link to avoid frequent inadvertent logoffs on mobile. Also, it's probably just me, but I haven't yet figured out how to post an Ask HN/Show HN.

> Also, it's probably just me, but I haven't yet figured out how to post an Ask HN/Show HN.

Isn't this just manually entering the prefix?

I don't know. Is it?

I've used search several times, but each time I want to use it again I still waste time looking for it in the header instead of scrolling to the footer.

Wow. I cannot believe I’ve been reading HN for so long and never noticed that. Thank you.

Your search engine is a spammer's dream. There's a million other things to consider. No thanks.

Pagerank solved that, right?

For a while, yes. That's part of what made it so novel. The fact that results are still full of SEO spam isn't really an indication that the right direction is to throw up your hands and make it easier to manipulate results.

> Actually return the results matching the words that people typed in your search box

That wouldn't take care of SEO spam, which is very good at stuffing its pages with whatever words people search for.

Also, as much as I hate to admit it, Google is pretty good at guessing wrong spelling or synonyms and getting good results in a majority of cases.

How hard would it be to assign websites an actual reputation again?

Include ads on the destination website as part of the reputation score. If a website is loaded with ads, sink it to the bottom. Google doesn't do this because they likely make money from the ads.

Heuristically determine spammy content. It's pretty easy to tell at first glance which content is bullshit, so it's probably not hard to create an ML model to do the same classification.

Manually assign positive weights to websites used by engineers and domain experts. You could even curate this list in the open and solicit help in maintaining it.

it seems that the web has for at least a decade already been at the point where a search engine that is built to be useful by humans should (ideally, but apparently impossible with the current skewed incentives due to ad business considerations, overheated stock market, over-enthusiastic ML expert workforce, etc.) index only sites that opt _in_, vetted by humans.

there is a clear upper bound on the amount of total legitimate web content, and that upper bound is not prohibitively high -- linear on the total number of coherent-content-producing humans with only so many hours in a day and only so many years of adequate brain functioning (and not on the amount of whatever computing resources that are thrown in to support whatever algorithmic content fire hoses that the currently-dominant search engines contend with).

The main problem: SEO.

Reply on news.ycombinator.com