Comment by diegocg | Hacker News Reader

Comment by diegocg | Hacker News Reader

Google was still useful for things that are non-marketable. But in the last year or so, I have started to realize how awfully bad has gotten even at that.

Case in point: Open source mailing list archives. For various reasons, some times I have to search for entire email threads that are available in the web interfaces of some mailing list archives. I have part of a phrase of some email, I put that into google with quotes, and it returns some results, usually with the main archive being in the first results. In the last year or so, Google has started returning no results for some phrases of emails that do exist and are available in one or more public archives.

When that started to happen, I tried Bing. And Bing returns results with links to archive. So does duckduckgo most times. But for Google, it's like if that particular email I'm searching at that moment didn't exist (for other emails in the same mailing list it works fine). It only happens occasionally, but it's getting worse. So I have started to rely on DDG and Bing more and more, because they always find what I want.

So Google is starting to fail at some of the most basic aspects of a search engine, it's not just the ordering of the search results - there are some public web pages that it won't see for whatever reason.

One thing I learned from the unreliability of Google search is that we should not rely on an external service to organize our information. No matter the service, it will change in some way, and you will lose access to some information.

Agreed. "Indexing the world's information" sounds like something that can't go wrong.

It's only in the last year of trying to index and organize information within my own company that whoever controls the index holds a _lot_ of power to index it poorly.

What do you suggest as an alternative? Index the internet yourself? Web search as a public utility?

Well there are semantic web concepts, and there are concepts like a searchable information graph. I read a really good article on this site a few days ago laying out a pretty good architecture for searching a distributed knowledge graph.

The core problem is a disconnect between the knowledge available on the internet and search tools. Indexing is a stop gap to enable the web to be usable without solving this problem, but it is becoming untenable and was always known to be non viable, just treated as a problem for another day. The sheer amount of information online and its disorganization, as well as the misaligned incentives of search tools are beginning to make this once far away manageable problem a day to day reality for all of us, the solution requires an architectural change to the way information is stored and accessed and updated online.

Do you have any idea of what that solution might look like? The existence of a problem doesn't mean that there necessarily exists a better alternative.

Why are browsers so forgetful? Local text search is an easy problem at single-user scale.

I am working on a browser extension to automatically grab text (visible phrases) from the pages I am reading in the browser, save them to my index and implement search. I want to always be able to recall anything I see. In order to match texts I will use neural embeddings computed with a pretrained Huggingface model, fast enough even on CPU.

An alternative way to implement this would be to take screenshots every few seconds (only after recent user interaction). Then I can OCR the images and search anything that passed on the screen, in any app.

It's as if I become the Google Bot responsible with exploration and filtering of potentially useful data.

In the future I might be able to use my index for neural question answering, add a voice interface to my data.

Of course the local index could work for images too. You can grab images and embed them with CLIP to get Google Photos like semantic search.

I think web browsers are going to morph into personal internet agents, responsible with logging, searching, interfacing with external APIs and services, filtering ads, news and mail, making sure we don't accidentally leak personal information, etc. In other words AI that works for the user, locally.

"I think web browsers are going to morph into personal internet agents, responsible with logging, searching, interfacing with external APIs and services, filtering ads, news and mail, making sure we don't accidentally leak personal information"

And you think that will happen, while the browsers are still mostly add funded?

But I am with you, that this is a change, I want to see, too.

There is an extension for Firefox (Memex Go?) that attempts something very similar and succeeds to a large degree.

I think I sponsored it for a year or something by paying for it even though I didn't use it much but I must have used it to some degree because on one or more of my machines it injects interesting results from my earlier browsing when I search in DDG (or was it Google?) at least.

I'm really impressed how far they've taken this idea even as Mozilla has simultaneously been busy dismantling the extension API that made Firefox great.

I made a small project that collect text content from the browser using userscript, and perform search locally

https://github.com/beenotung/personal-search-engine

Hey, can you email me (url in bio), would love to chat!

I for one am grateful that some of my worst mailing list accidents (like accidentally sending a privately intended message to a mailing list, rather then an individual email address) are starting to disappear from Google’s relentless claws.

Likewise, I constantly find that google won't turn up mailing lists posts and forum posts even when I have a fragment to search for exactly.

Some investigation seemed to suggest they were hard excluding any page which hadn't changed in ~5 years.

I have to wonder if the poor behavior from google doesn't come from automatically a/b testing behavior changes against advertising revenue. Up to the point that you make users give up completely giving worse results will result in more search traffic...

>Some investigation seemed to suggest they were hard excluding any page which hadn't changed in ~5 years.

I had another hypothesis: that they exclude pages which have no backlinks from other websites, and at the same time are hard to reach from front page of the website they are on. It seems old unpopular mailing lists posts and forum posts are exactly like that.

Blacklisting non advertisable media from the index. That's news to me. Hmm.

Why? Cost savings? Missing hard dependency on marketing ops knowhow? Utter domination of non-marketing material? Forcing experienced privacy-conscious users to a different platform to reduce bad media coverage?

Monkey run experiment, revenue goes up, monkey get promotion.

At this point I cannot be alone in thinking that what we are seeing is a runaway AI system?

Let me just speculate wildly here:

Seems no-ones Google is the same and how do one manage approximately 4 billion [1] a/b/c/etc experiments if not using artificial intelligence and reinforcement?

Now imagine 4 billion agents ("monkeys") who flip bits everywhere and every time one of them is unusually successful that pattern is reinforced across a part of the "population" of agents.

Every time one of them does something particularly stupid Google relies on users to report an error and swaps out that agent. (This explains why if you report an error in search it appears to magically be fixed 15 minutes later.)

Let's say this has been running since way before 2008 and 2008 just happened to be the time it got completely out of hand:

Mostly it works very well economically speaking but search quality suffers year by year and no one can do anything about it since by now so many optimizations have been applied that for a decade no single human has been able to really understand it.

The AI - using end to end optimization techniques - learns to prioritize ad-heavy sites (since thise are more likely to result in positive reinforcement in the form of juicy ad impressions or clicks) and sites that doesn't really answer the question (since that brings the user back to Google for a new chance to serve ads both in search results and on other high ranking pages.)

In the same manner the system learns to avoid easily understood clean pages without ads as that means game over for this query.

By now everyone who knows anything about it sees that it broken but nobody dares to fix it since that would result in a massive revenue loss.

[1]: assuming half the world uses Google

Curious to hear: what are some rising Google competitors that try to do a better job?

https://you.com

https://neeva.com looks similar

> 6. Fees. The Services are currently offered by Neeva free of charge. Neeva reserves the right to charge fees for the Services in the future.

https://neeva.com/terms

Sustainability TBD, but they do have a plan.

> We’re in beta and feel like our product is not yet fully realized.

Sustainability remains TBD.

I've been playing around with you.com after someone mentioned on HN. I dig the layout/quality of results. It's kind of interesting to have it feel like you're netflixing your search results by source (reddit, stack overflow, etc.) I'm still toggling back and forth on you to google because i feel like i might miss out something. Surprisingly some the results have exceeded my expectations. We'll see.

This site is quite laggy, I'm assuming because of all the JS. Tried to load it with JS disabled and it doesn't work at all.

The site isn’t laggy for me on an ipad7, as for disabled js I don’t see the point anymore this is like trying to load webpages without css or html at this point.

The side scrolling is nice, but when searching some code related question it doesn't show useful results. Even the Code Complete snippets aren't displayed in a readable way, lines are too short, and the language highlight is not recognized (every script is python by default?).

Thanks for mentioning it since I'm creating a snippet search engine and wanted some competitors to compare.

It's hard to compete with Google. Back when Google started, you could fit the world wide web on a single computer. Today you can only really get started the same way Google did if you focus on an unfancy subset of the web that hasn't been react'd and wasm'd into exabytes of opacity. https://wiby.org/ does it I think by refusing to index any HTML page that has <script> tags. Websites like https://millionshort.com/ provide pretty nice results sometimes, but I don't think they actually index the web on their own; they probably use the Bing or Google API on the backend. With million short if I want bike reviews for example, I get the authentic blog posts from 2010 with amazon affiliate links to bike listings that are no longer being sold lol.

The problem is I don't think the Wiby model (rejecting JavaScript) or the MillionShort model (rejecting popularity) have done a good job capturing what we loved most about the old web and systematizing its curation. There's definitely an opportunity for someone to come along and create a focused niche alternative that's better.

One way I suppose it could be done, is if you convinced a bunch of trustworthy high status trendsetter type people to subscribe to a paid service with a browser toolbar that lets them click a button for each website they visit to say "I like this" or "I dislike this" and then use that information to train a neural network that divides seo spam from content. Mix that with classic page rank and you might have something good. I'm not sure if it'd ever appeal to a more general public audience though.

Anyone who does it is also going to want to make a deal with the archive.org to somehow get a snapshot of the old web, to recreate those original experiments. Or possibly even resurrect an old build of it. Plus Gutenberg. There's enough content from 2005 web alone and all the books published before to last anyone several lifetimes. That's actually one of Google's blind spots. They're so good at up-to-the-minute indexing of current events that sometimes if you just want to get the text to something like Seneca it's like pulling teeth.

That browser toolbar was called stumbleupon.com :) Good memories of a somewhat different web!

I used to love SU! So much so that I gave it up for Lent one year. It brought back the serendipity of the old web.

Someone recently emailed me that they had found my website through https://stumbled.to/ and I was delighted to discover:

> Stumbled was created by Kevin Woblick to revive the famous discovery experience StumbleUpon offered back in the old Web 2.0 days.

https://www.mojeek.com/ no-tracking, independent crawler/index

Why is your search engine so bad?

Brave Search (with their own index) is new and is already better or as good as duckduckgo.

What search(es) did you try?

Lyrics search(es) like: https://www.mojeek.com/search?q=a%20split%20second%20Mambo%2...

https://search.brave.com/search?q=a+split+second+Mambo+witch...

Another:

https://search.brave.com/search?q=generator+7%2F8+lyrics

Edit: Here's another:

https://search.brave.com/search?q=neuroticfish+the+bomb+lyri...

Yet another:

https://search.brave.com/search?q=chrysalide+not+my+world+ly...

Many thanks. That's helpful.

We don't appear to have any of those pages or anything relevant indexed. We'll look into whether these lyric sites have put up some barriers to crawling. If they are big sites we won't have crawled and indexed many of their pages.

Another:

https://search.brave.com/search?q=snowy+Red+mad+man+lyrics

Brave Search is actually quite good, and better than DDG in my opinion. And it's their own index, not piggybacking on Bing or Google.

Does it offer similar privacy promises as DDG?

It certainly _promises_ to be private like DDG does, but AFAIK none of them have been audited.

Siri, Alexa, and Amazon.

When someone need to know the weather or a sports score they usually just ask Siri or Alexa. If they want to do product research most people go to Amazon. Google search is now below 50% and still falling.

Think about it. You want to buy a coffee maker. What is the first site you go? It is probably not Google.

Amazon has a horrible review scam policy. It's okay by Amazon policy for sellers to pay people to remove their bad reviews.

Being highly rated on Amazon is so valuable, that there are whole industries on making sure your reviews are high, including all sorts of malicious review hijacking.

Amazon the company does not seem to care.

kagi.com

> What does it mean that you’re in “private beta”? While we're in private beta, we don't feel like our vision is fully realized and ready for prime time

> How much will Kagi cost? Kagi will be completely free during the beta-test period for all users. Once we officially launch, we plan to offer entry level plans for as low as $10/month, ...

https://kagi.com/faq

Reply on news.ycombinator.com