Alexandria Search

Could you paste again as text links? Thanks.

Pro tip: Alt+E takes you directly to the search bar, then you can press Tab for selecting the search engine. The best part is that you never use the mouse this way. You can also use ddg bangs, they contain every search-engine/site by pressing Alt-D if you remember the bang for the site.

Might be an instance of Goodhart's law: https://en.m.wikipedia.org/wiki/Goodhart's_law

If all websites try to optimise for SEO, they undermine the assumption that the evaluation of a search engine is the pure consequence of how well a site satisfies a query.

> Is it just me, or I feel like Google does not provide anymore good results for me.

I am starting to suspect that there might be nothing to find.

I just don't think people (other then the tech oriented) are creating websites and running forums - and why would they? Reddit might be be only place you _can_ find that type of content. What should search engines do then?

With a tiny number of exceptions, it might be that people chat on reddit, read Wikipedia, ask questions on the stackexchange network/Quora, local communities use facebook groups, and businesses have a wordpress site with nothing more then a bit of fluff, a phone number and an email address.

DDG's organic link results are from Bing, sans personalization. DuckDuckGo advertises using "over 400 sources", which means that at least 399 sources only power infoboxes ("instant answers") and non-generalist search, such as the Video search.

you.com and kagi.com off the top of my head

I've been using the Kagi beta for a few months now, and it's awesome: https://kagi.com/

The biggest thing I've found is that when doing technical searches it always turns up the sources I'm actually looking for, actively filtering all the GitHub/StackOverflow copycat sites.

It also seems to up-weight official docs compared to Google. For example, "how to read a json file in python" turns up the Python docs as the second result, where in Google they're nowhere to be found.

I listed a bunch over at https://seirdy.one/2021/03/10/search-engines-with-own-indexe..., and I'm always adding more.

I first discovered Alexandria in early February: https://git.sr.ht/~seirdy/seirdy.one/commit/935b55f10f9024ee...

Around the same time, I also discovered sengine.info, Artado, Entfer, and Siik. By sheer coincidence they all were mentioned to me or decided to crawl my site within the same couple weeks. So yes, from my perspective there have been more than a few new smaller engines getting active on the heels of bigger names like Neeva, Kagi, Brave Search, etc.

am founder of Breeze, https://breezethat.com/

1. free version is curated topic filters that sit on top of Google -- best on laptop or desktop at moment / iterating mobile due to ad splash; it's unclear due to TOU that we can ever make that fully server-side legally, though do so have some things testing to see if free version can be ad-free or more tracking free

2. premium version will be mix of our scraping and Bing, depending on topic - standard web search via Bing + same curation as (1), closer to real-time for us

3. have tested out most other indie indexes or sites, for full web scale, money is on ahrefs or brave giving bing / google run for it

4. our primary emphasis is on bringing back some of the Yahoo! directory or Alta Vista look & feel of drilling into topics, so balancing 1-3 (^) as best we can atm, small team, fully bootstrapped modulo tiny F&F round

5. also @DotDotJames on twitter, still iterating when add team info to site

(Founder of Neeva here) I'd love for you to try us (www.neeva.com) and tell us what you think ... (either here or via the support button on the SRP)

(Founder of Neeva here) -- Chris -- I love the idea. Neeva does this in a contextually relevant manner to the intent of the query. For example, on health queries, we label all health sites as "trusted", "ad-supported" etc. and allow you to filter down to the appropriate subset of results. For programming queries, we label sites as "official sites", "forums", "blogs", "code repos", "programming websites" (the SEO-ed ones). We work with human raters to do this. Would love to hear if you find it useful and what other labels would be of use.

I've thought about something similar, basically "the good internet" that would be a hand curated list of sites that are not there just as a pretext for ads. I think a lot of software project documentation qualifies, lots of stuff on university sites like lectures notes for example. I assume that across different niches there is other stuff like this. I think the key would be something that can't be gamed, like it has to be legitimate content that is online for an existing purpose and not as a pretext.

Wikipedia.

Check the 'Small or non-commercial Web' search engines on this overview page: https://seirdy.one/2021/03/10/search-engines-with-own-indexe...

exploring adding that search / similar as breeze filter at https://breezethat.com/- not sure can do with curation + bing or google, may have to wait until we bring our scraper out of testing

This approach reminds me of YACY even though it's only somewhat related.

Instances of the engine indexing locally on the machine where a user is browsing the web.

I'm regularly reminded of this concept of a distributed search index, but at the moment it doesn't seem very likely that it'll gain traction.

There are benchmarks within the adjacent field of information retrieval, but in general it's hard to properly validate a search engine because real data is so noisy and misbehaved, and sample data is so different from real data.

The "Web track" task at the annual US NIST TREC conference ( https://trec.nist.gov/ ) is an open innovation benchmark that everyone can contribute; participants get a set of queries that they have to run on exactly the same corpus. Then they return the top-k results to a team that evaluates them.

Here is an example (2014) Web track paper of the 23th TREC: https://trec.nist.gov/pubs/trec23/papers/overview-web.pdf (TREC has a plentitude of difference benchmark tasks and you can submit your own: https://trec.nist.gov/pubs/trec29/trec2020.html - recent TREC 2020 papers)

(Founder of Neeva) Eval for search engines is about as hard as ranking for search engines. You need rating templates, raters, querysets, tooling and lots of time and patience. There are a number of vendors who can help you with the raters part, but the rest is still painstaking work you have to do yourself. Email me if you need help; we are happy to share findings.

Geeksforgeeks is toxic garbage poisoning the well of good solutions for common problems.

I've found that their content is often inaccurate or written by people who come across as novices. I actually emailed them to correct an inaccuracy in one of their articles once, which they did, so kudos to them for that.

I guess everyone has their go-to sites and their pet peeves. Geeksforgeeks may be less spammy than some, but I still think of it as that annoying site that got in the way of either the SO or documentation answer that I was looking for.

Just to expand, if I want the api reference, say I search for defaultdict (for some reason I like using them but always have to look at the reference), I want the python documentation. I definitely don't want a third party telling me about it.

And if I search a "make list of tensor into tensor" type question, I want SO where someone had asked the same question and got "tensor.stack" as the reply, so I can understand the answer and follow up by looking at the tensor.stack pytorch reference it I want.

Anything else is wasting my time, I think most users with similarly specific queries are not looking for tutorials, they are looking for the names of functions they hypothesize exist, or references. That's why intermediary sites that try to give an explanation are annoying, at least for me.

I generally find that sites like SO, GFG, etc. often play the role of "Reading the Docs as a Service". I prefer using them only after official documentation or specifications fail me. When I want an opinionated answer, I just ping some people I already know or check the personal websites of the developers of the language/tool I'm using. If I have further questions, I check the relevant IRC channel. Sites like SO are a "last resort" for me.

In other words, I'd rather see these at the bottom of the SERP than the top, but I wouldn't want to completely eliminate them.

I feel like this was tried a decade ago: https://developers.google.com/search/blog/2011/03/introducin... ("Introducing the +1 button", Google, 2011)

The index we are running right now are all URLs in commoncrawl from 2021 but only URLs with direct links to them. This is mostly because we would need more servers to index more URLs and that would increase the cost.

It takes us a couple of days to build the index but we have been coding this for about 1 year.

All the indexes are on disk.

Hi,

Yes our documentation is probably pretty confusing. It works like this, the base score for all URLs to a specific domain is the harmonic centrality (hc). Then we have two indexes, one with URLs and one with links (we index the link text). Then we first make a search on the links, then on the URLs. We then update the score of the urls based on the links with this formula: domain_score = expm1(5 * link.m_score) + 0.1; url_score = expm1(10 * link.m_score) + 0.1;

then we add the domain and url score to url.m_score

where link.m_score is the HC of the source domain.

I just think that the timing is right. I think we are in a spot in time where it does not cost billions of dollars to build a search engine like it did 20 years ago. The relevant parts of the internet is probably shrinking and Moore's Law is making computing exponentially cheaper so there has to be an inflection point somewhere.

We hope we can become a useful search engine powered by open source and donations instead of ads.

Hello, Ivan here (the other founder).

1. Yes, any structured data could definitely help improve the results, I personally like the Wikidata dataset. It's just a matter of time and resources :)

2. The first step will probably be to handle this in our "post processing". We query several servers when doing a search and often get many more results than we need and in this step we could quite easily remove identical results.

3. The ranking is currently heavily based on links (same as Google) so we will have similar issues. But hopefully we will find some ways to better determine what sites are actually trustworthy, perhaps with more manually verified sites if enough people would want to contribute.

4. I think that Gigablast and Marginalia Search are really cool and interesting to see how much can be done with a very small team.

Thanks!

The active index is running on 4 servers and we have one server for hosting the frontend and the api (the API is what is used by the frontend, ex: https://api.alexandria.org/?q=hacker%20news)

Then we have one fileserver storing raw data to be indexed. The cost for those 6 servers are around 520 USD per month.

Hello, Ivan here (the other founder).

1. We would prefer to be funded with donations like Wikipedia.

2. I don't think we can avoid it completely, perhaps with volunteers helping us determine the trustworthiness of websites. Do you have any suggestions?

3. I think programmers and people with experience raising money for nonprofits could help the most right now. But if you see some other way you would want to contribute, please let us know!

When I need a clock I'll be sure to use Google.

You still have the current UTC time in the top links. Googles knowledge graph is a part of the problem with their results.

Strictly speaking, that's more in the domain of BonziBuddy or Alexa than internet search engine.

What Google arguably struggles with is surfacing relevant documents, that is... search.

we're exploring adding instant answers in clean way at Breeze; leaning towards using open-source library &/or external API to compute vs. building in-house

also adding premium tier that's alerts + ad free + feeling lucky that would take user to top result, which is a UTC page, re: https://breezethat.com/?q=UTC+time

I agree, Google's instant answers are quite good and have improved but actually searching to find a site seems to be getting worse and is riddled with paid sites on the top.

Kagi does this. I have switched 100% to Kagi, not affiliated.

This doesn't add up, at all. I have a clock on my computer. This new search engine doesn't function like a clock for you, so Google Search is better.

Google has necessarily arbitrary criteria by which pages are ranked. Because Google is the game in town, anyone with a primary goal of driving traffic will pursue those metrics (i.e. SEO). To the extent that those criteria deviate even slightly from actual good results, large parts of the internet will dilute their content to pursue them, which both lowers their quality and further drives down the gems of the internet.

The ranking would have to vary over an infinite spread of purposes for webpages, and it would have to converge almost perfectly to what is actually most helpful. Among all the technical problems, Google will not optimize correctly against ads for the same reason that websites trying to drum up affiliate purchases and ad revenue won't put content quality above SEO.

When recipes return to having the recipe and ingredients first, followed by an optional life story,I'll revisit my assessment.

IMO search has had its goalpost moved. It used to be about scale, technical challenges, bandwidth, storage, etc. It is still about that, but a significantly harder challenge to solve has come up: searching in a malicious environment. SEO crap nowadays completely dominates search, Google has lost the war.

Simply put, I believe that Google sucks at search, in the modern context. It is great at indexing, it has solved phenomenal technical challenges, but search it has not solved. Why do I have to write site:stackoverflow.com or site:reddit.com to skip the crap and go to actual content? Why can my brain detect blogspam garbage in 0.5 seconds of looking but billion dollar company Google will happily recommend it as the most relevant result above a legitimate website?

I feel this 12 year old XKCD is still relevant: https://xkcd.com/810/ .

Google Research is also one of the top (NLR|IR) R&D gigs in town - they discovered BERT, a model that has re-defined how NLP is down and the respective paper describing it already collected 800 citations by the time it was published based on a pre-print spreading like a wildfire.

This technology is now part of Google search.

But what is different in terms of its indexing algorithm? The original secret sauce for google was the pagerank algorithm which was mathematically genius. Are you using a similar algorithm.

More about Common Crawl: https://en.wikipedia.org/wiki/Common_Crawl

> Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public. Common Crawl's web archive consists of petabytes of data collected since 2011. It completes crawls generally every month. ...

Thanks for that! It does look like it's in there and got crawled in January. I probably didn't search back far enough in my logs...

So uh ... 100 petabytes, then.

Ah.

google search often returns wrong, unrelated, or no result. i have been in that situation many times before and moved on to change my query.

i am (and most of us are) trying to solve my own issue. not google's.

that's why you get vague answers to your questions. it's not because it doesn't happen. it's because at that moment we care much more about solving our problem. that's what brought us to google search in the first place.

I've heard from other people who run engines (Right Dao, Gigablast) that this is a major problem; Common Crawl does look helpful, but it's not continuously updated. FWIW, Right Dao uses Wikipedia as a starting point for crawling. Kiwix makes pre-indexed dumps of Wikipedia, StackExchange, and other sites available.

Some sort of partnership between crawlers could go a long way. Have you considered contributing content back towards the Common Crawl?

First off, nice work!

This seems like a reasonable fallback option but it's also a weaker one. By "most of the issues are non-technical", do you mean that you need special permission from someone like cloudflare to get "crawl rights"?