boyter

I’d rather see them publish a federated search of their own content.

I have written this before but I’ll put it here again. What I would like to see is a federated search engine. Based on activitypub that works like mastodon. Don’t like the results from one source? Just remove them from your sources, or lower their ranking. Similar to yacy but you can work with the protocol to connect or build whatever type of index you want using whatever technology you like, and communicate over an existing standard. Want to build the worlds best index of Pokémon sites, then go do it. Want to build a search engine using idris or ats? Sure! I did note the professors are on mastodon so perhaps this may actually happen.

One of these days I’ll actually implement the above assuming nobody else does. I figured if I can at least get the basics done and a reference implementation that’s easy to run it could prove the concept. If anyone is interested in this do email my in my bio.

What I worry about for this project is that it becomes another island which prohibits remixing of results like google and bing, and its own index and ranking algorithms become gamed.

I wish the creators best of luck though. I am also hoping for some more blogs and papers about the internals of he engine. So little information is published in the space that anything is welcome, especially if it’s deeply technical.

Can confirm. I have previously done work for a broadcaster and they have packages ready to go for most older famous people. If not a package a collection of media ready to be cut into one.

Yes. I use it as a replacement for sphinx when it went closed. Very impressed with the documentation, the support on slack. It’s much faster than the last version of sphinx and more stable for me. This is searching over 300 million documents about 1TB in size.

Is it a replacement for elasticsearch? Maybe. Depends on your search requirements. I prefer elastic when you have power users needing to search over specific fields, and all of the syntax that it gives you. For a plain text box you want people to type terms into though manticore is much easier to configure.

Scaling is not a good as elastic currently but you don’t need it as soon due to its lower footprint.

That said I’m about to replace it with my own custom index, but that’s not because manticore is a problem. Just that I can do better for my specific use case. I’m abusing manticore in ways it’s not happy about. For other projects I’d happily use it again.

Please do write about it and your thinking behind it. There is so little out there written in the space.

> That might work for something small like a curated collection of a few hundred sites.

Probably more like a few million but otherwise 100% true. Once you really need to scale you have to start losing some accuracy or correctness.

It helps that the goal of a search engine is not to find all the results but instead delight the user by finding the things they want.

Glad to see this on the front page. One of those posts I reread every now and then. Better yet it’s written by Anna Patterson, who in addition to the mentioned searches at the bottom wrote chunks of Cuil (interesting even if it failed) and works on parts of Googles index both before Cuil and I think now.

Sadly it’s a little out of date. I’d love to see a more modern post by someone. Perhaps the authors of mojeek, right dao or someone Elise running their own custom index. Heck I’d pay for some by Matt Wells of Gigablast or those behind Blekko. The whole space is so secretive that for those really interested in the space only crumbs of information are ever really released.

If you are into this space or just curious the videos about bitfunnel which forms parts of the bing index are an excellent watch https://www.youtube.com/watch?v=1-Xoy5w5ydM and https://www.clsp.jhu.edu/events/mike-hopcroft-microsoft/#.YT...

Surprised to see 2 of my blog posts on there. Both about building search engines. Oddly missing the posts that I would consider a lot better about the subject. Rather flattering though.

Sourcegraph wisely chased the enterprise market, so use it to index your private repositories. Plus it can other things such as finding all definitions and modifying them across multiple repositories. They don’t really exist in the same space, at least yet anyway.

Also there is no money to be made in public code search. Take my word for it. However it is something that all code hosing repository should probably have as a useful too, even if it’s not a direct revenue stream.

Never even considered that. Although as an Australian it’s not even close.

Alas my poor entry into the ring https://bonzamate.com.au is missing. It is Australia specific though which might explain that. It is running its own index though, so might be interesting to some.

100% you can use it even in Germany. I was more pointing out that for some people contributing to public domain software or any work is problematic.

No. Explicitly licensing things is fine as you still retain copyright, but are allowing others to use and modify.

In another comment, but my understanding is that you cannot revoke your copyright over anything you create, but can make it free for use by others. Hence you must licence appropriately.

I am not a lawyer but my understanding is that copyright is considered a human right, hence being non revokable for citizens in Australia and Germany.

Hence you must explicitly licence to allow reuse.

It's why personally if I want something as free as possible I dual licence under Unlicense and MIT with the user able to take whichever suits them best.

The thing I always liked about linq was it’s lazy evaluation. You could write simple selects on your data model and join and filter in the service. Allowing unit tests that operated on the joins.

I’m yet to find a better way to write crud apps than that pattern.

Although that affidavit is still a problem for some. Both German and Australian citizens cannot revoke their human rights and as such cannot put things into public domain. Hence cannot ever contribute to sqlite.

This may be seen as advantageous however.

Same boat here. I ended up finding gron https://github.com/tomnomnom/gron which resolved that issue for me. Now I don't have to look up how to use jq each time I want to quickly find something in some JSON.

Its activity pub so no? Assuming they have not blocked either other you should just follow across the peers.

So much so I wrote a app that I used for finding things in my own confluence and then released as a marketplace application.

You can mitigate the issues in the search if you are organise your content well but the search is still deficient in other ways.

Using an ALB in front of a lambda, while neat is a pretty expensive way to do it. Something like $15 a month.

If you are Australian or German public domain is not an option.

For me that’s right on the money. $6 each way for me to visit the city on the train. I’m at the end of the line though, but not super far out of the city.

Hah! I actually took this sort of joke further than I probably should have a while ago with this https://boyter.org/posts/expert-excuses-for-not-writing-unit...

It also covers how to avoid writing them to begin with.

Nice. I love these posts so much I keep a collection of them.

For this though “one of the easiest ways to make a program faster is to make it do less work”. Correct me if I am wrong but isn’t that the only way to make programs faster? Do less work on a single core is the only method I have ever seen. Everything, from memory alignment to SIMD is just reducing the amount of steps. Would be happy to know an exception to this.

Wow even knows me by my first name too. Very humbled. Once again thanks for being so open with what you have done.

That roughly what I thought. I’m not familiar with activitypub at all. I will probably investigate this deeper.

You see to have some knowledge in this area. Do you have any suggestions of places to look to achieve something like this?

You can do this with Yacy right now, https://yacy.net but it's not great for results generally.

I have often wondered if something built on activity pub or like that would be an option allowing people to group servers with peers they like or trust. Its something I want to implement actually and may get around to doing one of these days.

Matt Wells who wrote the majority of the code for gigablast is someone I have been following online for a long time. I used to live for http://gigablast.com/rants.html updates. Gigablast being an amazing example of what one person can do given the time and effort. If you look around for articles and interviews by him you get some nice insights that would never come from the likes of Google, although Bing has some very good in depth technical discussions such as how bitfunnel works. It’s also nice to look through the code of it and see how things like porn filters were implemented.

It’s also nice to know that for a while in internet history gigablast was mentioned in the same breath as google. An amazing achievement at the time for a single person against the core google product.

I wish that someone with some design chops could work on it for a few weeks though. Or it was rolled back to the design from around 2006 with the rocket logo. I really liked that design.

I really wish I had the courage to strike out on my own like Matt has. I have written a few search engines, but a general purpose one from scratch on my own hardware is unlikely to ever happen, as much as I would love to do it.

That’s for inspiring me so much Matt if you do read this (notice me senpai!). Oh and sorry for abusing your XML api so much. I was poor at the time and needed some search results.

A few choice articles,

https://queue.acm.org/detail.cfm?id=988401 https://www.abc.net.au/news/science/2021-02-14/google-news-m...

I had a brief stab at this with https://bonzamate.com.au although its Australia specific to reduce the crawling and indexing requirements. It's main twist is that it runs entirely in AWS Lambda's meaning it costs nothing when it's not being used.