boyter

I highly recommend this software if its capabilities fall into what you need. It is very fast both in terms of indexing speed and search. It’s relatively simple to setup and start working against, and I have found it very reliable.

It works best when you have a SQL data store you want to index against, but with the real time index you can treat it more like elastic and other searches. However for that first use case of SQL, I don’t know of anything else that comes close to being as easy to use.

Simply point it at your database, give it a query to pull what you want to index and you are done. I suspect that this covers about 90% of use cases out there.

If you need more than what the DB native indexing is giving you give it a try.

I believe Craigslist has moved to manticore.

Cannot edit previous reply, but I would love to know more about how the sparse grams work. There isn't enough detail in the post, just a few tantalizing crumbs of information.

Seems a lot of others in this thread are interested as well.

That's what I get for a cursory glance at 4am when I wrote this. I will have a much better look after I get some coffee into me.

Thanks for the clarification. Looking forward to see what else you and your team end up writing about. Which reminds me to publish some other posts I have about searchcode.

I won’t hijack this thread with details but if you have questions you can find my details on my profile.

Believe me I am aware. I am one of those who implemented their own index for a code search engine :) I did it for my own learning, but find it interesting because something like elastic with trigrams can get you very close, albeit at a far greater cost.

The sparse grams solution to deal with stupidly common ngrams such as for or tes is very interesting.

I’d love to see more discussion on how they are dealing with the false positives though. It looks like a positional index is being used to achieve this, but that usually blows out your index size.

Additional information about deduplication would be especially interesting to me as well. It seems to solve this quite well. I usually try a search of Jquery to test this and it does not return multiple copies of different versions of it which is a good indicator that it’s slightly fuzzy.

What I find really interesting about all the code search engines I know of is that each one implemented its own index. Nobody is using off the shelf software for this. I suspect that might be down to no off the shelf software providing a decent enough solution, and none providing a solution that scales. At least none that scales with decent costs.

I did a small comparison of GitHub code search a while ago https://twitter.com/boyter/status/1480667185475244036?s=61&t... But I should note a lot has improved since then, and it looks like sourcegraph now also does default AND of terms rather than exact match, so my complaints there are resolved.

Impressive work by GitHub. I am sure some of the people behind it will read this comment, let me say well done to you all. I am very impressed. Also please post more information like this. There is so little out there.

The original show was about how a comedian finds their jokes. Jerry said as much in a Reddit AMA.

There are a lot of academic papers I want to read, either to advance my own knowledge or because I am doing a vanity search and want to see where I made some mistakes, or how I can improve things.

Without sci-hub I would not be able to get access to a lot of them. A good example being this one https://sci-hub.ru/10.1109/TASE49443.2020.00032 which I could had no other option than pay an exorbitant amount of money to access https://ieeexplore.ieee.org/document/9405312

I don't mind actually paying for things, but the prices they charge are insane. Especially for something were I there only thing I can see is a small footnote that just happens to make what I am interested in pop to the top of search results.

    What about the tech of Mastadon makes other nodes refuse to "federate with" me if I "federate with" something they don't like?

Nothing. The blocking situation on the fediverse is arcane at best.

That has been my statement as well.

It is going to happen, and they can go kicking and screaming, or try to get something that works with them.

Anyone doing anything slightly out of the norm on the fediverse gets blasted. I too experienced this just recently. Being open and responding just gets you even more heat.

100% agree the response is inappropriate.

If you don't do an apology like done here the mob will continue to attack you. So I totally understand why they did this.

RM Williams boots. Look good with jeans or suit. Repairable, and built to very high standards. Only need one pair. Highly recommend.

What got me was when my LG tv updated itself. In doing so it broke the Ethernet connection which I was using because the wifi has an issue where it just stops all of a sudden for no reason.

I was able to resolve the wifi issue by resetting the region to another and back and turning off the auto setting for it, but frankly it’s appalling an update can go out that breaks something as basic as Ethernet.

I wonder about the spam problem as well. Blocks will not work.

I have a feeling that Hashcash could be a solution, and where instances do not participate in it you could slow their connections in. Just an idea though.

I could totally see there being a mass movement to add robot.txt to opt everyone out of being indexed.

Then a movement to defederate anyone who allows indexing.

Then hopefully a move to allow-list federation which I personally have suggested for a while now as the only sensible move if they really want some privacy.

100% people will freak out over this.

I hope the author has their flame proof pants on. The mastodon community (well portions of it, its not one community) will not appreciate this sort of thing at all.

This was somewhat covered previously https://news.ycombinator.com/item?id=34221851 but is worth having a look at.

In short though, any attempts to do this tends to result in huge block lists, the hashtag #fediblock circulating around, and in some cases outright personal attacks on the people behind it.

It does. Not in Texas but Australia and one of he rooms that got solar became slightly cooler after it was installed on those very hot days.

Can attest to that. The worst online interactions I have had are on it.

I agree with you when it comes to private messages, and anyone sharing those is a jerk.

I don't agree in this case where the system is designed to spread those posts. Fair enough if you don't want that to happen, in which case don't use a federated system where that is a design goal.

Hence the crux of the issue. You can opt-out of indexing. The moment you federate, and your post leaves your server you no longer have that guarantee. All it takes is one instance to boost your post to someone who enables indexing.

Which is why they should move to allow-list federation, and only allow in other instances they can firstly check to ensure it won't happen again.

They can. Someone could get your post, then boost it, which distributes it to whoever follows them and so on.

Anyone thinking their post is only seen by their followers when they have federation turned on is grossly misinformed.

Author of Mastinator here which was linked in the article. Also its back online, so if you want to start testing your implementation it should work for you. I will be adding more functionality as I go.

While search is a problem, I think the bigger problem with the fediverse is expectations. People expect that their "follower" only posts are only shown to their followers. However you can get them anyway through so many other ways, its literally ludicrous that people expect anything they post to only be delivered to their followers. Boosting posts within Mastodon transmits them to federated servers, and this is by design.

My point has always been that if you play in a federated system, where you literally distribute your posts to anyone who requests them, you have to accept once it leaves your server you have lost all control. To then complain someone is getting that post is naive. That's the federated system working as intended.

If you don't want it public, don't distributed it via a public system.

I don't know if people were told otherwise, but they seem to be under the impression their content is fully under their control in the fediverse, when it is far from the case. Deletes/Edits of content are "should" in the spec, meaning yes people should do it, they are under no obligation to do so.

The only reaction the fediverse has to anyone they dislike is to block, and in some cases harass people till they leave. That wont work long term, and it isn't working now. Despite being blocked by hundreds of instances Mastinator is still getting thousands of posts an hour delivered to it, including posts from instances that blocked it because someone who is federated with those instances has not blocked mastinator and someone boosted it.

The current "elders" of the fediverse want to keep their behaviors and cultures. Which is fine. They should do so by having their own allow-listed federation, not accepting any follow request, or totally de-federating. They should not be bullying others into accepting their behavior. Not only does it not work now, its really not going to work once Tumblr, Flickr and others start integrating, as they will have enough people on that they will be the majority and they will make the rules.

No worries at all. I probably should have made it more extreme to get the point acros, my bad.

And this is why I like comments here.

You sort of missed my point.

It’s where some of the worst most toxic interactions I have ever encountered come from. Take the worst of YouTube and tumbler and then amplify it. Twitter by contrast is a paradise. HN for all its pedantry is at least useful most of the time.

It does not affect me personally. But I could see others being attacked on it then going and doing very stupid things.

A short time on the fediverse if you do something that someone doesn’t like might be enough to ensure you never want to interact there again.

Or try engaging with random people on the fediverse. Seems all of the people who used to write on tumblr and the YouTube comments migrated over there.

A few days there might make you swear off social media forever.

I’d like to know this as well. Consider that creating an account elsewhere or running your own is trivial I don’t see how this is a problem.

Sorry to disappoint. That would be a cool implementation.