Kagi Small Web
blog.kagi.com
blog.kagi.com
This has been a personal pet project of mine and I spent considerable time getting my hands dirty with the code, as the team was busy with other initiatives. When I said the "feed broke" for the launch I meant I broke it. Software is messy especially for an old school dev. I learned in the process I am not a very good coder anymore (if I ever was one?), constantly going back and fixing stuff I previously thought was solid. Check it out in the linked repo [1].
Most importantly - I found the site replace the need for discovery for me, and getting to know various different humans and their writing felt good! A lot of unexpected stuff surfaced and the web felt close again. I think there is a glimpse of hope in the concept and I hope you see it too. And the improvements to search quality and diversity this brings are real.
You can check the list of included websites here [2]. And all the recent posts already surface in Kagi results (for relevant queries).
[1] https://github.com/kagisearch/smallweb
[2] https://github.com/kagisearch/smallweb/blob/main/smallyt.txt
I understand the spirit of it and don't have any counter examples but seems like a bummer if someone has a nice indie blog but can't be added because they have a few ads or a sponsored post.
That is to say that very little other effort was made to curate the YT channels and we expect the user community will contribute to edit the list.
Also I was under impression that all YouTube channels had ads, so that is why this was not considered as a seperate criterion.
Now that I think of it, perhaps you can call Nebula "the Kagi of video streaming". Maybe you guys could collaborate somehow in the future ;D
So happy so see the small, independent and more humane web being highlighted.
I’m trying to do my part[0] but I have no doubt that a search engine—even if still a niche one— can have a much bigger impact.
Really well done Vlad!
Could be done, but it'd take time and effort.
And if the answer to that is no, then where do you draw the line when it comes to quality?
Curating anything is complicated and different people will have different opinions which is why you end up with different lists curated by different groups of people with different ideas about what should and shouldn’t be included.
Curation is still hard. Also, I personally don’t mind having different curated directories because those will reflect the taste and interests of the curator.
Right now everyone who is running any sort of curated discovery service needs to answer these same questions about roughly the same site.
*https://search.marginalia.nu, commonly mentioned on HN https://hn.algolia.com/?query=marginalia
And to be fair, I was simultaneously inspired by the blog thread to build a curated blog filter for my search engine, which led to a series of changes that overall tends to promote more of this types of results in general.
The fact that we're several to have similar ideas sort of validates the ideas I think.
I'm not complaining, glad there's more tools for searching outside the scope of SEO spam or content mills, and competition can only make the niche better (presumably, unless it gets big enough to incentivize disguising commercial content as niche blogs). Just felt like it might be a bit disingenuous to quote a blog list as inspiration for a search engine focused on "the non-commercial part of the web" which is exactly your engine's aim and wording, didn't know they've been working on small web initiatives since a similar timeline.
Hopefully you both find success and with it uplift hobbyist websites.
why not Matrix? I think that's a more realistic alternative. :)
It would be nice to also be able to just search within the small web, maybe using a lens in the future?
Regarding the topic of self promotion, I would disagree with the current rules and I would ask you to allow people to self promote. As long they have an old enough blog, maybe even cut that down to a year, would be helpful. Most users on the small/indie web lack visibility and this would do them service. My blog is already within the index because I think it might have been picked up during the "HN share your blog" post that happened a while ago, but others might not have been that lucky.
I'm a Kagi customer, and a very happy one. The search engine is amazingly good. This only makes me happier with my search engine choice.
If I notice something is wrong I can report it and expect to have a developer look into it within reasonable time.
They are grateful for reports from users and if there is an actual problem (and not a misunderstanding) it is also fixed, again with a reasonable.
(The first major reason for me is premium search quality, no tracking probably comes at a distant third or something.)
Imagine getting that level of support for any other web browser!
For me you represent an incredible accomplishment: the first search engine that gives better results than Google, respects privacy, offers customization and so much more.
Thank you.
I notice google/bing and quickly kick in my muscle memory built up over the years to type duckduckgo.com and do another search.
After this I still am confused why the results are so crap. Then I notice the duck laughing at me and think... shouldnt this be a dog?
And then I start feeling bad for the original user of the device and tell them to do the search themselves and let me know when they get a result
https://github.com/kagisearch/smallweb/blob/main/smallweb.tx...
It's expensive enough that I can't imagine anyone repackaging it profitably (2x Bing search prices for me) but having to email someone adds just enough friction to discourage a lot of tinkerers from even trying it.
Edit: just saw this:
> Do not submit your own website.
I see. I'm okay with that. Maybe it will show up there one day.
I noticed the example result for useyourloaf wasn't included if I switched it to "Sweden" and not sure if this is just an oddity or if the entire feature is nerfed because I just leave my locale on all the time.
Kagi could just admit they don't want to moderate notes or store them permanently. No need to push down the small web, because a lot of small sites preserve their content.
I get that Kagi probably has data indicating the reality of how often sites down, but it seems from my experience that content in big platforms disappears often as well, even in the cases where the creator hadn't forgotten about it. The "Small web" websites made by a creator that cares have the room to be much more permanent.
It would be nice if Kagi Small Web would have an ActivityPub interface so that the most appreciated sites of a day could be added to a timeline on mastodon or lemmy.
I hadn't considered this aspect of Kagi yet. I'm not a subscriber at this point, but I am strongly considering it. But I use search instead of typing domain names directly to avoid the typo phishing style attacks. I wonder how much "artificial" search that would generate based on my typical usage.
Just a suggestion. Search engines do supply their results with a level of confidence, and you're not wrong to use them like that.
On Chrome I think it is optimized so it always askes Google first.
Otherwise how would Google know all the pages you visit, and especially the ones you visit most?
It's a work-around, not an actual resolution, unless you already happen to like using bookmarks.
Obviously that means I'm using split url/search fields in the address bar, or that wouldn't work as easily. I also heavily use Firefox' keyword search to avoid doing a round-trip through a search engine if I already know which site I want to search; mostly using the same prefixes that DDG uses, so I don't need to adjust too much if I find myself using a different computer.
I have actually already split my address and search fields too, for a different reason. I got aggrevated with having the local machine names on my lan perform a public search, even though my local dns is configured to use only my opnsense router, which does resolve the local names correctly. Splitting off the search box makes the local names always do only what DNS says, and so I get my 3d printer instead of search results for it's name.
For that particular problem, going to "truenas" or "unifi" etc on my lan, the bookmarks are actually more convenient in nmost cases I have to admit! Especially with half of the services needing some special port number like unifi and jellyfin etc.
So this may not be as big of an issue as you think. Should also keep in mind that higher search quality means you don't have to search as much in the first place, which leaves a larger buffer.
Alternatively you could just use "hashbangs" to search domains via google (!g), since I don't believe those cost you anything.
This is not the worst thing in the world, but it is annoying to me and not a life upgrade.
Maybe the search results are better enough to make it worth it (on top of also paying money), or maybe not. I'm still determining that, but it's question and not a slam dunk.
I'm now halfway through my billing period (22 aug - 21 sep) and I'm on 488 searches. Would be tight on the pro plan (1000 searches), but I'm still on the early adopter pro plan (1500 searches).
On the $5 tier it's 1.7 cents per search for the first 300, then 1.5 cents after that. As expected I blew past the 300 before the month was out and am currently sitting at a total of almost $9 for this month and there are still 3 more days to go, and this includes being away for labor day weekend and only using my phone (without kagi) for a couple days, and this is after populating and using a bunch of bookmarks specifically to cater to the fact that I now pay for searches. And this is only my laptop. I have not used kagi on my phone or anything else yet.
I haven't used bookmarks in 20 years and don't particularly want to. I normally don't even have the toolbar visible but now I had to un-hide it and add that clutter back to my browser.
So I'm both paying money and contorting my usage pattern.
I guess now that the first month is about done, I can say it looks like I should go up to the $10 plan, where the searches are only 1 cent, but only for the first 1000, and only if I actually use all 1000! If I pay $10 and only do 250 searches that month, then they weren't 1 cent were they?
As much as I like it, I don't know if I'm going to keep it.
I will not pay $10 or $25 just to have it sitting there available "in case", and I apparently will at least some times (who knows how often? every month? 3 out of 12?) will blow past the $5 an end up paying $10 anyway.
If I complicate my usage to cater to kagi so that my default is ddg and just invoke kagi sometimes when I feel like it would help, then I'll probably forget it exists most of the time and do about 10 per month and pay 50 cents each. Probably only one or two months of that and I'll just decide it's not worth $5 for a handful of searches and just cancel it.
The only way it will be useful for me is if it can just be the default search that I don't have to worry about.
They should just figure out whatever the fair per search price is and bill that. The stupid tiers are probably going to drive me off.
$10 unmetered must always be a fallacy anyway. Surely I can't start up "bkwsearch" and sell searches for 3 cents each, and pay you only $10 for my million$ skin.
So I say don't even pretend.
Except really there probably needs to be both options, because some people value predictability more than any specific amount. So $10 or $20 or any number that never changes is better (for them) than paying only $2 on average but unpredictable each month.
I neither want to pay $10 for $2 worth of usage, nor soak you for $30 worth of usage while only paying $10. Even more than that I don't want to have to think about it to make sure if I choose the N tier that I'm making more or less full use of the N tier.
I appreciate that you took the time to read an individual users complaint, whatever you end up doing. Thanks.
It feels a bit like how it felt discovering Google back when AltaVista was still a thing.
I had some memory of AltaVista being full of SEO (as I recall it was mostly a keyword search engine) spam.
And that was probably true, but I also came across this Quora page https://www.quora.com/Why-did-Altavista-search-engine-lose-g... and it reminded me about two other things stand out to me as true
1. AltaVista was slow and you needed to know your syntax to find things 2. AltaVista was cluttered. Google had very clean results.
I think on 1 google is still fast, but 2. not so much. Google's results are far from "clean", instead it feels like the main goal is showing you as much ads as possible and preferably getting you to accidentally click on one.
This in and the fact that Kagi results are as good or slightly better (and much more customisable) than Google is probably the main reason I'm so happy with Kagi.
After all, Kagi sources results from Google so it's no surprise the quality is similar https://help.kagi.com/kagi/search-details/search-sources.htm...
This is an interesting observation given Google's roots. Although it is a different era, where we have ad blockers now, it would be really weird and neat if Google were displaced by a new search engine that did literally the exact same thing Google did in 1998: clean, no-bullshit page that just gives you search results + page-rank.
https://kagi.com/search?q=%s&r=fr
The search results are consistently better than anyone else's, including DuckDuckGo, so I am, and will remain, a happy paying customer.
I tried Kagi for about a week, and it felt more or less identical in quality to DDG, just infinitely more expensive as DDG costs nothing.
Plus, Searx supports many more search engines, and I can customize it exactly to my liking.
I wish them well, as they clearly have good intentions and a good product, but I prefer using an equivalent OSS and self-hosted solution over a proprietary SaaS I have to create an account for, even if it's not as polished or featureful.
EDIT: Actually, I'm wrong. Kagi apparently also has their own crawlers and indexes[1]. Still, I'm not finding Searx results to be deficient, so I'm not missing out on much.
One thing that does concern me with Searx, and partially with Kagi, is that those 3rd parties could decide to block these API requests at any point, leaving Searx unusable, and Kagi's results less relevant. I'm not sure this is a sustainable way to build a search engine, but I do appreciate both Kagi's and Searx's stance on ads. Using any mainstream search engine via their own frontends is a frustrating experience at best.
[1]: https://help.kagi.com/kagi/search-details/search-sources.htm...
I really appreciate Kagi's development matching what i feel like i'm buying. Thanks Kagi Team <3
Spotify: Really hostile, manipulative, and terrible to artists. On my end I hate how commercial the homescreen is, and the CarPlay interface is just a user-hostile to a degree that is frankly unsafe disaster.
TIDAL: Pretty good in a lot of ways. They pay their artists well. The recommendations are decent. The apps have some really stupid bugs that have persisted for years though, the most annoying of which is that if you shuffle a playlist it only shuffles the dozen or so tracks that the interface had pre-cached from the top. So if you try to shuffle your full library you wind up just hearing the same dozen songs only, over and over again.
Deezer: Wanted to like it, but the apps aren't great, and I ran into more missing tracks than I'd like.
So I finally settled on Apple Music. I've got an iPhone, so it's a natural fit there, and the CarPlay interface is great. They also pay artists almost as well as TIDAL. The recommendations are super good enough, and I don't really feel like the home page is constantly trying to push me to whatever the huge labels are paying them to promote (damn you Spotify). The Windows apps are terrible (like flat out embarrassing), and the Linux apps are non-existent, but luckily there's a pretty great open source app called Cider that solves that.
This is what blocked me from trying Apple Music as well. Maybe i'll give Cider a look, thanks!
It's a great desktop client.
<math display="block">
<msup><mi>e</mi>
<mrow><mi>k</mi><mi>t</mi></mrow></msup>
</math>
e
kt
[1] https://www.jefftk.com/news.rss
[2] https://www.jefftk.com/p/weekly-incidence-vs-cumulative-infe...
#1 What's the rationale behind favoring recent blog updates? In my experience, recent updates make the weakest search results, more prone to updates and link breakage, and overall tend to be of lower quality. I also wonder if promoting recent content might incentivize pumping out low-quality entries to increase the odds of being listed.
#2 In dabbling in the domain I've always ended up with an almost absurd skew toward technical programmer:y blogs. While there is a strong overlap between the cohort with a blog, and the cohort with programmer interests, I feel it would be more inviting to other groups if other interests were better represented. Is this something you've thought about, and if so, what do you think might be done?
#1 There are a couple of factors.
- More recent content tends to be more relevant given the same search query, or at least its freshnes will contribute to it not being completely irrelevant (which is the worse thing you want in a search engine). The quality of it is already guaranteed to some extent by this being a curated list to begin with.
- It was relatively easy to assemble and maintain the list because it relies on RSS feed tech, and there were a lot of sources to seed it.
- Focus on recent writing can encourage some people to write (more) as we had an example highlighted in the blog post. In general, the web needs more high quality, non-commercial content, and this is ultimately what we want to contribute towards with this. By providing a platform (even if very small) to encourage this behavior we get a step closer to the web we like.
#2 In general I agree. Although I should say we did spend effort to create a diverse pool of websites for this initiative (for example I am seeing a lot of economy or photography). Again, we can only encourage the creation of more content in various areas through platforms like Kagi (and Marginalia) and hope that it will work out at the end.
I wish kagi has something similar, one place where I can see all the links to the personal websites collected via all it's sources
They do. It's in the open repo https://github.com/kagisearch/smallweb/blob/main/smallweb.tx...
I really like what you're doing with Kagi Small Web -- love that you've taken the initiative to start surfacing all this excellent content. Keep up the good work. I think I'll try out Kagi search...
I think the only condition is that the article is less than one week old.
And of course it showed me Questionable Content, which I first got to via stumbleupon.
In the meantime you can toggle the 'non-commercial web' lens which will include KSW (and some other things).
https://kagi.com/smallweb displays the homepage for the Kagi Blog at the moment, though.
not that it matters since the old web wasn't good. it was as terrible as now. the UI of absolutely every website ever made has been terrible quirky garbage compared to something like windows 98. Even back then there was a massive difference going from windows 98 (sane GUI) to web (garbage hackjob GUI + ads (YES REMEMBER 40 POPUPS? ADS? TOOLBARS? THE OLD WEB WAS NOT GOOD IT WAS A HELL JUST LIKE NOW)).
the content was never good either. every topic discussed on the web is little cliques who believe some easy to digest nonsense and then if you go skim some books on the subject the meta is completely different. except programming since that just centers around the web [1]. think of anything else like cooking or engineering
the web is a terrible protocol that should have died 20 years ago and been replaced with something that was modern at the time like freenet (and they should have made an alternative to html etc).
1. and this is ironic too since programming is the one field that is steered by the web's body of pseudoknowledge and as a result you have people who think C, PHP, and OOP are legitimate programming practices.
what kind of utilities?
I am happy to pay $2 (or more) for tools I use if it is a one time payment or a payment for tokens (e.g. $n for $m ocr scans, generated images or whatever).
I loathe apps that demand monthly payments unless it is really understandable why they have to have it that way (service that require permanent storage comes to mind, although I think I only use iCloud for storage now).
On a side note, there is now a - to my best knowledge - completely unrelated product "ExpertGPT" from some totally different company. I am not talking about that one.
Second link I found: https://ambience.sk/quotes-from-books-the-universe-maker/
For example Kagi uses Google and even hosts on GCP - I think Google's technology and people are great, it is just the business model that is rotten and contributes to the deterioration of the web.
And interestingly enough, at least Discord (and to some extent Twitter) are trying to have a business model that does not put more ads down your throat (although admittedly I did cancel my Twitter subscription as unexplainably they still showed ads even when subscribing - you can't sit on two chairs).
Twitter isn’t really trying to do that. It’s just something the owner likes to claim, but what he’s actually done is the opposite:
- Hired a CEO from the ad world
- Increased personal data collection (with updated Terms of Service to reflect that)
- Limited tweet access without login (those views would be harder to target with ads)
As a Tesla customer I’m fed up with this person’s endless lies. Seems like you got a taste of it with your Twitter subscription too.
Yeah I see similarities to marginalia, but it’s great to have multiple services for the small web.
I need to get my website on the lists asap!
Now more than ever do we need a user friendly search engine.
Well this is disappointing. It's no harder to curate other languages. You're just say you don't care.