The Age of PageRank Is Over (2022)
blog.kagi.com
blog.kagi.com
None of the large companies would do it, but that would be the point.
No VC-backed or commercial search engine would do what OP is talking about. But I can see a use for a niche search engine that ranks websites inversely proportional to the number of trackers and ad networks they depend on. Heck, I would pay for that, but I'm a nerd.
Maybe marginalia.nu would like this idea.
Kagi maintains a "non-commercial index (Teclis) and non-commercial news index (TinyGem)" [1]. They also "prioritize non-commercial sources," implicitly downranking monetized sites. (Also, you can manually downrank problem domains [2].)
My devices have randomly switched back to Google or DDG from time to time. The first thing I did was check my ad blocker was working--I was simply stunned by the amount of blogspam puke.
> No VC-backed or commercial search engine would do what OP is talking about
No ad based, i.e. free, engine can, not sustainably. A paid one obviously can and does.
[1] https://help.kagi.com/kagi/search-details/search-quality.htm...
[2] https://help.kagi.com/kagi/features/website-info-personalize...
We do actually. We penalize the sites with a lots of ads/trackers on them in our results and boost non-monetized pages. It is one of the main reasons Kagi reults have a specific 'flavor' to them. (Kagi CEO here)
Which the average Joe wouldn't like, hence my comment that no commercial search engine would implement this feature correctly, on purpose.
And that also enables tons of user-centric features they talked about, starting with the earliest ones and my favorites: being able to uprank and downrank domains (like, "ban pinterest", "pin Wikipedia", "downrank w3schools", "uprank cppreference", etc.), and adding rewrite rules to results (like "reddit.com" -> "old.reddit.com"). Both of these are personalizations tied to your account, so they're active on any device as long as you're logged in.
They've added more cool stuff since, but these two alone were what has kept me a paying customer for the past years.
However I still find it a bit creepy to know that they know all about my searches and even up- and down votes.
Google at least can be used in incognito mode.
Kagi does not track search history, period. There's no history attached to your account even if you wanted it. The login is purely for authentication.
(And I say that as someone who owns such a site.)
Case in point, I wanted to link to http://bash.org/?5273, but bash.org no longer exists.
I wrote an alternate reality fiction short story about PR and SEO. It is here: https://github.com/jaronilan/stories/blob/main/Duplicitous.p...
Not the most “popular topic”, but thought people around HN might find it interesting.
I can imagine that Reddit could join the search war - first, they have so much user gen content that people are deliberately looking for; second, they could use the voting mechanisms that are already in place to give preference to the "better" content. Of course, I might be wrong :).
I wouldn’t hold my breath.
Funnily, I've used PageRank for some smaller search projects, and it still works very well if you use it in a vertical, say "educational resources" or "programming" or simliar. I just did a broad crawl, 3-4 steps from some known "good" seed URLs away, calculate PageRank and mix it in with Solr's classifiers.
Abortion in some states for example. It doesn't matter what any company says right now, the position is clear that the tune can and often does change the moment its inconvenient for them.
Paying for the privilege of being targeted is crazy talk.
Total false equivalence. Kagi accepts Bitcoin [1]. Logins are not identifiable in the way billing addresses are.
> doesn't matter what any company says right now, the position is clear that the tune can and often does change the moment its inconvenient for them
You think the free engines don't know who you are?
When data is collected, it will be trolled through for anything and everything. Absence of data is just as unique. The only way is to blend into the masses generically.
Free engines have a very hard time differentiating me from the mass of other web traffic.
I compile my own browser to make it that way though, even my phone doesn't register an overnight location, and runs GrapheneOS.
Importantly this should be a fundamental right of every person not just those that have the money and expertise to enforce it. The data collection shouldn't be allowed, its the equivalent of quartering a digital soldier in every home (something forbidden by the constitution).
Any business funded and dependent on arbitrary preferential loans (from the printer, regardless of how indirect) is state-run and nationalized industry, and should be bound by the same constitutional requirements of government.
I take my privacy seriously because of my job, and the indisputable fact that sensitive roles get targeted.
Sure. But it isn’t, and as long as the choice is free, ad-supported and paid, privacy contractually (but not technically) guaranteed, there is an obvious answer as to where the smart money goes. And that’s before we get to search quality.
A unicorn third option would be nice. But if even people who can pay for this aren't willing to, it condemns the idea of a public option in the cradle--it screams people don't value privacy in search enough to a quantifiable degree.
Funny thing is, I've told the team about Kagi, but not everyone's willing to pay/see the benefits yet. Meanwhile, I'm wondering why they're asking if a good search engine could answer it so easily.
Producing content was harder back then and the web was a lot smaller. Google most definitely still use links to rank but are much more likely to discount/devalue links than they did historically.
The Age of PageRank Is Over - https://news.ycombinator.com/item?id=33537513 - Nov 2022 (373 comments)
The only moment these kids use a search engine is when they do homework. In any and every other moment they just search "locally" in TikTok, or insta.
It scares the shit out of me.
(Edit: paying kagi customer here! Keep it up, kagi! I still love and need you!)
I think it's more accurate to say that the internet and perhaps personal computing in general is dead.
Practically everything about what makes a computer tick has been abstracted away, because rightfully or otherwise people today just don't care. When was the last time you saw someone actually using the address bar in a web browser instead of Googling? Or indeed Tictok'ing or Insta'ing. Nobody knows what a file or folder is either.
It’s always been abstractions. Even when we didn’t know it. (We mastered the technological use of transistors before we understood why they work.)
I see the guy in the mirror every once in a while, why?
That's just elitism.
MS and Windows gave access to computers to millions of people who otherwise wouldn't have been interested. MS products allowed them to be productive, and enabled thousands of businesses to function. MS was instrumental in the explosion of the internet and WWW in the 90s. No niche hacker-oriented or consumer OS had as large of an impact as Windows.
We can argue whether MS has lost its way since then, but claiming that people weren't in control of their computing in the 90s because they used MS products is silly. The aggressive tracking, SaaS business models and everything else MS is criticized for today came much later.
Well, you answered that yourself. Software in the 90s was running locally, and the data it used was also local. That's an important difference from running software on someone else's computer with data you don't control. That access can be cutoff at any point, and there's nothing you can do about it. With local software and data you would at least have the option to export the data to another format (assuming the software supports it), or to read the data with another software. For the specific case of MS Office formats, even the closed ones prior to the XML formats were readable by 3rd party software. I would qualify this as being in "control of their computing".
Vendor lock-in doesn't deprive users of this control. You have exactly the same issue with F/LOSS today. Whether you choose to use PostgreSQL/MySQL or Oracle/Access, you're always locked in to that specific vendor. Hopefully that vendor has good integrations and interoperability features so that you can migrate to something else if you wanted to, but the actual license has no bearing on this.
I suppose our disagreement is with what it means to be in "control of computing". If you define it as being able to read, modify and share the source code, then proprietary software doesn't fit that definition. But most users don't practically need this level of control. As long as the software has decent interoperability, and you actually have control over how and where you run it, then proprietary software can qualify. SaaS, OTOH, does not.
Yeah, sure -- but that came long after the 1980s / early '90s, which was the period at issue here.
When did people 'care'? Complex technologies have always been pushed on people. Regulation, consumer protection etc. is supposed to be the informed, delegated 'caring' filter.
The interesting long-term dynamic is that the problem is self-correcting. If you use amazing technologies to breed masses of addicted, exploited idiots instead of informed empowered citizens, eventually your walled garden will collapse onto itself.
Back when using a computer required having some idea of what was going on. Even if all you really cared about was playing Doom, you still needed to make sense of all those levers with nonsensical labels on them and boy did we figure them out.
Mind you, I don't see where we are today as necessarily a bad thing. It's a very good thing that most people can just use a computer as just another tool like a screwdriver or a car.
But on the other hand, we've also lost the joys and miseries of getting our hands dirty.
In the short term it triggered a race to a low-information, consumerist bottom in many respects (privacy invasion, addictive dark patterns, locked-down platforms, general enshittification etc.).
But this state of affairs, while unfortunate and lamentably gross waste, does not feel terminal. It is very unfullfiling beyond superficial sugar rushes and essentially hostile to the user-product being exploited. Especially so for the talented people that itch to get their hands dirty.
Informed and able minorities are what moves the needle, not dazed and confused masses. The history of (online) computing does not end here. It is still very early days.
The people who made things work will die of old age or withdraw their support (on strike), LLM's will prevent new workers from developing the same expertise (since entry level positions will be removed).
Systems that have stood strong as oaks for centuries will suddenly fail, and with that collapse so too goes the food production.
Non-market socialist systems don't work, but you get those same systems during currency collapse (where ponzi outflows exceed inflows, or debt growth exceeds gdp).
No one knows a thing because socialism has done its dirty work, having captured academia over the past 50+ years, and indoctrinated the masses.
The benefits were front-loaded (as all ponzi's are), and it happened slow enough that no one noticed over multiple generations, and the generation that got the most benefit won't cede political power (they took power in the 1990s, and remain the majority today). They'll give it up only once its pried from the dead hands, which will come from natural aging.
Menticide from the Totalitarian state has a deleterious effect, making it harder for people to see the problems to take any action. Joost wrote extensively about this with regards to the Nazi's and Mao.
What we are seeing today is hubris and a natural consequence of ignoring lesson's learned.
Why do I even read this stupid fucking website
Forewarning of the issues when it matters, following rational principle, is being forearmed in preparation (should you choose to act on that warning and hedge your survival bets for yourself and loved ones).
Those that accurately predict and prepare survive. Those that don't are culled when the environment becomes disadvantaged towards survival.
There is always an element of chance, but adaptation and flexibility greatly bias towards continued persistence.
I used to laugh when people Googled a websites name instead of entering it manually. But these days, I find myself often either Googling (or DuckDuckGoing usually, but as a verb that just doesn't have the same ring to it)for the name, or relying on autocomplete from my bookmarks or history.
I feel website names have become less predictable mainly because of the explosion of possible top level domains: even if I know the exact name of a website, I can't reliably remember which TLD to use. Plus I'm more and more worried of accidentally using the wrong URL (through a misspelling or a wrong TLD or whatever) for fear of ending up on some scam site instead of the real thing.
I also Google for websites I really should know by heart more often than I want to admit.
Why? Because I've accidentally typed googl.com or google.colm and then immediately slammed ALT+F4 enough bloody times that I actually no longer trust myself to type straight.
At least if I typo "googl" into Google I'll either get an autocorrected result or utter gibberish instead of a drive-by trojan to my face.
What is it with the Kagi sycophancy on this site? Right now there's yet another discussion on the front page full of glowing Kagi adoration interspersed with some reality checks about the actual quality of the search results. I understand that Kagi is a Y-Combinator company but does it have to be laid on so thickly?
As to what I use for search: a self-hosted SearxNG (a meta-search engine, i.e. it proxies search results from other search engines) instance (started with Searx but followed when development moved there) combined with Recoll for local search, recoll-webui and the recoll 'engine' to integrate results into SearxNG. I also experimented with YaCy (a fully self-hosted search engine with its own web crawler) but have not gotten useable results yet, the system seems to get bogged down once the index grows behind a certain size.
But in the end, I noticed that for me, improving search results is mostly about suppressing garbage. Kagi lets me filter out Pinterest and some of the worst SEO spam farms. And with them gone, the results already feel much better.
I'd guess Kagi is popular here because they sell what people crave.
You'd think Kagi has millions of paying users, and not ~33k, seemingly half of which are on HN.
Hopefully this time around they don't actually start referring to their community as a cult like Roam did.
I am not a sycophant. Just a happy customer. And happy to have the chance to be a customer, not just a data point. I love kagi trying to serve me, with actual usefulness, instead of serving some dark marketplace of advertisers that have no interest in my wellbeing.
You didn't know it, because they aren't.
There's good reason its being considered a national security threat when all user data is being shipped through China, and their businesses are a government partnership.
Political Warfare and Subversion are real and difficult issues to deal with given our 'open' society.
It should because anytime austerity, or other diverse circumstances happen where life is on the line, these are the first people that end up dying and worst the responsibility for the travesty is entirely on the previous generation in terms of political power. These are summer childs, and winter is coming.
People aren't naturally stupid. Its a long process of drugs and torture that makes them like that. Only the drugs are flouride and dopamine, and the torture is arbitrary struggle sessions built into every process to eliminate rational reason and thought where they have to interact with it in society during critical identity formation periods (permanently damaging them).
No doubt there will likely be a great dying in the near future. You can only kick the can so many different times in cycles before those cycles all line up at the same time. Mother nature is a bitch, and the unprepared die when safety nets fail.
The first most important and crucial tool for survival is having your brain and knowing how to reason following rational first principles and thought.