Hacker News Reader

Comment by marginalia_nu | Hacker News Reader

I don't think it's too late to fix this. We can build better search engines, or better yet, tools more suitable tools for discovery. It doesn't have to be big to be useful.

Right now the big problem as I see it is that there is great content being created but its audience just isn't finding it. There's so much noise, the signal is almost completely drowned out.

Out of largely the same frustration permeating the link, I built search.marginalia.nu alone in less than a single year in my spare time without prior knowledge of search, paying for the hardware out of my own pocket.

I don't say that to promote how great I am. The point is specifically that I'm just some guy on the Internet. Just imagine what could be possible if actual resources were put toward finding solutions to this problem.

View on HN

> Right now the big problem as I see it is that there is great content being created but its audience just isn't finding it. There's so much noise, the signal is almost completely drowned out.

It's a curation problem. We have billions of pieces of content being created and only a handful of curators to bring that info to us. All the big search engine players want to be "the one" but it's a fools errand IMO.

No one search engine will be able to become "the one" because there is simply too much information. Google with all of their cash in the bank and talented engineers can't even crack the problem. The problem though is even though each search engine offers something different, they all think they offer the same thing and compete to muscle out any smaller players.

So you get a situation where you want a metal roof for your house, metal roofer gives you a quote. Big roofer guy in town hears about this and runs the metal roofer guy out of town, then tells you he cant offer you a metal roof anyways.

PageRank worked when humans were creating content across a huge number of sites. Once you start consolidating the humans (Medium), then the organic content and the machine generated content start to blur together. And the machine sites can collude to make their content look more organic than organic. Today there's 10x as much of it so if only 20% of the organic content is high quality, that's practically in the noise floor for the sorts of graphs I stare at in any given week.

Perhaps what's missing here is a garbage collection algorithm. Whatever they are using as 'roots' is not working, and they can no longer infer trust by association. They need to curate and review sites. Or maybe they already do, but the humans' focus is directed by heuristics and those heuristics have been gamed.

Does anyone know if Google still has a functioning QA department? Perhaps this is the same disease we see everywhere of developers thinking they understand testing when really we know just enough to be dangerous.

PageRank worked when humans were creating content across a huge number of sites.

PageRank worked when there was very little tangible value besides vanity in being number one for a particular search term, and when there was little competition between sites. Once SEO became a key strategy to growing a business PageRank had to shift from finding the most relevant results to removing the things that were pretending to be relevant results. That challenge is much harder.

I contend that practically every update to Google Search over the past 15 years has been to try to defeat SEO spam. There doesn't appear to be anything else. And, unfortunately for all of us, it's a battle Google are not winning.

> Google are not winning.

Even worse, they are completely disincentivised to do anything about it, since presently the ads occupy most or all of the real estate most users see anyway. Buying ads to show up in that section so you aren't drowned out by spam is exactly what benefits GOOG's shareholders. And I say this as a GOOG shareholder, lol.

I've come to believe that the organic results are a niche product that Google could eliminate tomorrow and most consumers wouldn't really realize anything had changed. They'll go to the first result 99% of the time, and 99% of the time that's an ad.

I agree and disagree. From my view it's like they went mole hunting with a bazooka - and kept making better bazookas. The collateral damage is that many good sites were taken out, or downgraded.

They got a lot of bad sites, but they got rid of a lot of good ones too.

Now google is the modern yellow pages imho. It's more shallow than ever, but it's okay for many, and they can shake down legit sites for adwords to show up.

Totally agree with the main article, sadly however it's like what, 90% of people never go past the first page? And many still assume the top ads are vetted results that are better than all the rest.

Looking forward to leaving google as the yellow pages and encouraging more niche search engines, web rings, and similar.

I can’t fault any of your reasoning.

Perhaps it might be accurate to say that as email has declined in popularity, “spam” has simply moved to the Web.

Indeed - and a glance at the ads on even mainstream sites sure back that up. They're mostly shilling casino mobile games, scam tech (like 'portable air conditioners' that are just a fan that blows over a wet sponge for $75) and the big one, unregulated dietary supplements sold with disgusting imagery and "One weird trick" ad copy. It is almost entirely just utter bottom-feeding trash. And this is what's paying the bills for 'reputable' sites like CNN, Bloomberg, CNBC, whatever.

> Google with all of their cash in the bank and talented engineers can't even crack the problem.

Don't forget that Linux became only possible because 20 years of OS research was carefully studied, analyzed, discussed and thrown away.

Ingo Molnar

From a message to the Linux Kernel mailing list in 1999.

Can you explain? Was there a competitor to Linux that never surfaced?

I wonder if it was a veiled reference to the debate between Linus and Andrew Tanenbaum, who wrote MINIX based on OS research that, according to him, showed that microkernels are better than monolithic kernels.

https://en.wikipedia.org/wiki/Tanenbaum%E2%80%93Torvalds_deb...

In a funny twist of history MINIX is running at ring -3 on most modern Intel CPUs. So it's probably running on more desktops than MacOS or Linux.

what is the ratio between dynamic and static content amongst say the top100 web sites on the web? Can we say 0% static pages? [yes, i accept Dean Gaudet's point that webpages can be split up into dynamic and static parts, probably that will be the future.]

This is delightful

Here's the mailing list entry referenced: http://lkml.iu.edu/hypermail/linux/kernel/9906.0/0746.html

My point was that as a newcomer in either operating systems (Linux late last century) or a new search engine (Marginalia, Kagi) you cannot always win by repeating the same stuff incumbents do.

Especially not when the incumbents are busy digging deeper holes.

Yeah I'm very much betting on the null hypothesis when it comes to fancy machine learning-based methods and implementing the most bog-standard keyword search you could imagine.

It's not perfect, but it's scary how well it works sometimes.

I assume they're talking about all the OS research done at places like XEROX and Bell Labs that never really came to fruition, but formed the basis for modern OS's.

> Google with all of their cash in the bank and talented engineers can't even crack the problem.

"Can't they" or "won't they" because it's simply not in their business interest?

Ah yes, the amazing hn cynicism - it's not that the problem is really hard, and there is a really complex system here with all sorts of incentives, variables, and issues on all sides. Instead, everything is easy, and the reason the "right" thing (as defined by the poster) doesn't happen can be reduced to some single cartoon-villain level bullet point like "not in business interest".

Small problem: There are plenty of folks, foundations, companies who are not resource limited in attacking something like this, yet it still doesn't happen. So maybe it's not as simple as you think?

Please, the cynicism pointed in the Googs direction is well deserved. It is clear and obvious by their behaviour and changes to company ethos that serving ads is priority #1. Any thing that diminishes that ability will not be tolerated.

Just look at the recent (as of yesterday) stories of the changes that Googs is pushing to re-enable tracking that has been previously manually/intentionally disabled by the users. Look at how Maps is forcing users to enable wifi in order to provide routing directions. This one happened to me personally as well.

So you don't at all address whether the problem is hard, and instead deflect completely.

The wifi thing is just a silly example. The wifi is required to do fast start GPS. 99.99999999% of their users probably don't want slow start GPS. If you do, sure, it's no longer the app for you. Yes, they are optimizing for the customers they want and giving them a good experience . I'm not sure exactly how this is something abnormal. Like, you are basically complaining they won't add knobs for you. That's pretty standard in good products - but also means if the knobs they aren't what you want, it's not a good fit for you.

I got the email in the latter, and that is a super weird take. They had one control for literally everything, which was owned by admins. They almost certainly got complaints that turning it off it made search suggestions not work in sheets/docs/etc. So they introduced a new separate control for that, and made it controllable by the user. Because it's controllable by the user, it also means the admin can't see the doc search history of the user, only the user can. Nor can they delete your history for you - only you can control it. Your auto-delete preferences, etc apply to it.

None of this seems bad or horrible to me. The only thing even worth picking a bone about is whether the new setting should take it's default from the old setting or not. There are issues in there of course, because as mentioned, before admins could turn it on/off for everyone - and this led to lots of complaints. do you take the admin setting? Do you take the setting the user had if it hadn't been overridden by the admin? etc

The claim, in that story, i that it is forced on by default even if you had it off. This claim is wrong, afaict. It was not worth wading into that clear firestorm of a thread.

The support doc clearly says: "If you disabled Web & App Activity in the Admin console, it is turned off for Google Workspace users. Users can turn it on if they want."

IE if the admin had it disabled, it will be turned off. It also says if you had it disabled yourself, it will default to off.

It goes through this quite clearly if you click "what if i had it disabled" and "what happens if i don't do anything".

Overall presenting this as "re-enable tracking that was manually/intentionally disabled by users" is a real stretch.

>So you don't at all address whether the problem is hard, and instead deflect completely.

That's right, because I wasn't attempting to address that. Instead, I was addressing that the cynicism towards Googs is not unjustified.

Except both the comments you reply to are talking about a specific problem and context. If you are saying that you are just claiming "in general, cynicism is warranted", that seems like a generalized opinion that doesn't really add anything interesting to anything.

It'd be much more interesting if you would engage with the actual conversation being had.

I did. The conversation stated that the cynicism was not warranted. Just because you brought it up, but now don't want to talk about doesn't mean it wasn't part of the conversation. I did not just randomly reply to a comment with spambot level lack of context.

It definitely feels like you did, though you don't seem to believe it. FWIW - There is literally no claim anywhere that the cynicism was unwarranted. Only that it was typical of HN. So like you are making up a strawman and then beating it up, while everyone else is having a conversation about a different thing. I'm simply inviting you to contribute to the latter rathr than the former.

I have posted in other comments about this subject, and have kept the two topics separate. You seem to be the one unable to handle that people have different views from yours. Not sure why/how you are unable to realize this. Continue the tone deaf hurrahing for the Googs if you must, but it is quite tiresome.

The wifi is required to do fast start GPS.

This is 100% factually untrue. AGPS data is downloaded via the net, giving a quick bootstrap.

This statement of your is just used to deflect the truth ; Google wants more, moar, more, always more.

> The wifi is required to do fast start GPS

No, it isn't. A cell phone connection is sufficient for that.

This seems very pedantic, and doesn't change what i said at all.

I'm happy to change it to "wifi helps with a bunch of features they have". AGPS is not even the only one. Wifi scanning is happening in the phone anyway to know when to switch networks, etc, so they are saving power when it is used instead of cell phone.

Not when I've turned off the WiFi to conserve battery while out and about not anywhere near wifi signals. Unfortunately, because I've decided to use Bluetooth earbuds, I now leave the bluetooth radio on as well. Until then, that was kept off specifically to avoid bluetooth tracking.

GPS is just fine for Garmin, TomToms, etc. Why suddenly for a phone that has WiFi in them Maps suddenly "requires" wifi is just not convincing me of anything other than Googs wants all the data.

> doesn't change what i said at all

Yes, it does. See below.

> I'm happy to change it to "wifi helps with a bunch of features they have".

This does change what you said, because "a bunch of features they have" is not at all the same as "GPS, a feature which pretty much everybody wants to use and wants to start fast".

> Wifi scanning is happening in the phone anyway to know when to switch networks, etc

Not if wifi is turned off. If I'm in my car, for example, I might well turn wifi off to prevent the phone from wasting power scanning for wifi networks that aren't there. But I'm still going to have my cell phone connection on.

> they are saving power when it is used instead of cell phone.

Nonsense. See above. Plus, the cell phone antenna is always scanning too, but on a phone it is far less likely to be turned off to prevent that than wifi is.

What I really can't understand is how any of this is connected to your (correct) claim that search is hard and content curation is hard and so we shouldn't expect Google, or anyone else, to just magically solve it. Google could easily serve the needs of users much better than it does, even if it can't do it perfectly. But it won't because it's not in the business of serving user needs, it's in the business of serving advertiser needs, and serving users is just a by-product.

If Google really wants to solve the problems of users, instead of advertisers, it should have users pay them instead of advertisers. But of course it won't since it has spent more than two decades now getting users used to the idea that search and other "hard" services should magically appear for free, and hoping those users won't notice that they are actually the product, not the customer.

"Google could easily serve the needs of users much better than it does, even if it can't do it perfectly. But it won't because it's not in the business of serving user needs, it's in the business of serving advertiser needs, and serving users is just a by-product."

This is such a uncharitable assumption i don't even know where to begin. Nobody can serve everyone well. Trying is a quick ticket to failure.

The choice not to serve certain people is clearly the correct one, even if one could theoretically do more than you do now. Why isn't the simpler, more charitable answer that they are just serving the majority of their large userbase, or whoever in that user base they care about? Why is it everyone who feels like they aren't served has to decide the reason is because someone is doing the wrong thing, rather than "guess i'm not the user they are looking for".

As for whether you can do more - Good strategy (and product choices) are framed as mutually exclusive choices: https://hbr.org/2012/09/bringing-science-to-the-art-of-strat...

That is always going to make someone upset, because yes, you could do something more than you do for them. That doesn't make that path make any sense. Trying to do more of everything for everyone is pretending that you don't need to make choices, and that those choices don't have consequences (for example, what users you will retain or not). That is, as i said, a quick ticket to failure.

> Nobody can serve everyone well.

I never claimed that anyone could. You are attacking a straw man.

You are completely ignoring the actual point I made, which is that Google is a business, and every business serves the needs of those who pay it money, and in Google's case that is not its users, it's advertisers. So serving user needs has to be a by-product.

> Why isn't the simpler, more charitable answer that they are just serving the majority of their large userbase, or whoever in that user base they care about?

Why isn't the simpler, more obvious answer the one I gave, that Google, like every business, serves its paying customers first, and users are not paying customers?

You seem to be going to extraordinary lengths to ignore this obvious fact.

On the contrary, yours is a befuddlingly charitable explanation. Why on earth would an advertisement company like google, a joint-stock company, be "trying to serve the users' needs"?? They make money by pushing as many ads onto users as possible, almost by definition they must be doing the opposite of serving the users' needs x)

My wifi is never on. This attitude is what is broken at google.

You do a thing (wifi alway on), so clearly everyone does, and therefore clearly we should force it!

Yes, it's everyone else's fault they don't cater to you. How dare they. Why do you think you know their users better than they do? What data do you even have?

The much simpler explanation, given that the product doesn't seem to be failing by any measure, is that maybe you just aren't who they are targeting?

One of the main reasons products fail is because they try to please everyone, and have so many knobs that the UI is horrible.

Good product design requires choice. You don't like those choices. Okay, awesome, it's not the product for you. That doesn't make it the wrong choice, or a bad design. It just means you aren't in the user base they are catering to.

Most of the time, if it turns out the user base thy are catering to is unsustainable, the product will fail, and something that does cater to that user base will take over. So how is this not a uniformly good thing?

The market has products for people who like different things - those users will decide what wins or not.

Clearly, being a google hiring person means you will never see the other side of this. So I really feel like every reply is just wasted electrons.

Google does not need WiFi in order to give directions to locations in the Maps application. However, gaining access to WiFi locations is beneficial to Googs in other fashions. They choose to force people to do it for their gain, not the benefit of the Maps user.

Now, you're trying to say that because some people are technically aware about things and have become vocal and push back against the tide that is Googs' overreaching into people's lives is some how in the wrong. You also seem to be implying that the fact that the vast majority of people are not vocal on the point makes it okay even though they have no idea what is actually occurring. Since they are not complaining, everything is okay, right? That's not how that works.

You literally have zero idea what you are talking about.

The location options for wifi or not, are in android settings, part of system wide settings.

No change to google maps will make this setting disappear. It is not an extra control google maps can drop.

Your entire UI tact us baseless, and makes zero sense.

You're been spreading misinformation and misconceptions through this entire thread.

Can you at least argue with facts?

Serious question (I don't mean this as an attack): is online astroturfing part of your job description? It's the only way I can make sense of the tireless apologetics you've been doing all over this thread.

1-99.99999999% of people on earth is 0.8 of a person. Forgive me if I don't buy into your thesis. Seeing as the vast majority of people don't even understand the implications and extent of always-on pervasive ubiquitous tracking and surveillance, an honest metric would be: what percentage of tech-literate people don't want to be forced to have wifi-tracking enabled to access a feature which doesn't need wifi-tracking. As you can see from this forum, that percentage is quite high.

I think we saw a glimpse into this kind of response with how FB was affected by the recent changes made by Apple.

Good points.

But a large part of the problem is that Google can't be bothered to follow their own rules:

Again and again I catch them ignoring my doublequotes and verbatim options.

There aren't any conflicting incentives around that, are there?

Either you respect user queries or you don't.

Also, look at it another way:

Kagi manages to do this while building on Google! It is in another league, just like Google used to be.

Even nano scale Marginalia (1 guy, 1 tower pc, 4 drives) manages to respect my queries.

I don't think this is particularly cynical. Ads are how Google makes money. Show results with many ads, make money. Show results with no ads, make no money. I don't think it's an example of mustache twisting villainy, but rather their business model. I can't really blame them, either.

Same reason electrical cars took ages to take off before Tesla. The status quo companies (and most are, because nobody likes a disruptor) dragged their heels to no end before that.

This is a similarly over-simplified take.

Things are simply never this binary.

> This is a similarly over-simplified take.

Is it though? There are plenty of examples of businesses or even entire industries trying their level best to slow down changes and innovations that will hurt their profits. And in many cases the products may become worse, but more profitable, over time. In the 1970's, you could buy a washing machine in the United States that would last you 25 years. Try doing that now. I'm sure a lot of folks would like their washing machine to last two and half decades, but that's not that profitable if you're in the business of selling washing machines.

How does that reconcile with google results getting worse over time then, with worse defined as relevancy vs. profit motives? With UI changes that de-emphasize the results and promote the ads? With pulling in cursory summarizes to keep you on a google property and isntead of qucily & easily getting you to the definitive source?

There is lots of support for assuming the worst of google in this case.

Because it's not just Google. Every web search engine has been getting worse; it's why Google remains "the best" in every head-to-head comparison. The whole web has been getting harder to search.

I thought this too.

Then I found Kagi last year who manages to exeed my expectations, proving that my original assumption is correct:

It is still possible to provide good results today, Google just fails at it.

I'm not saying it is easy. But it is possible.

(Extra interesting: Kagi builds on Google and Bing, two engines that drive me crazy, yet Kagi manages to be great!)

With a few small exceptions, every search engine uses either Google or Bing-data.

But that's just it - Google's business interest is always about protecting revenue from search advertising, which is never simply aligned with the search user or with search quality, but exactly as you describe it "a really complex system here with all sorts of incentives, variables, and issues on all sides".

It’s perfectly reasonable to speculate about whether search quality is sacrificed in the interest of some incentive Google is trying to balance in favor of its own revenue stream. The fact that it’s complex is immaterial.

Both things can be true. Perhaps the problem is really hard, and Google also isn't attempting to solve it anymore.

...or they are different problems? Google is really good at solving their problems, maybe they only care about quality search results as related to financial incentives?

Moving the goal posts and declaring a victory aided by buying up/supressing other competitors. Become winner by default judgement.

I would agree except Google used to do a good job.

Who even "gets content" through searching? I'm sure many people here are all basically using the internet through link aggregators (reddit, fb, HN etc).

Has the web ever been much beyond that? Have people ever really been just hitting "give me a random web page about X"?

Of course search engines are very useful, just that these conversations about content and "the web", I feel like there's not much focus on how so much traffic is guided through link aggregation.

I can't tell you the number of times I've wished there was a subreddit or a facebook group for the types of content I'm interested in, only to find that there isn't. Not because the content doesn't exist, mind you.

Link aggregators are also very much part of the problem. Besides the astroturfing, they tend to favor the lowest common denominator in content. Popular vote just doesn't seem to be a good way of surfacing quality content, especially not with the social incentive of winning internet acclaim for submitting a good result.

HN does surface the odd gem, but why can't I pick one of those and say "Hey, this topic interests me, show me more like this"? I'm currently working on build something to that effect, and so far it seems at least reasonably possible.

RSS was good in one way, but it also sort of turns everything it comes into contact with into a blog.

>> I can't tell you the number of times I've wished there was a subreddit or a facebook group for the types of content I'm interested in, only to find that there isn't. Not because the content doesn't exist, mind you.

Why not start it? One of the beauties of reddit subgroups is that not only do you find random very specific ones, but if you don't, you can start your own.

I've started subreddits for niche topics. Only visitors I got were spambots.

What search engine were people using to find your niche topic's thread.

Oh, wait

I often search for articles/books about X, or reviews about Y. The top results for those things is a mix of SEO spam, Amazon links etc. which I all try to avoid.

I use search for information, but I often have to dig quite deep to find something worth clicking on or reading. And that's true for Google and DDG (which I mostly use).

When it's raw, basic knowledge, then I'm using !wiki or !mdn etc. in DDG to search those sites directly. Note: Wikipedia and Mozilla Developer Network are two of my favorite sites.

But when it's something more subtle and specialized that I'm not very familiar with then often my only hope of finding something useful is to search via aggregators and forums like HN, Reddit etc. because then I can gauge the quality of the links via the discussion around it and the people who post them. Or in some cases I know about a specialist or journalist and can use their name to filter out spam.

Discovering these kind of things via search is really hard, especially if their authors are not known. It's the spammers and low effort creators that get rewarded the most, partly because of search engines, partly because of visitors who are content with that kind of quality.

Whenever I'm searching for something (using DDG, too) I usually scroll down the results page until I see a website that I don't recognize or one I remember as being small and targeted. The big company websites usually have shortest blurbs on the topic and the worst "NoScript minigames". Smaller websites usually have more information and less reliance on massive ad networks. This is especially true if you are looking for reviews on games.

I don’t get to play video games anymore as often as a couple of years ago. Then I was naturally informed about interesting titles (typically indie). Now when I have a couple of hours to kill and would like to play something new/original I usually give up after 20min of searching and grab a book.

I wish I knew some decent curators or magazines, but most of the stuff I come across is just too far from what I‘m looking for. Search engines don’t help much here either.

If you find yourself looking for a game in the future, just get "Vampire Survivors". Trust me. It's the pinnacle of casual gaming. $3 and as simple as it is plain fun. Can easily spend 10 hours on it if you want to, or 10 minutes if you don't have any more time.

Thank you, will check it out!

Some of these raw looking indie games have turned out to be gems. Thinking of Faster than Light, Loop Hero etc. It's hard to gauge the game-play from a quick look. Not even "similar to" searches help too much. It's not really about the money spent, but rather about the time both finding and then playing. I wish I had a list of curators that I really dig. So far, hit and miss.

I keep an eye on r/gamingsuggestions on reddit for game discovery. Follow that for long enough and you'll hear some more interesting games talked about upon occasion enough times that you'll seek them out. If you can stand reddit, that is.

Thank you for helping me to waste my time :)

A fools errand indeed. But we currently aren't far from "the one" in three world(view)s: China - Baidu and Sogou; Russia - Yandex; Elsewhere - Google and Bing/syndicates. Korea (Naver) and Czechia/Slovakia (Seznam) are not international but local exceptions.

We choose not to offer the same thing based on our values; no-tracking and infrastructural independence, meaning we aren't captured like other search options: https://blog.mojeek.com/2021/03/to-track-or-not-to-track.htm...

For me it would be enough if RSS feeds could include <recommended-feed href="https://someone-elses/rss.xml" reason="Friend's travel blog" /> and RSS reader allowed to browse through the web of recomendations and add allowed to add selected ones to my feedreader list.

A potentially standard way to find other related feeds, instead of just consuming the content of one author/website. Curated by real humans that produce useful content (that's why I have them subscribed).

Simple thing that could have been very powerful for discovery if this was thought about early enough.

webrings for rss?

yep

> Google with all of their cash in the bank and talented engineers can't even crack the problem.

Search is a solved problem. But you're starting from the false premise that Google and your expectations are aligned, they're not. Google shows you what they want you to see, typically spam sites carrying their ads. That's not what you're looking for but from their perspective they've 'solved search'.

I see nothing wrong with this comment, and find my understanding of Google to be very same.

Google has two problems. One is that they're actually Doubleclick in a trenchcoat, and serving you relevant information isn't their goal. Serving you ads is.

The other is Goodhart's Law: When a measure (being highly-ranked on google) becomes a target, it ceases to be an accurate measure. Pages now optimize for search as measured by Google.

If Marginalia because the #1 search engine in the world tomorrow, I suspect SEO spam would start trying to beat it a few days later, and the results would be crap inside a month.

> If Marginalia because the #1 search engine in the world tomorrow, I suspect SEO spam would start trying to beat it a few days later, and the results would be crap inside a month.

This is only actually a problem if there is one big search engine. Monocultures of all varieties are vulnerable to specialization, this is as true in a country that grows nothing but wheat to a world where every computer runs Windows XP. If there would be two or three big search engines with conflicting means of evaluating a site, it would decimate the effectiveness of SEO.

A lot of what I'm doing to find good results is promoting the sort of sites that Google rejects. While there are search engine optimized sites that aren't SEO spam, there is no SEO spam that isn't search engine optimized.

Whoa. Searching a few random terms you’re hitting your goal of “serendipity” results. Can you share more about the search basics? I’m learning search now

Don't really know what to say, a lot of the algorithms are snore-inducingly standard.

TF-IDF, an approximation of BM-25. Only twist is that I do Personalized PageRank biased toward a secret segment of the internet to promote the right kinds of search results.

> We can build better search engines

Yeah, they're called people.

We can curate better results.

Sheds a tear for Yahoo like search. What would the web be today if Sergey and Larry hadn't met?

I really enjoyed the Halt And Catch Fire take on curated vs algo search.

> Right now the big problem as I see it is that there is great content being created

Is that so? Where?

We'd have to define 'great'.

If you mean Tik Tok dance videos and instagram photos/art, then yes. If you mean anything related to philosophy, sociology or explaining how the world works at large - no.

You don't need any tools. You just need to pay people, who have a working brain they've accomplished something with.

It's not an algorithm problem, it's a 'fuck you money' problem. Until people with a brain have fuck you money, they aren't going to waste their life fighting braindead Karens and corporate/political interests who don't want people with a brain to explain how the real world works.

Look at the amount of shit Joe Rogan is getting for having accomplished scientists on to talk about their work. He has fuck you money. Do you know how many other people have fuck you money and don't say shit? A lot. You need to spread that fuck you money around thick, so that one in ten thousand will have the guts to put up with braindead Karens of the world.

> If you mean anything related to philosophy, sociology or explaining how the world works at large - no.

I'm the wrong person to ask for sociological links, but here's a whole flock of actively maintained black swans pertaining to philosophy and other topics:

https://lileks.com/

https://stpeter.im/

https://www.philosophizethis.org/

https://antiqueradio.org/

http://attalus.org/

> I don't think it's too late to fix this. We can build better search engines, or better yet, tools more suitable tools for discovery. It doesn't have to be big to be useful.

Over the next few years, things will change. When people don't like the product, it gives us an opportunity to make products people do like. Right now, a lot of the internet is just not very likable.

I'm not sure. This is a problem that affects not just websites, but physical products as well. It's really hard to find what you actually want. Producers are rewarded more for paying for reviews and search results, for reddit astroturfing, for influencer recommendations; than they are for actually making something worthwhile.

It's not that good things aren't being made, it's just that it's incredibly hard to find them among all crap. There doesn't seem to be a good mechanism that lets the actually good products float to the top.

I doubt this is a new phenomenon. I certainly risk buying things just by myself where pre-internet I would find someone that knew about the thing (cars, knitting, lamps, whatever) and tell them my issue and see what they said. now there is some expextation on my part that fifteen minutes of reading will somehow allow me to drop a lot of cash in some specialized area i have no edperience in and not be taken advantage of as a sucker in the market place?

If you are spending money, possibly just having the worlds info at your fingertips doesn't make up for a lack of experience.

Back in the brick and mortar days, you could at least go look at the thing. Not that there weren't lemons, but the sort of extreme low quality alibaba crap that you find everywhere nowdays wouldn't sell at all because it feels so obviously low quality. Even having it on the shelves would stain the reputation of the store.

Simple laissez faire competition leads to many solutions that are more optimal than naively creating a product people want at an attractive price. The game needs to be regulated to force the "right" kind of solutions to the top but that would be a difficult thing to do even if regulators weren't already part of the "wrong" way of competing.

We can still "vote" with our wallets but just like in real elections sometimes it's really hard to even get some worthwhile candidates on the ballot.

Your search engine is awesome, I've been using it since I saw it elsewhere on HN a few months ago.

Better search is not the issue, it distorts the spirit, google became the focus of attention, now you want to be listed and seen, to monetize too.. before that people spent less time formatting stuff to exist. The word of mouth levels is a different (maybe more fertile) soil.

Yeah I don't think search is actually the solution to the problem of discovery, it was just the first avenue I tried. Although the metadata from a search engine database does open up a lot of options.

Oh sorry btw, it was not an attack against you. And I actually enjoyed your engine and found a new lisp dialect I never heard of (even though I googled that a few hundred times).

none of the real issues breaking the internet (which is not quite broken yet) has technical solutions. No better tool nor "more technology" will solve the problems at hand.

The problems are social and economical, they require "political" solutions, not more technology.

Hey, edit your comment so your link is clickable.

It's past the event horizon and can't be edited anymore.

Reply on news.ycombinator.com