Hacker News Reader

Comment by marginalia_nu | Hacker News Reader

> What I dream of is Google circa 2006. A time where a search engine searched what you asked for. A time before aggressive SEO. A time before social media, when the only people on the internet had a reason to be there. A time before sticky headers and full-screen modal pop-ups asking you to subscribe to a newsletter before reading the article. A time before click-bait and subscription-only websites which tease you with a paragraph before blurring out the rest of the content.

Well, I built sort of exactly this: https://search.marginalia.nu/

It's not great, but it sure has its moments.

Should be noted, regarding Koh Lanta, that travel is one of the most aggressively SEO-spammed topics, along with pharma and online casinos. It's extremely difficult to cut through the noise and reach any sort of signal.

View on HN

Social media existed way before 2006, it just wasn't so popular.

But this was indeed a great time. Remember being on a social network ca 1999 and people used it as a diary where others could comment, give advice and meet others with similar interests. There was a timeline and posts from people you "followed". I think it didn't have likes or ratings though. Most people were super friendly and we were doing meet ups, so very much everyone knew each other. At peak time there was about 20k people and at that point owners couldn't cope with it. With that amount of people you'll find some nasty ones that will ruin it for everyone else and that's what happened. Owners couldn't afford to host it, so they were doing crowdfunding to cover the server costs and maintenance. Some bitter people didn't like that and started making unfounded accusations of theft or that they will report everything to the tax man and the police. So owners eventually closed it.

[Quoting your quote] - A time before aggressive SEO? No, just that Google was able to defeat it with Page Rank (one of their possibly few innovative ideas to the search arena). Mathematically, Page Rank did very well to weed out the SEO'd websites of the time.

Point is, I don't think SEO at the time was any more or less "aggressive", per se. Just different. And clearly, the game has since escalated altogether.

> Well, I built sort of exactly this: https://search.marginalia.nu/

What's the elevator pitch? What do you actually do that is special?

From the about section:

> This is an independent DIY search engine that focuses on non-commercial content, and attempts to show you sites you perhaps weren't aware of in favor of the sort of sites you probably already knew existed.

> The software for this search engine is all custom-built, and all crawling and indexing is done in-house. The project is open source. Feel free to poke about in the source code or contribute to the development!

But that doesn't really give me much info.

The point is to surface human websites and discriminate against content mills, click funnels and e-commerce.

For one I have a sort of budget for how much javascript I will tolerate. Some is fine, like your standard wordpress config probably will fly, but not much more. I also do a personalized pagerank biased toward the blogosphere. The likelihood your website shows up in the results is directly determined by whether real humans link to the website.

It's not a Google replacement by any measure, it's meant as a complement, but if you are looking for more in-depth content on a topic than you can find elsewhere, it's often a good starting point.

I like the idea of a search engine that acts as a sort of filter for known dark patterns along with returning useful results. Limiting on javascript quantity seems like a sledgehammer approach, but I like the spirit of it all the same.

> For one I have a sort of budget for how much javascript I will tolerate. Some is fine, like your standard wordpress config probably will fly, but not much more.

What about sites built with React, etc? Like would you be filtering out cool things like Ableton's "Learning Synths"[1] or do you mostly mean third party scripts?

[1] https://learningsynths.ableton.com

Yeah, those sited go out the window entirely. It's a search engine for documents, not applications.

Why should it matter if a website is marked up on the server side or client side to a search engine?

I don't know why it matters, but enforcing a JS budget turns out to rather effectively reduce the amount of problem websites down to nearly none.

Do babies go out with the bathwater? Almost certainly. But when you're running a small scale search engine, you're never going to index everything anyway, so that's entirely fine.

Why should I download 50M of Javascript implementing a buggy, bespoke browsing engine to render perhaps 1k of text when I'm using a browser that is already capable of rendering 1k of text? For example, I used to enjoy browsing the C2 wiki, but recently they decided they needed to experiment with their own Javascript based browser, thus rendering their site unreadable unless you are using the very latest web browser (that needs to be updated every 20 minutes mind you).

Thanks for the link! I’ve tried it on a few queries and I really really like the results so far! Cheers!

I know it’s of low value to ask, but who really gives a shit?

Sigh, I guess I should’ve clarified I meant “who cares whether or not there’s an elevator pitch, this is just something that someone thought we all needed and would be cool to build”.

FWIW I think it is cool and much needed.

I do!

(also low value, perhaps :)

Neat, I wonder if applying something like GPT-3 to understand what each page gives might help with that a little. Kind of like reversing the SEO spam. I wonder if that's something that ML tools could detect given enough input?

My guess is doing exactly that is most the source of Google's problems right no, meanwhile the spammers do the exact same thing except trying to teach a ML model to successfully pose as authentic content, and what you get is the gloriosu mess we are in. (Even though it's just the last battle in a long war against spam)

Too much / too little, both are problematic I guess. This ends up becoming a war of algorithmic generation vs understanding, and devolves to effectively answering the age old question "is this spam"?

I was going to suggest that perhaps adding info about whether the site had ads, etc. might help the usability. Then I tried marginalia ... :) Perhaps we need to invest in similar detection for other antipatterns (listicles, blogspam etc.) - this is mostly what I was talking about. Less the understanding for indexing, and more the classification for user safety and authenticity of results.

But again, this is very cool.

Yeah maybe.

I don't know how useful these "hobo signs" actually are, but they did feel like a natural response to a lot of web sites justifying their use of trackers and cookies and affiliate links and whatnot with "everyone is doing this!"

Impressive! I'm genuinely curious: Where do you host the index and the DB, and how do you finance it? Hosting an index of that size on AWS is not cheap, and maintaining the server infrastructure yourself seems quite some effort.

It's literally hosted off a beefy PC in my living room. I could definitely not afford to host this on AWS or rented servers.

About ~5k USD total investment including a small UPS, operational costs are $40/mo. Doesn't actually require all that much maintenance, at least not compared to the work I've put into building a search engine (including crawler, index) from scratch.

Brilliant! This is fantastic. I'm in love with it already :)

Reply on news.ycombinator.com