renegat0x0

- https://wikispeedruns.com/

- https://sixdegreesofwikipedia.com/

- https://www.wikitok.io/

- https://httpgoats.com/

- https://httpcats.com/

- https://http.garden/

- https://httpducks.com/

- https://http.fish/

- https://http.pizza/

- https://httpstatusdogs.com/

I maintain a list of domains, so it was quite easy to find them

- https://github.com/rumca-js/Internet-Places-Database

I made also something for my own search needs. It's just an SQLite table of domains, and places. I have your search engine there also ;-)

https://github.com/rumca-js/Internet-Places-Database

Demo for most important ones https://rumca-js.github.io/search

Scraping is hard, and is not hard that much at the same time. There are many projects about scraping, so with a few lines you can do implement scraper using curl cffi, or playwright.

People complain that user-agent need to be filled. Boo-hoo, are we on hacker news, or what? Can't we just provide cookies, and user-agent? Not a big deal, right?

I myself have implemented a simple solution that is able to go through many hoops, and provide JSON response. Simple and easy [0].

On the other hand it was always an arms race. It will be. Eventually every content will be protected via walled gardens, there is no going around it.

Search engines affect me less, and less every day. I have my own small "index" / "bookmarks" with many domains, github projects, youtube channels [1].

Since the database is so big, the most used by me places is extracted into simple and fast web page using SQLite table [2]. Scraping done right is not a problem.

[0] https://github.com/rumca-js/crawler-buddy

[1] https://github.com/rumca-js/Internet-Places-Database

[2] https://rumca-js.github.io/search

A little bit different, but also allows to scrape efficiently. Json http communication rather than cli.

https://github.com/rumca-js/crawler-buddy

More like a framework for other mechanisms

Still working on

https://github.com/rumca-js/Internet-Places-Database - database of Internet places

https://github.com/rumca-js/crawler-buddy - crawling server

https://github.com/rumca-js/Internet-feeds - list of RSS feeds

Also curated list of rss sources

https://github.com/rumca-js/Internet-feeds

I disagree with a notion that a page needs to work without javascript. It is only design choice of author.

How is it different from searxng then?

searxng allows me to use various locations

https://searx.space/

Don't care. I use my own domain index, which also contains youtube channels, which I use through RSS.

https://github.com/rumca-js/Internet-Places-Database

Most important links are available through a simple search

https://rumca-js.github.io/search

I have also RSS search

https://rumca-js.github.io/feeds

It is a fine project to limit 'doomscrolling', but I think the premise is wrong.

- I have created my own RSS readers, that contains 500+ sources. I do not doom scroll

- doom scrolling appears when social media algorithm feeds you data, even from a month ago

- I have various filters so I can browse whatever I want

So RSS should just have filters, categories, search extensive capabilities to solve doom scrolling, and on the other hand it will be able to provide you extensive amounts of data.

If anyone is interested in RSS feeds, here are mine in SQLite table:

https://github.com/rumca-js/Internet-feeds

A browser is a tool that allows you to browse the internet. It should be able to display HTML elements, and stuff.

LLMs are also a tool, but it is not necessary for web browsing. It should be installed into a browser as extension, or integrated as such, so it should be quite easily enabled, or disabled. Surely it should not be intertwined with the browser in a meaningful way imho.

that is hilarious :-)

I use my own library of domains exactly for scenarios like that

https://github.com/rumca-js/Internet-Places-Database

For me capturing list of RSS sources is fun

https://github.com/rumca-js/Internet-feeds

But i do not check all 3k sources regularly

I think I use SQLite like that (to some extent):

- https://github.com/rumca-js/Internet-Places-Database

For UI I use HTML, because it already provides components with bootrap, and everybody can use it without installation of any software.

All data comes from a single SQLite that is easy read, and returns data.

My database is really big, so it takes time to browse it, I wanted to provide more meaningful way to limit scope of searching

I maintain list of personal domains, together with many other things in a database

https://github.com/rumca-js/Internet-Places-Database

- Companies had to go in that direction. You cannot just fall behind AI gold rush

- These solutions are easier for people, and therefore will win in the long run

- these solutions benefit companies because of the surveillance data they have access to now. They always had some data, but now they collect and process even more

- those who control AI will be the kings of the future, so naturally everyone will be running toward this goal

This reminds me that I really like self-hosting. While it is true that many of things do not work, all my services do work. It has some tradeoffs of course.

- How many people use only Google search engine nowadays? More and more people use chatbots, with Google search.

- Google search also does not provide good results for finding stuff in all walled gardens, so we also use niche search engines for individual platforms. I am not sure if it finds good results for posts in facebook and x.com

- I also use my own index of pages, YouTube channels, and github pages. Contains tags, page scoring system, related links, social information like number of followers etc.

https://github.com/rumca-js/Internet-Places-Database

So in a way, it has being replaced. It just takes some time for people to switch.

Even I, who does not know much, implemented a workaround.

I have a web crawler and I have both scraping byte limit and timeout, so zip bombs dont bother me much.

https://github.com/rumca-js/crawler-buddy

I think garbage blabber would be more effective.

F-Droid can not work in the future. Android is controlled by Google. It already tries to block sideloading apps.

Still capturing index of the Internet

https://github.com/rumca-js/Internet-Places-Database

- user reports a bug

- dev asks vibes for help

- llm rewrites half of the files

- it does seem to fix the bug

- 50 more bugs enter the chat

I have seen a lot recommends for karakeep.

I disagree with people suggesting removing noisy RSS feeds, as some are noisy, yet sometime useful. I think RSS client needs advanced filtering and search.

I use my own project for RSS https://github.com/rumca-js/Django-link-archive, but it should be considered 'alpha' as I move fast and break things. It provides me functionality that I miss in most of the clients, and it is a web client, so I can access it through mobile and PC, no need for sync. I am not a front-end dev so it does not look that much appealing.

I use RSS to follow these I want to hear. I do not follow recommendation, since RSS reader is my window to the world.

I follow even AI slop via reddit RSS.

I control however what comes in.

Searching the web has changed:

- There are more walled gardens, so engines legally cannot enter some spaces

- There are more legal problems with data, so more things are not accessible

- to find stuff you have to check google, but also yandex, or kagi, or chatgpt

- I also check my own index for stuff https://github.com/rumca-js/Internet-Places-Database