I created something different yet similar.
You make GET request to server with any of supported crawlers and obtain result in JSON
https://github.com/rumca-js/crawler-buddy/
Supports request, selenium, Httpx, curl cffi, etc
1,208 karma · joined October 25, 2022
You make GET request to server with any of supported crawlers and obtain result in JSON
https://github.com/rumca-js/crawler-buddy/
Supports request, selenium, Httpx, curl cffi, etc
Crawler results depend on domain authority. If page owner, or page contents page change the ranking may, or should change.
However original author also could change contents, and page ranking should not be changed. So this is not easy to determine what to do with domain of it becomes inactive, or changes contents dramatically.
Currently I use only 30 day window to keep track of domains. After that period inactive domain is thrown out of the window.
However valuable domains, even if dead, reside longer. My UI provides easy link to wayback machine. So even for dead links I can browse them.
I noticed also that some domains, even if expired do serve contents, even if author left it alone. Page contents is served, but with a text that it expired.
There are also good crawlers that search for sites, like Google, or marginalia which gives your page recognizibility. If you lock everything from the web, we'll it disappears from the web.
This is something that was very useful for me not to setup selenium for the x time. I just use one crawling server for my projects.
Link:
Today, it's different. I do think Google played a major role in shrinking the visibility of the independent web. If Google only shows 300 links per search (30 pages x 10 results), anything beyond that effectively doesn't exist for most users.
Some might argue that only the top few results matter, but I disagree. For broad topics like "video games," "war," or "love," you'd expect there to be thousands of worthwhile links—not just the same recycled content from YouTube, TikTok, Reddit, or SEO-heavy content farms.
As a side project, I run a hobby web crawler to explore what's still out there. That's where my perspective comes from regarding the current state of the open web:
https://rumca-js.github.io/quickstart/public/posts/2025-05-2...
- karakeep
- grimoire
- omnivore
- wallabag
- linkwarden
I myself use RSS reader / bookmark manager that I wrote [1]. Everything is open source. Even data [2] [3].
Links
[1] https://github.com/rumca-js/Django-link-archive
Dave? I am afraid I cannot let you search your emails right now. It contains bad stuff from your
Gmail to SQLite describes 6 steps to get credentials working, but it is not true for me. After 6 steps:
- that Google said that my app was not published, so I published it
- Google said that app cannot be internal, because I am not a workspace user
- for external apps
- then it said I cannot use the app until it is verified
- in verification they wanted to know domain, address, other details
- they wanted to have my justification for scopes
- they wanted to have video explaining how the app is going to be used
- they will take some time to verify the data I provided them
It all looks like a maze of settings, where requiring any of users to go above the hoops required by Google is simply too much.
Links:
I decided to take some risk and write my own version.
What grinds my gears about email monopoly is that you cannot create or integrate easily email client now.
To access mail it is not enough to provide user & password & whatever. Things can provide access without compromising security.
To access gmail? User needs to add access from some bullshit console settings, and stuff. I asked chatgpt how thunderbird is not required to do this, and it said that it has keys pregenerated, and big corpos can operate like that. I have not verified that any longer. Sounded credible. So annoying
Now somebody might also find interesting what I have done.
- I decided that implementing RSS reader for 100x time is really stupid, so naturally I wrote my own [0]
- my RSS reader is in form of API [1], which I use for crawling
- can be installed via docker. User has to only parse JSON via API. No need to use requests, browsers, status codes
- my weapon of choice is python. There is python feedparser package, but I had problems in using in parallel, because some XML shenanigans, errors
- my reader, serves crawling purpose, so I am interested in most basic elements, like thumbnails, so all nuance from RSS is lost
- detects feeds from sites automatically
Links
[0] https://github.com/rumca-js/crawler-buddy/blob/main/src/webt...
https://github.com/rumca-js/Internet-Places-Database
Contains Internet links, channels, etc. Work in progress. You can find various domains, channels etc.
Currently I have decided that I can add "Email" as source, to be able to read not only news, but emails in my app.
I added advanced button, for any link, which shows menu where you can navigate to Google translate, internet archive, schema validation, whois pages.
With the menu I can check links, and navigate easily
https://github.com/rumca-js/Django-link-archive/blob/main/RE...
My links about bookmarks and links below.
https://github.com/rumca-js/Django-link-archive - django app, RSS client, simple web crawler, under construction
https://github.com/rumca-js/RSS-Link-Database - my bookmarks repository
https://github.com/rumca-js/RSS-Link-Database-2025 - RSS links for year 2025
https://github.com/rumca-js/RSS-Link-Database-2024 - RSS links for year 2024
https://github.com/rumca-js/RSS-Link-Database-2023 - RSS links for year 2023
https://github.com/rumca-js/Internet-Places-Database - I also maintain a list of domains found in the Internet
https://rumca-js.github.io/search - search that uses links maintained in zip files
https://rumca-js.github.io/music - my music library, browsable
https://rumca-js.github.io/bookmarks - my bookmarks, browsable
- I use RSS to see 'what's new', and to search it. My RSS client support search
- I maintain list of domains, so when I want to find particular place I check my list of domains (I can search domain title, description, etc.). I have 1 million of domains [0]
- If I want more precise information I try to google it
- I also may ask chatgpt
So in fact I am not using one tool to find information. I use many tools, and often narrowing it down to tools that most likely will have the answer.
https://github.com/rumca-js/Internet-Places-Database
I must admit, that this is a difficult task. There are many domains for "hotels", "casinos", so I have to protect myself, just as google agains spam.
https://rumca-js.github.io/music
https://rumca-js.github.io/search
I wrote my own RSS reader. It is self hosted, so tracking is also done on phone.
I gather pages I visit through my reader, browse history, times a page was visited.
It is my window to the internet.
Many apps are apps just because they can collect your data, and create walled gardens. It is harder to create extensions for existing apps, for web pages it is easier.
Besides companies focus on apps, not on web pages. Less money, less focus, therefore worse experience
what data is collected, how/where it's stored
...also if it is sold to anybody, or handed to.
I have bookmarks for most common Internet places: https://rumca-js.github.io/search
I have my music bookmarked also https://rumca-js.github.io/music
I have some parts of the Internet crawled. It takes a long time to boot, but once project is running I can search all domains: https://rumca-js.github.io/internet
Some people complain about AI slop. I do not encounter it.
That does not mean I do not use google, or chatgpt, but only for things I cannot find at my bookmarks.