renegat0x0

Most web scrapers, even if illegal, are for... business. So they scrape amazon, or shops. So yeah. Most unwanted traffic is from big tech, or bad actors trying to sniff vulnerabilities.

I know a thing or two about web scraping.

There are sometimes status codes 404 for protection, so that you skip this site, so my crawler tries, as a hammer, several of faster crawling methods (curlcffi).

Zip bombs are also not for me. Reading header content length is enough to not read the page/file. I provide byte limit to check if response is not too big for me. For other cases reading timeout is enough.

Oh, and did you know that requests timeout is not really timeout a timeout for page read? So server can spoonfeed you bytes, one after another, and there will be no timeout.

That is why I created my own crawling system to mitigate these problems, and have one consistent mean of running selenium.

https://github.com/rumca-js/crawler-buddy

Based on library

https://github.com/rumca-js/webtoolkit

Mobile is where it is all going. PCs will be like android in the near future.

I was expecting OneDrive subscription message to go up from time time time.

I use python poetry, i have not genereted requirements file for a long time. For current deps you can check pyproject file.

I do not use feedparser, because it could not parse properly some rss files. I implemented my own lib for rss parsing.

I run something I call an "ethical crawler". It’s designed to avoid being a burden to websites - it makes requests very infrequently. Crawling the internet reliably has become increasingly difficult, as more and more content is protected or blocked. It’s especially frustrating when RSS feeds are inaccessible to bots.

404 definitely are not a problem for me. My crawler tests different mechanisms and browser headers while exploring the web.

My scraping mechanism:

https://github.com/rumca-js/crawler-buddy

Web crawler / RSS reader

https://github.com/rumca-js/Django-link-archive

Not really sure if that counts but I also created a db centered app.

I do a lot of meta link archiving and I wanted to create a link browser. My project supports data in zipped sqlite, or JSONs.

I believe data are as much important as programs, So everything is public.

Links

https://rumca-js.github.io/search - search (JSONs)

https://rumca-js.github.io/music - music (JSONs)

https://rumca-js.github.io/bookmarks - bookmarks (JSONs)

https://github.com/rumca-js/Internet-Places-Database - scripts use SQLite

https://github.com/rumca-js/web_link_browser - clean repo of scripts

Still gathering internet links

https://github.com/rumca-js/Internet-Places-Database

Still crawling framework

https://github.com/rumca-js/crawler-buddy

Still RSS client

https://github.com/rumca-js/Django-link-archive

I self host my Search Engine / RSS reader. I track every page I visit from nearly all devices.

Since my basic search engine is self hosted nobody actually sees what I visit, and what I watch.

This is my conclusion seeing that social media algorithm is totally lost at what I would like to watch next.

Also I am in control over UI, and changes, which is a good and a bad thing

Some links

https://github.com/AboutRSS/ALL-about-RSS

https://github.com/plenaryapp/awesome-rss-feeds

My problem with most RSS do not have great search. With 500+ sources this can become problem.

https://github.com/rumca-js/Django-link-archive - my own project

Dogs will always bark. I daily drive linux, and I am happy with it. Majority will not make the switch because they either are dependent on the office, adobe, or video editing software.

Linux user base grows. One tiny percent of percent every year. Too little to make a dent.

Two things:

- I wanted to install minecraft for my boy. I couldn't just start a game. It required me to tie in account to the system. First I don't like that. Second is that we have lost pin, and could not remember password or something, so we have gone through maze of xbox, microsoft online settings and whatnots. On Linux? Pass account, and password, we are set to go

- The second encounter was with playing a older FIFA game. After launching it complained about some dll file. Turned out it was missing VC redistributable. You had to manually find, and visit, Microsoft page, download old type installer, which my kid was not familiar with (Next, Next, C: program files, I accept).

After so many occasions I see that linux is even easier than Windows. I use bottles to play Warhammer Boltgun. Sure it stutters some times, but not that much. The pros and cons of having Linux looks better every day.

- I can confirm that the web archive can be really slow

- I think I have seen that AI scrapers create bottleneck in the bandwidth

- To some digital archives you need to create scientific accounts (I think Common Crawl works like that)

- Data quite easily can be very big. The goal is to store many things. We not only store Internet, but with additional dimension of time

- Since there is a lot of data, it is difficult to navigate it, search it, so it easily can become unusable

- For example that is why I created my own meta data link, I needed some information about domains

Link:

https://github.com/rumca-js/Internet-Places-Database

Most people complain about the signal-to-noise ratio in news consumption. I believe the issue isn’t the news sources themselves, but rather the lack of a proper RSS application.

A great RSS app should offer a powerful search function. It should support tagging, bookmarking, scoring or point systems, categories, and a "read later" feature, among other things.

You don’t need to eliminate news sources — just use filters and search tools to surface what matters to you.

An ideal RSS reader should also be smart enough to bypass things like Cloudflare and other unnecessary protections that break RSS functionality. Unfortunately, many mobile RSS apps fall short in this regard — and mobile is king these days.

To get something truly useful, you often need to self-host. But most people won’t go that far.

Personally, I self-host my RSS reader. I even built my own client, since I wasn’t aware of KaraKeep (formerly Hoarder) at the time. I’m still using my custom app because it’s now very versatile, and I’m not sure KaraKeep would meet all my needs.

Links:

https://github.com/rumca-js/Django-link-archive - my own project

https://github.com/AboutRSS/ALL-about-RSS - all about RSS

https://rssisawesome.com/

https://rssgizmos.com/

https://github.com/plenaryapp/awesome-rss-feeds

What I have seen it is hard to tell what "serious scrapers" use. They use many things. Some use this, some not. This is what I have learned reading webscraping on reddit. Nobody speaks things like that out loud.

There are many tools, see links below

Personally I think that running selenium can be a bottle neck, as it does not play nice, sometimes processes break, even system sometimes requires restart because of things blocked, can be memory hog, etc. etc. That is my experience.

To be able to scale I think you have to have your own implementation. Serious scrapers complain about people using selenium, or derivatives as noobs, who will come back asking why page X does not work in scraping mechanisms.

https://github.com/lexiforest/curl_cffi

https://github.com/encode/httpx

https://github.com/scrapy/scrapy

https://github.com/apify/crawlee

There are two sides of the story, as usual

- we already have some forms of IDs. Most of us have personal ID, driving license, you data exist in government tables

- we already have some forms of IDs online. Everything is stored per email address, and phone address

- this makes some people do not care

- until government starts using it for nefarious reasons. All it takes is one president to change, one super power to shift

- people are ignorant, until it is too late

- the problem with digital ID is with centralization, that it is goldmine for hackers, it can be used to spy on you if all your data are in one bucket

- therefore people who say that they have digital ID lack imagination on what can happen

- it all works, when everything is disconnected, platforms, governments, etc.

Whoa. That is 20% more than I thought

I maintain my own link meta archive. Just because I know that it will stay, but I am aware of link rot. I know also that internet archive exists, and also I know that it works painfully slow.

I want to be able to at least browse headlines and titles about Jeffrey Epstein after 20 years, when they start erasing all history about him.

Links:

https://github.com/rumca-js/RSS-Link-Database-2025 - Year 2025

https://github.com/rumca-js/RSS-Link-Database-2024 - Year 2024

https://github.com/rumca-js/RSS-Link-Database-2023 - Year 2023

https://github.com/rumca-js/RSS-Link-Database-2022 - Year 2022

https://github.com/rumca-js/RSS-Link-Database-2021 - Year 2021

- there is no escape from digital techno feudalism

- you will have to obey corporations

- sooner or later everything will work using digital ID, or some other IDs

- sooner or later phones, PCs, browsers, will be locked in

- majority of populations will have no problems about that, aka golden cage

- I do not such a future exists when it will not look like this

- I am uncertain what is the future of open source. I think it also will be regulated by accounts, digital IDs. You will not be able to participate in open source without verification

Yesterday I seen someone claiming on Reddit that they stopped using mailing list in favour of using rss

I was wondering whether old web means people, or the old web is entirety of old web. Certainly companies will not provide features of old. No RSS for you, since they want to control what you see with the "algorithm".

What about people? Well some of them, as most of you pointed, want to monetize your attention, so it is not necessarily that they create something from passion, but to use you.

Besides that most of content is Audio-Video now, so it has two outcomes. One is that most of people chasing attention will create Audio-Video. This allows us to find niche and simple blogs, which is difficult, because Google will not index them, or not rank them.

What we are left are curated list of blogs. There are some.

That is why I use my own domain index

https://github.com/rumca-js/Internet-Places-Database

I think it is not about making a change, it is putting money where your mouth is.

To buy premium to support creators.

Once yt becomes hostile the deal between me and yt is off.

Microsoft new about piracy, and used it to become popular.

YouTube plays a game with downloaders to be and stay popular, supported even by niche devices.

On the other hand they will use vague license statements to strike opponents.

Even RSS YouTube is accessible to bots, but block d by robots. They can always claim you are doing something wrong of they wanted.

I performed some transition from django pure template solution to more javascript oriented.

- my server was running on raspberry PI, now heavy responses are returned via JSON, I think REST-like data. Which makes me offload burden to clients

- javascript makes the page more responsive. Parts can be loaded via separate calls, and it divides something that would be otherwise a monolith into subsequent small calls

- In my experience nearly all simple binary decisions "use only X", and "use only Y" solutions are bad

thanks, I replaced it with a other link demo

Any link list, or domain list is not worth much without any rating, or meta. I lead a hobby project, and I am not expert, so I provide ratings based on what kind of data pages provide (title, social, description), and my own manual voting system. It is not ideal, but it is something. Also I provide tags, so it is easily known what the domain provides, or domains can be filtered by tags.

I know that you cannot count and visit every domain, so the list will never be finished, but I am happy with the results.

Well, I created my own domain index. I have not crawled every page inside domains, but it is not my goal.

I have 1542766 domains. Might not be much, but it is an honest work.

It is available as a github repo, so anybody that wants to start crawling has some initial data to kick off.

Links

https://github.com/rumca-js/Internet-Places-Database

- I still use RSS

- Some major platform still provide RSS, which makes me use them (I do not use twitter, because it does not provide RSS

- If not for RSS I would not be using Reddit

- the moment platform drops RSS, I drop the platform

Links:

[0] https://github.com/rumca-js/Django-link-archive - my own RSS reader

To be honest most of my apps are web pages now. Even on my phone I do not use any more than the default apps. For what is missing I have written my own self-hosted pages.

I sometimes wonder why people "synchronize" anything, since everything is in my self-hosted instance.

Huh, I thought that AI was made to be magic. Click and it generates code. Turns out it is like magic, but you are an apprentice, and still have to learn how to wield it.