Luckily all that is becoming a non-issue, as most content on these websites isn't worth scraping anymore.
In fact, at this point in time (it won't last), one of the most useful applications of LLMs is to have them deal with all the user-hostile crap that's bulk of the web today, so you don't have to suffer through it yourself. It's also the easiest way to get any kind of software interoperability at the moment (this will definitely not last long).
What you want is to just download the 10-20kb html file, maybe a corresponding css file, and any images referenced by the html. Then if you want the video you just get the video file direct.
Simple and effective, unless you have something to sell.
> unless you have something to sell
Video hosting and its moderation is not cheap, sadly. Which is why we don't see many competitors.
(before you ask: Vimeo is getting sold to an enshitification company)
And a decent list here: https://ideaexplainers.com/video-sites-like-youtube/
Not actually heard of the first two, what's their USP?
Twitch has the issues the other commenter described, and both Twitch and Facebook are owned by billionaires who are actively collaborating with the current authoritarian regime. Facebook in particular is a risk space for actually exercising free speech and giving coherent critiques of authority.
Dailymotion is... maybe okay? As a company it seems like it's on life support. There's a "missing middle" between the corporate highly produced content that's distributed across all platforms and being a long tail dumping ground. I did find things like university lectures there, but there isn't creators actually trying to produce content for Dailymotion like there is on YouTube.
So, just like Youtube, then?
What we don't see is more web video services and services that successfully trick varied content creators to upload regularly to their platform.
https://en.wikipedia.org/wiki/PeerTube also must be mentioned here.
Soon, LLMs will be able to complete any Captcha a human can within reasonable time. When that happens, the "analog hole" may be open permanently. If you can point a camera and a microphone at it, the AI will be able to make better sense of it than a person.
That's true for static content, but much of it is forums and other places like that where the main value is that new content is constantly generated - but needs to be re-scraped.
Everyone thinks they're a master fisher, and gets up in arms about those pesky users not acting dumb fish, instead trying to eat the content bait without biting the monetization hook.
If you are just asking Siri to load a page for you, that probably gets tolerated. Maybe very sensitive sites will go verified mobile platform only and Apple/Google will provide some kind of AI free compute environment like how they can block screen recording or custom roms today.
Yes it is 100% the death of the free and open computing environment. But captchas are no longer going to be sufficient. It seems realistic to block bots if you are willing to fully lock down everything.
This seems like an awful future. We already had this in form of limited ipv4 addresses wher each IP is basically an identity. People started buying up ip addresses and selling them as proxies. So any other form of ID would suffer the same fate unless enforced at government level.
Worst case scenario we have 10,000 people sitting in front of the screens clicking page links because hiring someone to use their "government id" to mindlessly browse the web is the only way to get data of the public web. That's not the future we should want.
I can literally just go write a script that uses headless firefox + mitmproxy in about an hour or two of fiddling, and as long as I then don't go try to run it from 100 VPS's and scrape their entire website in a huge blast, I can typically archive whatever content I actually care about. Basically no matter what protection mechanisms they have in place. Cloudflare won't detect a headless firefox at low (and by "low" I mean basically anything you could do off your laptop from your home IP) rates, modern browser scripting is extremely easy, so you can often scrape things with mild single-person effort even if the site is an SPA with tons of dynamic JS. And obviously at low scale you can just solve captchas yourself.
I recently wrote a scraper script that just sent me a discord ping whenever it ran into a captcha, and i'd just go look at my laptop and fix it, and then let it keep scraping. I was archiving a comic I paid for but was in a walled-garden app that obviously didn't want you to even THINK of controlling the data you paid for.
this is absolutely not the case. I've been web scraping since 00s and you could just curl any html or selenium the browser for simple automation but now it's incredibly complex and expensive even with modern tools like playwright and all of the monthly "undetectable" flavors of it. Headless browsers are laughably easy to detect because they leak the fact they are being automated and that they are headless. Not to even mention all of the fingerprinting.
I made a web scraper in Perl a few years ago. It no longer works because I need a headless browser now or whatever it is called these days.
Web scraping is MUCH WORSE TODAY[1].
[1] I am not yelling, just emphasizing. :)
I think he means the JS part is now easy to run and scrape compared to the transition time from basic download scraping to JS execution/headless browser scraping. It is more complex but the tools haven’t been as evolved as they are now a couple of years ago.
* PeerTube and similar platforms for video streaming of freely-distributable content;
* BitTorrent-based mechanisms for sharing large files (or similar protocols).
Will this be inconvenient? At first, somewhat. But I am led to believe that in the second category one can already achieve a decent experience.
2. N/A, but enough content creators on YT are very much aware of the kind of prison it is, especially in the years after the Adpocalypse.
3. Obviously, nobody should be able to monetize the copying of content. If it is released, it is publicly released. But they can use LibrePay/Patreon/Buy me a coffee, they can sell merch or signed copies of things, they can do live appearances, etc.
All thanks to great ideas like downloading the whole internet and feeding it into slop-producing machines fueling global warming in an attempt to make said internet obsolete and prop up an industry bubble.
The future of the internet is, at best, bleak. Forget about openness. Paywalls, authwalls, captchas and verification cans are here to stay.
Personally, when it became available, o3 model in ChatGPT cut my use of web search by more than half, and it wasn't because Google became bad at search (I use Kagi anyway) - it's because even the best results are all shit, or embedded in shit websites, and the less I need to browse through that, the better for me.
I suppose that's thanks to Google and their search algos favoring ad-ridden SEO spam. LLMs are indeed more appealing and convenient. But I fear that legitimate websites (ad-supported or otherwise) that actually provide useful information will be on the decline. Let's just hope then that updated information will find its way into LLMs when such websites are gone.
It's a bit of a gamble at this point - will the larger models, or new architectures, or training protocols, be able to reject all that noise and extract the signal? If yes, then training on the Internet is still safe. If not, it's probably better for them to freeze the datasets blindly scrapped from the Internet now, and focus on mining less poisoned sources (like books, academic papers, and other publications not yet ravaged by the marketing communications cancer[0], also ideally published before the last 2 years).
I don't know which is more likely - but I'm not dismissing the possibility that the models will be able to process increasingly poisoned data sets just fine, if the data sets are large enough, because of a very basic and powerful idea: self-consistency. True information is always self-consistent, because it reflects the underlying reality. Falsehoods may be consistent in the small, but at scale they're not.
I want them to go overboard. I want BigTech to go nuts on this stuff. I want broken systems and nonsense.
Because that’s the only way we’re going to get anything better.
At this point I don't know - I still have the feeling that "they just need to make it 50% worse again and we'll get a competitor," but I've seen too many of these platforms get 50% worse too many times, and the network effect wins out every time.
It doesn't work. There aren't any collapses like that to be had. Big change happens incrementally, a bit of refactoring and a few band-aids at a time, and pushing to make things worse doesn't help.
You can step away from the world (right now, no waiting required). But the world can remain irrational longer than you can wait for it to step away from you, and pushing for more irrationality won't make a dent in that.
The basic governing principles of the economy were completely rewritten in 1971, were completely rewritten again in 2008, were completely rewritten again in 2020 - probably other times too - and there are only so many more things they can try. The USA is basically running as a pseudo-command economy at the top level now - how long do those typically last? - with big businesses being supported by the central bank.
The economy should have collapsed in 1971, 2008 and 2020 (and probably other times) as well, but they kept finding new interventions that would have seemed completely ludicrous 20 years earlier. I mean, the Federal Reserve just buying financial assets? With newly printed money? (it still has a massive reserve of them, this program did not end, that money is still in circulation and it's propping a lot of economic numbers up)
All predictions about when the musical chairs will end are probably wrong. The prediction that it'll end in the next N years is just as likely to be wrong, as the prediction that it won't. Some would argue it already has ended, decades ago, and we are currently living in the financial collapse - how many years of income does it take to get a house now? The collapse of Rome' took several centuries. At no point did the people think they were living in a collapsing empire. Each person just thought that how it was in their time was how it always was.
During the "things get worse" phase, why not make it shorter?
The year is 2003. Svn and cvs are proving to be way too clunky and slow for booming open source development.
As an ethical accelerationist, you gain commit access to the repos for svn and cvs and make them slower and less reliable to accelerate progress toward better version control.
Lo and behold, you still have to wait until 2025 for git to be released. Because git wasn't written to replace svn or cvs-- it was written as the result of internal kernel politics wrt access to a closed-source source management program Bitkeeper. And since svn and cvs were already bad enough that kernel devs didn't choose them, you making them worse wouldn't have affected their choice.
Also, keep in mind that popularity of git was spurred by tools that converted from svn to git. So by making svn worse, you'd have made adoption of git harder by making it harder on open source devs to write reliable conversion tools.
To me, this philosophy looks worse than simply doing nothing at all. And this is in a specific domain where you could at least make a plausible, constrained argument for accelerationism. Your comment instead seems to apply to accelerationism applied to software in general-- there, the odds of you being right are so infinitesimal as to be fatuous.
In short, you'd do better playing the lottery because at least nothing bad happens to anyone else when you lose.
Because it never gets better for the people actually living through it.
I imagine those in favor of the idea of accelerating collapse aren't all so purely selfless that they're willing to see themselves and their children suffer and die, all so someone elses' descendants can live in a better world.
Nah, they just aren't thinking it through.