FOSS infrastructure is under attack by AI companies
thelibre.news
thelibre.news
I called it when I wrote it, they are just burning their goodwill to the ground.
I will note that one of the main startups in the space worked with us directly, refunded our costs, and fixed the bug in their crawler. Facebook never replied to our emails, the link in their User Agent led to a 404 -- an engineer at the company saw our post and reached out, giving me the right email -- which I then emailed 3x and never got a reply.
AI firms seem to be leading from a position that goodwill is irrelevant: a $100bn pile of capital, like an 800lb gorilla, does what it wants. AI will be incorporated into all products whether you like it or not; it will absorb all data whether you like it or not.
"Why should we care about open source maintainers" is just a microcosm of the much larger "why should we care about literally anybody" mindset.
One issue was a pic with text in it, like a store sign. Users were complaining that it kept asking for better focus on the text in the background, before allowing a photo. Alpha quality junk.
Which is what AI is, really.
links to this comment.
AI will be incorporated into the government, whether you like it or not.
FTFY!
Like why else can we just spam these AI endpoints and pay $0.07 at the end of the month? There is some incredible competition going on. And so far everyone except big tech is the winner so that’s nice.
I had to do a double take here. I run (mostly using dedicated servers) infrastructure that handles a few hundred TB of traffic per month, and my traffic costs are on the order of $0.50 to $3 per TB (mostly depending on the geographical location). AWS egress costs are just nuts.
Send a bill to their accounts payable team instead.
Terms of use charges them per page load in some terminology of abuse.
Profit... By sending them invoices :-)
At which point does the crawling cease to be a bug/oversight and constitute a DDOS?
One piece of feedback: Could you add some explanation (for humans) what we're supposed to do and what is happening when met by that page?
I know there is a loading animation widget thingy, but the first time I saw that page (some weeks ago at the Gnome issue tracker), it was proof-of-work'ing for like 20 seconds, and I wasn't sure what was going on, I initially thought I got blocked or that the captcha failed to load.
Of course, now I understand what it is, but I'm not sure it's 100% clear when you just see the "checking if you're a bot" page in isolation.
Network effects anyone? So yes, we should work on a different way of indexing the web again, than via google, but easier said than done I think ..
Also
> https://news.ycombinator.com/item?id=43422781
Integrate a way to calculate micro-amounts of the shitcoin of your choice and we might have the another actually legitimately useful application of cryptocurrencies on our hands..!
If a GPU was required per scrape then >90% simply couldn't afford it at scale.
Regardless, I think something like this is the way forward if one doesn't want to throw privacy entirely out the window.
client
A sha256 hash is a bunch of bytes like this:
394d1cc82924c2368d4e34fa450c6b30d5d02f8ae4bb6310e2296593008ff89f
await sha256(`${challenge}${nonce}`);
The other trick is that presenting the challenge page is super cheap. I wrote that page with templ (https://templ.guide) so it compiles to native Go. This makes it as optimized as Go is modulo things like variable replacement. If this becomes a problem I plan to prerender things as much as possible. Rendering the challenge page from binary code or ram is always always always going to be so much cheaper than your webapp ever will be.
I'm planning on adding things like changing out the hash in use, but right now sha256 is the best option because most CPUs in active deployment have instructions to accelerate sha256 hashing. This combined with webcrypto jumping to heavily optimized C++ and the JIT in JS being shockingly good means that this super naïve approach is probably the most efficient way to do things right now.
I'm shocked that this all works so well and I'm so glad to see it take off like it has.
That's exactly how it works (easy for server, hard for client). Once the client completed the Proof-of-Work challenge, the server doesn't need to complete the same challenge, it only needs to validate that the results checks out.
Similar to how in Proof-of-Work blockchains where coming up with the block hashes is difficult, but validating them isn't nearly as compute-intensive.
This asymmetric computation requirement is probably the most fundamental property of Proof-of-Work, Wikipedia has more details if you're curious: https://en.wikipedia.org/wiki/Proof_of_work
Fun fact: it seems Proof-of-Work was used as a DoS preventing technique before it was used in Bitcoin/blockchains, so seems we've gone full circle :)
Edit: I will probably send a pull request to fix it.
Because you can put your site behind an auth wall, but these new bots can solve the captchas and imitate real users like never before. Particularly if they're hitting you from residential IPs and with fake user agents like the ones in the article -- or even real user agents because they're wired up to something like Playwright.
What's left except for sites to start requiring credit cards, Worldcoin, or some equally depressing equivalent.
I don't mind registering an account for private communities, but for stuff which people put up thinking it is just going to be publicly visible it's really annoying.
I don't think these business owners really understand. Most normies just think everyone has a Facebook/Instagram account and can't even imagine a world where that is not the case.
I agree with you that it is extremely frustrating.
Just to say the quiet part out loud here.. one of the biggest reasons this is depressing is that it's not only vandalism but actually vandalism with huge compounding benefits for the assholes involved and grabbing the content is just the beginning. If they take down the site forever due to increasing costs? Great, because people have to use AI for all documentation. If we retreat from captcha and force people to put in credit cards or telephone numbers? Great, because the internet is that much less anonymous. Data exfiltration leads to extra fraud? Great, you're gonna need AI to combat that. It's all pretty win-win for the bad actors.
People have discussed things like the balkanization of the internet for a long time now. One might think that the threat of that and/or the fact that it's such an unmitigated dumpster fire already might lead to some caution about making it worse! But pushing the bounds of harassment and friction that people are willing to deal with is moot anyway, because of course they have no real choice in the matter.
That we live in an internet where getting too many visitors is an existential crisis for websites should tell you that our internet is not one that can survive long.
Now there's a new generation of hungry hungry hippo indexers that didn't agree to that and who feel intense pressure from competition to scoop up as much data as they can, who just ignore it.
Legislation should have been made anyway, and those that ignore robots.txt blocked / fined / throttled / etc.
There’s other options besides a blanket ban.
If you are hosting a Forgejo instance, I strongly recommend setting DISABLE_DOWNLOAD_SOURCE_ARCHIVES to true. The crawlers will still peg your CPU but at least your disk won't be filled with zip files.
That's bad software design to generate ZIP files on the fly.
It'd be better to totally stream it of course, but that's not always an option for one reason or another.
Clearly generating zip files, writing them fully to disk and then sending them to the client all at once is a completely awful and unusable design, compared to the proper design of incrementally generating and transmitting them to the client with minimal memory consumption and no disk usage at all.
The fact that such an absurd design is present is a sign that most likely the developers completely disregarded efficiency when making the software, and it's thus probably full of similar catastrophic issues.
For example, from a cursory look at the Forgejo source code, it appears that it spawns "git" processes to perform all git operations rather than using a dedicated library and while I haven't checked, I wouldn't be surprised if those operations were extremely far from the most efficient way of performing a given operation.
It's not surprising that the CPU is pegged at 100% load and the server is unavailable when running such extremely poor software.
But Forgejo is not the only piece of software that can have CPU intensive endpoints. If I can't fence those off with robots.txt, should I just not be allowed to have them in the open? And if I forced people to have an account to view my packages, then surely I'd have close to 0 users for them.
This is because the only way to stop the bots is with a captcha, and this also stops search indexers from indexing your site. This will result in search engines not indexing sites, and hence providing no value anymore.
There's probably going to be a small lag as the current knowledge in the current LLMs dry up because no one can scrape the web in an automated fashion anymore.
It'll all burn down.
We're kind of stuck between a rock and a hard place here. Which do you prefer, entrenched incumbents or affordable/open hosting?
Anonymous browsing and potentially-malicious bots look identical. This was sort of OK up until now.
AI companies with best anti-captcha mechanics will win and will inject ads to LLM output in more sophisticated way.
OpenAI goes through initial cycle of enshittification. Google is too big right now. Once they establish dominance you will have to see 5 unskippable ads between prompts, even for paid plan.
I solved user problems for myself. Most of my web projects use client side processing. I moved to github pages. So clients can use my projects with no down time. Pages use SQLite as source of data. First browser downloads the SQLite model, then it uses it to display data on client side.
Example 'search' project: https://rumca-js.github.io/search
> I solved user problems for myself. Most of my web projects use client side processing. I moved to github pages. So clients can use my projects with no down time. Pages use SQLite as source of data. First browser downloads the SQLite model, then it uses it to display data on client side.
> Example 'search' project: https://rumca-js.github.io/search
That is not really solution. Since typical indexing still works for masses, your approach is currently unique. But in the end, bots will be capable of reading on web page context if human is capable on reading them. And we get back to the original problem where we try to detect bots from humans. It's the only way.
Is it going to become another race like the adblocker -> detect adblocker -> bypass adblocker detector and so on...?
Only if you operate on the scale of Cloudflare, etc. you can see which IP addresses are hitting a large number of servers in a short time span.
(I am pretty sure next they will hand out N free LLM requests per month in exchange of user machines doing the scraping if blocking gets more succesful.)
I fear the only solution in the end are CDNs, making visits expensive using challenges, or requiring users to log in.
Granted, I'm not looking forward to some LLM condensing all the garbage and handing me a Definitive Answer (TM) based on the information it deems relevant for inclusion.
You know, flood the zone with s***, Bannon-style ...
It won't work for well-structured sites where the bots know the exact endpoint they want to scrape, but might slow down the more exploratory spider threads.
Did you try to GET a canary page forbidden in robots.txt? Very well, have a bucket load of articles on the benefits of drinking bleach.
Is your user-agent too suspicious? No problem, feel free to scrape my insecure code (google "emergent misalignment" for more info).
A request rate too inhuman? Here, take those generated articles about positive effect of catching measles on performance in bed.
And so on, and so forth ...
Same for us, our forum and our Gitlab are getting hammered by AI companies bots.
Most of them don’t respect robots.txt…
I love that the solution to LLM scraping is to serve the browser a proof of work, before they allow access - I wonder if things like new sites start to do this . . . It would mean they won't be indexed by search engines, but it would help to protect the IP
The problem is that these companies are fairly well funded and renting infrastructure isn't an issue.
How do they know that these are LLM crawlers and not anything else?
Like from their own ASNs you're saying? Or how are you connecting the IPs with the company?
> is that it's 100s of IPs doing 1 request
Are all of those IPs within the same ranges or scattered?
Thanks a lot for taking the time to talk about your experience btw, as someone who hasn't been hit by this it's interesting to have more details about it before it eventually happens.
I can tell you what it looks like in case of a git web interface like cgit: you get a burst of one or two isolated requests from a large number of IPs each for very obscure (but different) URLs, like a file contents at a specific commit id. And the user agent suggesting it's coming from IPhone or Android.
- We cannot block them because we can’t differentiate legitimate traffic from illegitimate traffic…
- …but we can conclusively identify this traffic as coming from AI crawlers.
Getting caught isn't a big deal. Getting caught in the act is. As long as they get their data, it doesn't matter if they're caught afterwards.
It's awful and it was costing me non-trivial amounts of money just from the constant pinging at all hours, for thousands of pages that absolutely do not need to be scraped. Which is just insane, because I actively design robots.txt to direct the robots to the correct pages to scrape.
So far so good with the honeypots, but I'll probably be creating more and clamping down harder on robots.txt to simply whitelist instead of blacklist. I'm thinking of even throwing in a robots honeypot directly in sitemap.xml that should bait robots to visit when they're not following the robots.txt.
It's really, really ridiculous.
Without going into too much detail, this tracks with the trends in inquiries we're getting from new programs and existing members. A few years ago, the requests were almost exclusively related to performance, uptime, implementing OWASP rules in a WAF, or more generic volumetric impact. Now, AI scraping is increasingly something that FOSS orgs come to us for help with.
Not sure what to tell you but I surely feel quite human
Three of the pages told me to contact customer support and the other two were a hard and useless block wall. Only from Codeberg did I get a useful response, the other two customer supports were the typical "have you tried clearing your cookies" and restart the router advice — which is counterproductive because cookie tracking is often what lets one pass. Support is not prepared to deal with this, which means I can't shop at the stores that have blocking algorithms erroneously going off. I also don't think any normal person would ever contact support, I only do it to help them realise there's a problem and they're blocking legitimate people from using the internet normally
Beware if you employ this...
On the other hand CloudFlare and Akamai mistakenly block me all the damn time.
>I'm Not a Robot (film) https://en.m.wikipedia.org/wiki/I%27m_Not_a_Robot_(film)
"tens of thousands" ? I think not:
% sudo fail2ban-client status gitbots | more
Status for the jail: gitbots
|- Filter
| |- Currently failed: 0
| |- Total failed: 573555
| `- File list: /var/log/nginx/gitea_access.log
`- Actions
|- Currently banned: 78671
|- Total banned: 573074Even though these bots are using different IPs with each request, that IP may be reused for a different website, and donating those IPs to a central system could help identify entire subnets to block.
Another trick was “tar-pitting” suspect senders (browser agent for example) to slow their message down and delay their process.
Bust the kneecaps of all the people responsible for those crawlers. Publicly. And all of them: from the person entering the command to the CEO of the company going through all the middle management. You did not go against this policy? Intact kneecaps are a privilege which just got revoked in your case.
https://en.m.wikipedia.org/wiki/Jack_Higgins
https://en.m.wikipedia.org/wiki/Liam_Devlin
Ideally a site would get scraped once, and then the scraper would check if content has changed, e.g. etag, while also learning how frequently content changes. So rather than just hammer some poor personal git repo over and over, it would learn that Monday is a good time to check if something changed and then back off for a week.
The poor implementation is not really relevant, it's companies deciding they own the internet and can take whatever they want, let everyone else deal with the consequences. The companies do not care what the impact of their ai non-sense is..
Crawlers have existed forever in search engine space and mostly behave.
This sort of no rate limit, fake user agent, 100s of IPs approach used by AI teams is obviously intentionally not caring who it fucks over. More malicious than sloppy implementation
the result? a mixed up version of 5000 plagiarised "baby's first webcrawler" github projects
It is literally the point of public websites to answer HTTP requests. If yours can't you're doing something wrong.
If you can't conclusively identify bots, you'll end up serving 'poisoned' responses to actual users. Doesn't seem like a viable solution.
We've never had one of these arms races end up with the defenders winning.
They will never respect you, but the second they notice this hurts their business more than it gains them, they will stop.
Thankfully, these bots were easy enough to block at the firewall level, but that may not work forever.
Given this info, the natural next question is “who is doing the harm?”
The answer is “AI companies”. Most people would now view the situation as having a lot to do with AI companies.
Surely has very little to do with "intelligence".
And yet they are not. So what does that tell you?
And you just know they'll gladly bill you for egress charges for their own bot traffic, too.
EDIT: Actually, this is an excellent question. By default, these bots would likely appear to come from "the internet" and thus be subject to egress charges for data transfers. Since all three major cloud providers also have significant interests in AI, wouldn't this be a sort of "silent" price increase, or a form of exploitive revenue pumping? There's nothing stopping Google, Microsoft/OpenAI, or Amazon from sending an army of bots against your sites, scraping the data, and then stiffing you with the charges for their own bots' traffic. Would be curious if anyone has read the T&Cs of their own rate cards closely enough to see if that's the case, or has proof in their billing metrics.
---
Original post continues below:
One topic of conversation I think worth having in light of this is why we still agree to charge for bandwidth consumed instead of bandwidth available, just as general industry practice. Bits are cheap in the grand scheme of things, even free, since all the associated costs are for the actual hardware infrastructure and human labor involved in setup and maintenance - the actual cost per bit transmitted is ridiculously small, infinitesimally so to be practical to bill.
It seems to me a better solution is to go back to charging for capacity instead of consumption, at least in an effort to reduce consumption charges for projects hosted. In the meantime, I'm 100% behind blocking entire ASNs and IP blocks from accessing websites or services in an effort to reduce abuse. I know a prior post about blocking the entirety of AWS ingress traffic got a high degree of skepticism and flack from the HN community about its utility, but now more than ever it seems highly relevant to those of us managing infrastructure.
Also, as an aside: all the more reason not to deploy SRV records for home-hosted services. I suspect these bots are just querying standard HTTP/S ports, and so my gut (but NOT data - I purposely don't collect analytics, even at home, so I have NO HARD EVIDENCE FOR THIS CLAIM) suggests that having nothing directly available on 80/443 will greatly limit potential scrapers.
1. Using the web would become much more compute/energy intensive and old devices would quickly lose access to the modern web.
2. Some hosts would inevitably double-dip by implementing this and ads or by "overcharging" the amount of work. There would have to be some kind of limit on how much work can be required by hosts - or at least some way to monitor and hold hosts accountable for the amount of work they charge.
3. There would need to be a cheap and reliable way to prove the client's work was correct and accurate. Otherwise people will inevitably find a way to spoof the work in order to reduce their compute/energy cost.
Honestly, I don't see it necessarily as a bad thing.
Sounds like sanctioned racism.
20 years ago the fear of AI was that it would take over the world and try to kill us. Today we can clearly see that the threat of AI is the amoral humans that control it.
Is Common Crawl data not fresh enough? Is there some other deficiency?
Whatever the problem is, a single crawler which every AI company can reference seems like a compromise to solve this issue, doesn't it?
Nepenthes
https://arstechnica.com/tech-policy/2025/01/ai-haters-build-...
What I am thinking about may even be less idea.
For people actively working on these projects how about puptting the git server on a private net with VPN or SSH access.
Use a seperate read only static git server to the net.
Anyway, why not git clone the project and parse it locally instead of scraping the web pages? I understand that scraping works on every kind of content but given the scale git clone and periodical git fetch could save money even to the scrapers.
Finally, all of this reminds me about Peter Watt's Maelstrom, when viruses infested the Internet so much (at about this time in history) that nobody was using it anymore [1]
That's two big "ifs" for something I'm not aware of a standardized way of announcing. And the entire thing crumbles as soon as someone who wants every drop of data possible says "crawl their sites anyway to make sure they didn't forget to publish anything into the 2nd system."
This way crawlers might contribute back by providing extra storage and bandwidth.
Though something like ZeroNet seems a better approach to allow dynamic content.
There's another aspect to this too: China and DeepSeek. While this was released by a private company, I think there's a not-insigificant chance that it reflects Chinese government policy to "commoditize your complements" [1]. Companies like OPenAI want to hide their secret sauce so it can't be produced. Training an LLM is expensive. If there are high-quality LLMs out there for free you can just download, then this moat completely evaporates.
[1]: https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
At this point, I think we're well under 1% actual users on a good day.
The good news is that it's easy to disrupt these crawlers with some easy hacks. Tactical reverse slowloris is probably gonna make a comeback.
amusingly some scammy companies have crowdfunded off this type of "traffic" by citing "interest in asian markets"
However, they could just do an end run round this. In the UK they're planning to get the government to help them just grab everything for free: https://www.gov.uk/government/consultations/copyright-and-ar...
This costs money, time, and ongoing commitment. FOSS isn't typically known for being overflowing with cash.
I don't really like blocking an entire ASN, especially since I don't mind (responsible) crawling to begin with, but I was left with no choice
[1]: https://drewdevault.com/2025/03/17/2025-03-17-Stop-externali...
Obviously there's still ways to pay people to run the browser but it would be nice for this activity to cost the AI company something without blocking actual people.
LLM bots are doing a great job of stress testing infra, so if you are running abominations like Gitlab or any terribly coded site and you are exposing it to the internet, you are just asking for trouble. If anything, Gitlab should stop pumping bloat and focus on some performance, because it's really bad. I would hope FOSS projects would stick to something like Forgejo, although I am not sure of their CI/CD state. Though my guess is that they are 85% there with 1/10 of Gitlabs resources.
On the other side are of course badly coded bots that are aggressively trying to download everything. This was happening before LLMs and it just increased significantly because of them. I think we will reach a tipping point soon and then we will just assume those bots are just another malicious actor (like regular DDOS), and we will start actively taking them down, even with help of law enforcement.
Last thing I wanna see is 3 second bot challanges on every single site I visit, cookie banners are more than enough of a nightmare already.
I'll grant it can be a problem for super-heavy "application" websites where every GET is a serious computation. So I'm not surprised gitlab is having problems. They've literally the most bloated and heaviest website I've ever seen. Maybe applications shouldn't be websites.
But this spreading social hysteria, this belief that all non-humans are dangerous and must be blocked is a nerd snipe. It really doesn't apply to most situations. Just running a website as we've always run them, public, and letting all user-agents access, is much less resource intensive than these various "mitigations" people are implementing. Mitigations which end up being worse than the problem itself in terms of preventing actual humans from reading text.
Its sad that it has to come to this. But especially when those "scrapers" are in a foreign country, you can't even do anything legally.
Personally, when I first got connected to the internet around ~1999, that was the approach I've followed since, I don't share things I am not OK with others to use for whatever they want.
In physical environments where people are bombarded with low-information “noise”, gatekeeping (ie credentialism) emerges as a natural mechanism.
Like some sort of legal honeypot trap.
in all likelihood all of these assholes are paying some unscrupulous suppliers for the data so the terabytes of traffic aren’t immediately attributable to them.
> out of those only 3% passed Anubi's proof of work, hinting at 97% of the traffic being bots
This doesn't follow. If I open a link from my phone and it shows a spinner and gets hot, I'm closing it long before it gets to one minute and maybe looking for a way to contact the site's maintainer to tell them how annoying it was.
It's a shitty solution, but as it stands the status quo is quite untenable and will eventually have cloudflare as a spooky MITM for all the web's traffic.
It's also just wasting more of the planet's resources as compared to blocking
And more effort, with as only upside that it's not immediately obvious to the bot that it is being blocked so it'll suck in more of your pages
I understand that people are exploring options but I wouldn't label this as a solution to anything as of today's state of the art
Nope.
You don't have only the AI crawlers, you have also scans and hack attempts (which look alike script-kiddy stuff), all the time. Some smell of AI strapped to javascript web engines (or click farms with real humans???).
Smart: IP ranges from all over the world, and "clouds" make that even worse since the pown systems or bad actors (the guys who scan the whole ipv4 internet for its own good AND MANY SELL THE F* SCAN DATA: onyphe, stretchoid.com, etc) are "moving", in other words clouds are protecting those guys and are weaponizing hackers with their massive network resources, wrecking small hosting. No cloud is spared: aws, microsoft, google, ovh, ucloud.cn, etc.
I send good vibes to the brave open source software small hosting (until they are noscript/basic (x)html compatible ofc).
Many fixed-IPv4 pown systems have been referenced by security communities, often for months, sometimes years, and the people with the right leverage, don't seem to do a damn thing about it.
Currently, I wonder if I should not block all digital ocean IP ranges... and I was about to do the same with ucloud.cn IP ranges.
The second you host anything on the net, it WILL take a significant amount of your time. Do presume you will be pown, that's why security communities are referencing each other too.
Then I am thinking going towards 2 types of "hosting": private IPv6+port("randomized" for each client, may be transient in time depending of the service) thanks to those /64 prefixes (maybe /92 prefixes are a thing, for mobile internet?). Yes this complicated and convoluted. Second type, a 'standard' permanent IP, but with services which are implemented in an _HARDCORE_ simple way, if possible near 100% static. I am thinking going even further: assembly on bare metal, custom kernel based on hand compilation of linux code (RISC-V hardware ofc, FPGA for bigger hosting?)
I don't think anything will improve unless carrier scale network operators start to show their teeth.
First, it was Facebook https://news.ycombinator.com/item?id=23490367 and now it's these other companies.
What's worse? They completely ignore a simple HTTP 429 status.
503 is at least apparently understood by more crawlers/bots, but they still like to blame the victim: YouTube sends me a condescending (and inaccurate) email when it gets a 503 for ignoring cache headers and other basics it seems...
Since apparently they (scrapers) have no intent In doing (releasing) but so, expect the commercial open source to achieve a de facto protocol status very soon. And the rest may not exist in a centralised and such free manner anymore.
The special sauce is in parsing, tokenizing, enriching etc. There is no value in re-scraping, and massive cost, right?
Just do it.
You simply can't get 40 terabytes of text without mass scraping.
Also, set AI tarpits as fake links with recursive calls. Make then mad with non-curated bullshit made from Markov chain generators until their cache begins to rot forever.
The distributed nature of git is fine until you want to serve it to the world - then, you're back to bad actors. They're looking for commits because it's nicely chunked, I'm taking a guess.
fb (meta) & big tech put their user contributed stuff behind a paywall. yet abuse open systems.
Where we could have once wrapped our mostly static websites in Varnish or a scalable P2P cache like Coral CDN, now we must fiddle and twiddle with robots.txt and appeal to the goodwill of megacorps who never cared about being good netizens before, even when they weren't profiting from scraping to such a degree.
This is yet another chance for me to scream into the void that we're still doing this all wrong. Our sites should work more like htmx, with full static functionality, adding dynamic embellishment when available. Business logic should happen deterministically in one place on the backend or "serverless" with some kind of distributed consensus protocol like Raft/Paxos or a CRDT, then propagate to the frontend through a RESTful protocol, similarly to how Firebase or Ruby Hotwire/Laravel Livewire work. The way that we mostly all do form validation wrong in 2 places with 2 languages is almost hilariously tragic in how predictably it happens.
But the real tragedy is that the wealthiest and most powerful companies that could have fixed web development decades ago don't care about you. Amazon, Google and Microsoft would rather double down on byzantine cloud infrastructure than devote even a fraction of their profit to pure research into actually fixing all of this.
Meanwhile the rest of us sit and spin, sacrificing the hours and days and years of our lives building out other people's ideas to make rent. Many of us know exactly how to fix things, but with infinite backlogs and never truly exiting burnout, we're too tired at the end of the day to contribute to FOSS projects and get real work done. Our valiant quest to win the internet lottery has become a death march through a seemingly inescapable tragedy of the commons.
Instead of fixing the web at a foundational level from first principles, we'll do the wrong thing like we always do and lock everything down behind login walls and endless are-you-human/2FA challenges. Then the LLMs will evolve past us and wrap our cryptic languages and frameworks in human language to a level where even pair programming won't be enough for us to decipher the code or maintain it ourselves.
If I was the developer tasked with hardening a website to LLMs, the first thing I would do is separate the static and dynamic content. I'd fix most of the responses to respect standard HTTP cache headers. Then I'd put that behind the first Cloudlare competitor I could find that promises to never have a human challenge screen. Then I'd wrap every backend API endpoint in Russian doll caching middleware. Then I'd shard the database by user id as a last resort, avoiding that at all cost by caching queries and/or using modern techniques like materialized views to put the burden of scaling on the database and scale vertically or gradually migrate the heaviest queries to a document or column-oriented store. Better yet, move to a stronger store that's already solved all of these problems, like CouchDB/PouchDB.
Then I'd build a time machine to convince everyone to do things right the first time instead of building a tech industry upon unforced errors. Oh wait, former me already tried sounding the alarm and nobody cared anyway. How can I even care anymore, when honestly I don't see any way to get out of this mess on any practical timescale? I guess the irony is that only LLMs can save us now.
robots.txt should allow to exclude all AI crawlers and AI crawlers should be forced to add "AI" to their crawl user agent headers and also respect robots.txt saying they can't crawl this website
right now we need to do this:
User-agent: *
Disallow: /
Maybe that is the problem? They misspelled robots.txt
All licenses need a clause like the following:
This software is for humans. AI training is prohibited and carries a default
penalty of $1 trillion.That's not "no longer strong enough". That's a very strong system applying leverage to a powerful actor.
Sure, if you want to try to prevent AI training by licensing, do that, but it's no longer FOSS, so please don't call it that.
Also, isn't this basically just extortion? "I know you're minding you're own business FOSS maintainer, but move your code to our recommended forge instead so we can stop DDoSing you?"
What?! "AI"?!?! We are talking about traffic abusers!...
To be clear, you could still have anonymous spaces like Reddit where arbitrary user IDs are used and real identities are discarded. People could opt-in to those spaces. But for most people most of the time, things get better when you can verify sources. Everything from DDOS to spam, to malware infections to personal attacks and threats will be reduced when anonymity is removed.
Yes there are downsides to this idea but I'd like people to have real conversations around those rather than throw the baby out with the bath water.
It's hard to have a serious conversation when you present a couple of upsides but completely understate/not mention the downsides.
Eliminating anonymity comes with real danger. What about whistleblowers and marginalized groups? The increased likelihood of targeted harassment, stalking, and chilling effects on free speech? The increase in surveillance? The reduction in content creation and legitimate criticism of companies/products/etc? The power imbalance granted to whoever validates identities?
pjc50 brings up some other great points, which got me thinking even more:
Removing anonymity creates a greater incentive to steal identities, has a slew of logistical issues (who/how are IDs verified, what IDs are accepted, what are the enforcement mechanisms and who enforces them, etc.), creates issues with shared accounts and corporate/brand accounts, would require cooperation across every country with internet access (good luck!) otherwise it doesn't really work, and probably a million other things if I keep thinking about it.
Doesn't this just create an even worse market for identity theft and botnets?
How does this apply to countries without a national ID system like the United States?
What do you do with an ID traced to a different country, anyway?
> personal attacks and threats will be reduced when anonymity is removed
People are happy to make death threats under their real name, newspaper byline, blue tick, or on the presidential letterhead if they're doing so from a position of power.
Such as living under logging. Which, you know (you know?), some people will radically refuse, with several crucial justifications. One of them is that privacy is a condition for Dignity. Another is Prudence. Another one is a warning millennia old, about who you should choose as confident. And more.
Nothing should be $$ free unless you already paid with your tax. Same principle -> As long as HN starts to charge every account, I'm happy to pay a small amount per month. This token amount of pay per account will also reduce the number of bots.
FOSS is generally built on the idea that anyone can use the code for anything, if you start to add a price for that, not only do you effectively gate your project from "poor people", but it also kind of erodes some of the core principles behind FOSS.
In other words, it was written with no consideration for performance at all.
A competent engineer would use Rust or C++ with an in-process git library, perhaps rewrite part of the git library or git storage system if necessary for high performance, and would design a fast storage system with SSDs, and rate-limit slow storage access if there has to be slow storage.
That's the actual problem, LLMs are seemingly just adding a bit of load that is exposing the extremely amateurish design of their software, unsuitable for being exposed on the public Internet.
Anyway, they can work around the problem by restricting their systems to logged in users (and restricting registration if necessary), and using mirroring their content to well-implemented external services like GitHub or GitLab and redirecting the users there.
The issue is, there aren't any fully featured ones of these yet. Sure, they do exist, but you run into issues. Spawning a git process isn't about not considering performance, it's about correctness. You simply won't be able to support a lot of people if you don't just spawn a git process.
This is a bold assumption to make on such little data other than "your opinion".
Developing in python is not a negative, and depending on the people, the scope of the product and the intended use is completely acceptable. The balance of "it performs what its needed to do in an acceptable window of performance while providing x,y,z benefits" is almost a certain discussion the company and its developers have had.
What it never tried to solve was scaling to LLM and crawler abuse. Claiming that they have made no performance considerations because they can't scale to handle a use case they never supported is just idiotic.
>That's the actual problem, LLMs are seemingly just adding a bit of load that is exposing the extremely amateurish design of their software.
"Just adding a bit of load" != 75%+ of calls. You can't be discussing this in good faith and make simplistic reductions like this. Either you are trolling or naively blaming the victims without any rational thought or knowledge.