I fear for the unauthenticated web
sethmlarson.dev
sethmlarson.dev
<https://thelibre.news/foss-infrastructure-is-under-attack-by...>
394 comments. 645 points. Submitted 3 hours ago: <https://news.ycombinator.com/item?id=43422413>
It's reminiscent, perhaps, of the feel and motivation for Tumblr reblogs - and Tumblr continues to be vibrant by virtue of this culture: https://www.tumblr.com/engineering/189455858864/how-reblogs-... (2019)
Now, is driving attention and reputation to their site (in the broadest senses) part of a blogspammer/reblogger's motivation? Absolutely!
But should we be concerned about rewarding their act of curation, as long as there is at least some level of genuine curation intent? A world where that answer is categorically "no" would be antithetical, I think, to the concept of the participatory web.
A very useful hellscape though, for all its flaws
In other words, democracy sucks but we have not found something so much better that it would pay for the benefit of freedom for everyone...
"L402" is an interesting proposal. Paying a fraction of a penny per request. https://github.com/l402-protocol/l402
"Hey, we'd happily give these companies clean data if they just paid us instead of building these scrapers."
I think there is a psychological aspect that made micropayments never work for humans but machines may be better suited for it.
It seems like a good fit for micropayments. They never took off with people but machines may be better suited for them.
L402 can help here.
I think the paying approach is superior (after all you make money out of people using your service) but Cloudflare is a straight forward/simpler one.
*Edit: typo
The premise of this thread is that somehow the LLM builders are reading too much. I bet it's less than google.
I continue to believe, if you don't want everyone in the world to see and use your stuff, don't put it on the internet.
> This practice started with larger websites, ones that already had protection from malicious usage like denial-of-service and abuse in the form of services like Cloudflare or Fastly
FYI Cloudflare has a very usable free tier that’s easy to set up. It’s not limited to large websites.
And every once in a while, Cloudflare denies me entry to a site because it thinks I'm a robot, they are not even able to make proper heuristics to separate robot from humans so I don't know why anyone should trust them...
meanwhile: "I'm proud of how our team continued to deliver ground-breaking innovation, especially in AI" (Matthew Prince, co-founder & CEO of Cloudflare)
Don't worry, they charge plenty from big websites.
If there's profit on the table, capitalism will not allow it to sit there at any cost.
Did you see a mass exodus from Apple after that? I have seen barely any coverage this being covered outside of the French speaking world…
There's too many shady practices for the consumer to track, and you cannot even move to an alternative since everybody is doing bad stuff.
[1]: https://www.politico.com/news/2020/05/20/apple-whistleblower...
Apple privacy stance is bullshit and disgusting. I know for a fact that they collect as much information as Google (in fact in my case, looking at the takeout of data they know much more about me than Google) but the pretense is that they don't use it or sell it. Yet the average user has zero way to make sure of that and the incentive are so bad that's it just a question of when Apple stance will change (anything is possible for profits, Tim Cook has very much proved that).
If Apple was truly serious about privacy, they would refuse any type of cloud offering and completely disowned their App Store model. But the only thing Apple cares about is money, privacy is just a very easy marketing target without having to prove anything.
Microsoft is shady, Google is borderline, Facebook is careless but by far, the evilest of the bunch is Apple.
Cloudflare is the company I hate the most: I think (what I know of) their tech is done right, and they’re just too big to put my eggs in their basket.
Using Cloudflare is not a permanent fix, just a bandage, and it's particularly bad that they'll use their quasi monopoly into strongarming business to pay large fees. It's basically racketeering, legal.
If we are talking about the CDN and associated "software", last time I checked (a long time ago admittedly) it was nothing special.
Considering recent events especially, is all the worlds traffic passing through a cdn subject to a certain jurisdiction a good thing?
A user running an online casino claimed that Cloudflare abruptly terminated their service after they refused to upgrade to a $10,000/month enterprise plan. The user alleged that Cloudflare failed to communicate the reasons clearly and deleted their account without warning.
Quote: "Cloudflare wanted them to use the BYOIP features of the enterprise plan, and did not want them on Cloudflare's IPs. The solution was to aggressively sell the Enterprise plan, and in a stunning failure of corporate communication, not tell the customer what the problem was at all."
——
Tell HN: Don't Use Cloudflare: https://news.ycombinator.com/item?id=31336515
Summary: A user shared their experience of being forced to upgrade to a $3,000/month plan after using 200-300TB of bandwidth on Cloudflare's business plan. They criticized Cloudflare's lack of transparency regarding bandwidth limits and aggressive sales tactics.
Quote: "A lot of this stuff wasn't communicated when we signed up for the business plan. There was no mention of limits, nor any contracts nor fineprint."
——
Tell HN: Impassable Cloudflare challenges are ruining my browsing experience: https://news.ycombinator.com/item?id=42577076
Summary: A user expressed frustration with Cloudflare's bot protection challenges, which made it difficult for them to unsubscribe from emails or access websites. They highlighted how these challenges disproportionately affect privacy-conscious users with non-standard browser configurations.
Quote: "The 'unsubscribe' button in Indeed's job notification emails leads me to an impassable Cloudflare challenge. That's a CAN-SPAM act violation."
If you don't need them, they'll make you think you need them (so they can monitor your needs) and when you do need them, they will extort you any way they can.
The vast majority of websites don't need Cloudflare, very often people do because they run things in a very terrible way. Instead of paying Cloudflare extortion feed, pay competent people for proper infrastructure development.
Looks like the GNOME Gitlab instance implements it: https://gitlab.gnome.org/GNOME
1. headless browser 2. get cookie 3. use cookie on subsequent plain requests
Good bots: search engine crawlers that help users find relevant information. These bots have been around since the early days of the internet and generally follow established best practices like robots.txt and rate limits. AI agents like OpenAI's Operator or Anthopic's Computer Use probably also fit into that bucket as they are offering useful automation without negative side effects.
Bad bots: bots that have a negative affect website owners by causing higher costs, spam, or downtime (automated account creation, ad fraud, or DDoS). AI crawlers fit into that bucket as they disregard robots.txt and spoof user agent. They are creating a lot of headaches for developers responsible for maintaining heavily crawled sites. AI companies don't seem to care about any crawling best practices that the industry has developed over the past two decades.
So the actual question is how good bots and humans can coexist on the web while we protect websites against abusive AI crawlers. It currently feels like an arms race without a winner.
https://developers.google.com/search/docs/crawling-indexing/...
https://www.bing.com/webmasters/help/verifying-that-bingbot-...
Mastodon has AUTHORIZED_FETCH and DISALLOW_UNAUTHENTICATED_API_ACCESS which would at least stop these very naive scrapers from getting any data. Smarter scrapers could actually pretend to speak enough ActivityPub to scrape servers, though.
Sad things are getting to this point. Maybe I should add this to my site :)
(c) Copyright (my email), if used for any form of LLM processing, you must contact me and pay 1000USD per word from my site for each use.
Of course, the big AI companies are already trying to get the government to codify AI training as fair use and sidestep the litigation which doesn't seem to be going entirely their way on this matter (cf. https://arstechnica.com/google/2025/03/google-agrees-with-op...).
IANAL, but I do wonder how this ruling will be used as a point of reference whenever we finally ask the question "Does material produced by GenAI violate copyright laws?" Specifically if it cannot claim ownership, a right that we've awarded to trees and monkeys, how does it operate within ownership laws?
And don't even get me ranting about HUMAN digital rights or Personified AIs.
I am personally happy to have everyone, people and LLM alike, learn from my wisdom.
No, it doesn't. There are four factors for fair use, and whether the use is transformative is part of one of them. And you don't need to win on all four factors.
> LLM is as transformative as it gets.
The current ruling precedent for "transformative" is the Warhol decision, which effectively says that to look at whether or not something is transformative, you kind of have to start by analyzing its impact on the market (and if you're going "doesn't that import the fourth factor into the first?" the answer is "yes, I don't like it, but it's what SCOTUS said"). By that definition, LLMs are nowhere near "transformative."
Even pre-Warhol, their role as "transformative" is sketchy, because you have to remember that this is using its legal definition, not its colloquial definition.
> If I'm on the jury
Fortunately, for this kind of question, the jury isn't going to be involved in determining fair use, so it doesn't matter what you think.
The argument in Warhol relies on the fact that the derivative work, ie, Warhol's painting, is substantially similar in function to the original photograph. If Warhol had used the picture as stuffing for a soft sculpture, it would not infringe.
LLM is closer to the latter than the former.
A perfectly fine, if incorrect reply, but then you have to be a dick. Why?
I think you’re confusing copyright with a EULA. You would need users to agree to the EULA terms before viewing the material. You can’t hide contractual obligations in the footer of your website and call it copyright.
"Copyright? Well if you are a big label, we probably need to talk. Little people? Oh fuck you, just give us your money and creative output."
I would say it's not reasonably likely that LLM training is fair use. Because I've read the most recent SCOTUS decision on fair use (Warhol), and enough other decisions on fair use, to understand that the primary (and nearly only, in practice) factor is the effect on the market for the original. And AI companies seem to be going out of their way to emphasize that LLM training is only going to destroy the market for the originals, which weighs against fair use. Not to mention the existence of deals licensing content for LLM training which... basically concedes the point.
Of the various options, a ruling that LLM training is fair use I find the least likely. More likely is either that LLM training is not fair use, that LLM training is not infringing in the first place, or that the plaintiffs can't prove that the LLM infringed their work.
When talking about parody, they note that the usage as the foundation for parody is always substantially different from the original and thereby allowed, even if it would otherwise infringe. LLMs are always substantially different from the original, too.
If I want to write software that draws that picture exactly, the code would not be a copyright violation. It is text and cannot be printed in a magazine as a picture. If I used it to print a picture that was a derivative work and sold that, it might be.
A large language model has no intersection with the picture or, for that matter, anything that it absorbs. It is possible that someone might figure out how to prompt it to do exactly the same picture as Goldsmith did but fairly unlikely.
Unless you could show that this was easy, common and part of the intent of the LLM creator, I can see no possibility that it is infringing.
Randos on the internet with a betting addiction are distinctively different from a court of law. I wish people would stop talking about prediction market as if they mattered.
There's a long history of economic research on the "wisdom of crowds" that backs up their value.
in theory that legislation has teeth, too. they are not allowed to access your system if you say they are not; authentication is irrelevant.
every GET request to a system that doesn't permit access for training data is a felony
I'd certainly trust their predictions more than those given by most "experts".
The only reason copyright is so strong in the US is that there are big players (Disney, Elsevier) who benefit from it. But gig tech is much bigger, and LLMs have created a situation where big tech has a vested interest in eroding copyright law. Both sides are gearing up for a war in the court systems, and it's definitely not a given who will win. But, if you try to enter the fray as an individual or small company, you definitely aren't going to win.
You could sue, if you can afford it, meanwhile all of your data is already training their models.
Meta is stating in court that knowingly downloading pirated content is perfectly fine (ref https://news.ycombinator.com/item?id=43125840) so they for one would have absolutely no issue completely ignoring your copyright notice and stated licensing costs. Good luck affording a legal team to try force them to pay attention.
Copyright is something for them to beat us with, not the other way around, apparently.
The amount of spam that happens when you let people freely post is a much bigger problem.
Most of content, blogs could be static sites.
For mastodon, forums I think user validation is ok and a good way to go.
The thing to watch out for is platforms like Vercel or Google Cloud Run where you get charged more for compute if you attract crawlers, potentially unbounded (make sure to set up spending limits if you can.)
This is scary
Deregulation is ultimately antithetical to our personal freedom.
I just hope the spirit of the internet that I grew up with can be rescued, or reincarnated somehow...
Move fast and break things apparently has a bonus clause for the things you break not being your responsibility to fix.
On the legal end though, I do think there's a few things that should be done:
* Scrapers should be CLEARLY, and CORRECTLY identified as what they are, and who they are being dispatched from. Changing user agents to get around blocks should not be permitted, ever. If you only get a certain amount of content or a certain subset of pages when you identify as a scraper, that is a choice the website operator is making and it should be respected.
* Scrapers MUST OBEY robots.txt. We didn't create that for a fun hacker weekend. It's an important technical component of how we organize websites and how we want them crawled, if we want them crawled. It should be the first stop for any scraper on any website, and again, it should be respected.
* Scrapers should always meter their traffic with respect to the website owner. Pounding an entire website's library of content request after request with only milliseconds between is, to put it bluntly, being a fucking asshole. And not just to the owner, but to anyone else attempting to use the site at the time.
If a website operator configures their site incorrectly and pages they don't want scraped are, or pages they do want scraped aren't, then that is on them and they need to fix that. It is not in the scraper's purview to end-run around that configuration to "be real sure" they got everything they were meant to, and it's especially not that to get things the web operator has explicitly tried to not let the scraper have.
And yes, all of these things should be legally actionable, with financial penalties attached and for serial offenders, we should have a registry of scraper bots that we disallow entirely because they are acting in bad faith.
I didn't give any LLM permission to train on my data, Chinese or otherwise. It's theft and I have zero recourse to do anything about it.
I don't want people to redistribute my data without attribution, claiming it as their own.
It's a not a binary thing to me: LLMs are not god, but even without AGI, they have proven wildly useful to me. Calling them "shitty chat bots" doesn't sway me.
Further I have always assumed that everything that I post to the web is publicly accessible to everyone/everything. We lost any battle we thought we could wage some 2+ decades ago when web crawlers started hoovering up data from our sites.
You, as a user, might not care, but as servers keep going down, more and more website owners start blocking LLMs. Good riddance, hopefully all good stuff gets locked down.
Or to use an analogy, your comment is similar to: "sure those delivery vans violate speed limits and occasionally hit the pedestrians. I don't care, those fast deliveries have been proven wildly useful to me"
Which now I realize is not different from people amaking YouTube videos. I feel there is a difference but I don't know how to explain it. Maybe there isn't. Ouch, writing this comment was not a good idea...
Individuals who have to do work in order to use your content to do work to create their own content is qualitatively different than automation trivially doing whatever.
1. https://en.wikipedia.org/wiki/The_Work_of_Art_in_the_Age_of_...
To me this feels almost like the news complaining that they want a "link tax." Weren't their headlines and summaries used? It seems inconsistent to somehow say that AI and scraping is not okay; but that news companies should also not be entitled to their link tax. It's okay to index, but not that kind of index.
In the "link tax" case, there were plenty of trivial ways to opt out of headline usage - robots.txt, http headers, http tags. The problem was newspapers did not want to opt out (as they were benefiting from Google themselves), so they wanted a 3rd option. Which was pretty stupid of course - if you don't like the deal, don't take it; suing the offering party for a better deal is not a good long-term strategy.
In the AI case, there is no opt-out. All those websites already indicated they want to opt-out via robots.txt, but the AI companies ignore robots.txt, change user-agent, fake IPs, and so on - do the things that are normally done by shady malwar-ish services rather than multi-billion-dollar companies.
It really bothers me when people don't see the difference between those two cases.