The people behind the LLM behind this blog post are. They're trying to pretend their robots are people to sell other websites' data to their customer. It's easier to pass bot detection gates if you pretend to be a physical machine running Windows or macOS than if you honestly admit you're using Linux on a VM.
I'd be fine if they throttled the connection, or a contact page. Every little bit helps build a high trust society.
In the bad old days there were so many differences between html, css and js behaviors that if you wanted your site to be nice you had to change it for the browser. The way css padding worked wasn't even the same. Feature detection was rarely viable for any of this.
No user agent would probably have only entrenched IE6 dominance even more by blocking you from deliberately making a site that works at all on other browsers (including IE7 for that matter)
JS devs were kinda able to patch around the nonsense because they were able to feature-detect - part of the reason this stuck around was because no legitimate user or dev cared (or should care). But the header was mostly (useless) noise, and the people spoofing were dealing with the couple bad apples of the time.
Of course, defining features is easier said than done, and a standards body is a challenging environment to define these in...
I get why people are fingerprinting bots and others are working around it, but neither are "legitimate" applications - if your content is public, it's public, end of story. And working around these controls to sell botnet access to sites is equally illegitimate - nobody has a right to resell content they do not own...
I wish this was the case. Unfortunately, companies that work on non-Chromium browsers need to employ dedicated web compatibility teams to either a) help website users fix non-standard (i.e. Chrome only) HTML/CSS/JS, or b) replicate Chromium-like behaviour for specific (very popular) websites so that they work "correctly".
There's also the websites that deliberately block certain browsers which is what tools like "chrome-mask"[1] are built to solve.
[1] https://addons.mozilla.org/en-GB/firefox/addon/chrome-mask/
In practice when you make your new set of hacks the string you can always evaluate whatever cruft in the useragent today, but next browser shows up.
It does make the user agents insane but I don't know if there's any obviously better system for the problem, even with hindsight
Not really, there's just only one browser (Chrome). Firefox has declined so low it's a rounding error, and all other browsers are Chrome forks.
There's been some pushing for browsers to automatically decide what language you want to view a webpage in, and that is even more insane than the User-Agent header is. The correct solution for content that exists in multiple languages is to make each language available at its own URL. Just like I already said. Why in the world would you want to lose the option to select the language you want?
User-Agent spoofing was added to NCSA Mosaic in 1996. The public www was three years old and text-only clients were still in widespread use
https://raw.githubusercontent.com/alandipert/ncsa-mosaic/mas...
NCSA Mosaic is the early graphical browser that begat Netscape Navigator that begat Firefox. Later came Internet Explorer, Safari, Chrome and so on
What was the point of spoofing in 1996
Maybe it was just for fun, judging by the examples in the "mosaic-spoof-agents" file
As a matter of practice, by default I do not send a user-agent header. I only send the minumum headers required. For me, that's almost always 1-2 for GET and 4 for POST
For the vast majority of websites I have accessed,^1 this header minimisation has zero effect on the success of the HTTP request
1. For example, I have used a database of sites submitted to HN as way to test if header minimisation affects HTTP request success
Generally I do not use a web browser to make HTTP requests. I perform text processing on the HTML, JSON or whatever is returned, using custom utilities. I store information in SQLite. I read this information as plain text, preferably 7-bit ASCII. I dislike UTF-8
Onlilne debates about "browser fingerprinting" always seem to focus on trying to "blend in", e.g., via "spoofing"
As such, because browser continue to get more bloated with "features", online commenters argue in favor of sending more and more data points to servers that can be used to create a fingerprint instead of reducing the amount of data sent
Because, according to their reasoning (or lack thereof), sending less data would "stand out"
True, but it's generally easier to "spoof" a client that sends less data than one that sends more. And the number of sites that require a user-agent header is still smaller than the number that don't
They're pretty brutal in blocking non-browsers, I tend to have to add a lot of useless headers.
Blocking "non-browsers" and blocking IP addresses are two different things, so to speak
The first is based on dumb heuristics and (incorrect) assumptions about behavior based on what software someone is (mistakenly) presumed to be using
The second is based on past behaviour
I send the minimum HTTP headers
I only request what I want; I do not send requests for ads, tracking or telemetry
I accept text formats, e.g., HTML, JSON, etc.
No need for images, CSS, Javascript, etc.
IMO, there is a fundamental difference between (a) not using the data collection, surveillance and advertising-friendly software, i.e., popular graphical web browsers controlled by so-called "tech" companies that, surprise, said so-called "tech" companies and their business partners want people to use and (b) engaging in bad network behaviour, what some used to call poor "netiquette"
The heuristics used to allegedly identify bad behaviour, for example, sending the "wrong" value in a User-Agent header, are beyond stupid
But the false positive totals, "collateral damage", are apparently not large enough to matter
For example, something like "90% of the current internet is behind Cloudflare, Google, Akamai, etc."
The www is only subset of the internet and "90% of the internet" does not use a CDN
In any event, even if "internet" means "www" there is no evidence to support this statement
https://www.netcraft.com/blog/may-2025-web-server-survey
Further, being "behind Cloudflare, Google, Akamai, etc." does not necessarily mean a User-Agent header is required
For example,
https://www.cloudflare.com does not require a UA header
https://example.com is "behind CF" and does not require a UA header
https://www.google.com does not require a UA header
https://developers.akamai.com does not require a UA header
And so on
Individual www users, unless doing mass scanning or the like, probably do not access 100% of the www's sites (according to Netcraft, most are inactive)
More likely, individuals access only a fraction of the active sites
It's unlikely that individuals access exactly the same list of websites^1
For me personally, the vast majority of sites that I access do not require a UA header
The sample set I use as evidence to support comments I make about this on HN is a list of all sites submitted to HN over a recent period.^2 Currently this is 12,641 sites
1. To have a meaningful discussion about UA header requirements, a sample set of sites must be agreed upon
2. Perhaps HN readers access these sites and possibly access some of the same sites. Of course it's also possible that there are HN commenters who do not actually access HN submissions and instead only read comment threads and submit replies
https://news.ycombinator.com does not require a UA header
For example, https://www.apple.com does not require a UA header
For example, http://www.slackware.com does not require a UA header. However, it does require Accept and Accept-Encoding headers
This site can also be accessed over HTTPS
And detecting their platform doesn’t stop you from presenting supplemental links for all platforms. In fact I’d say that’s completely normal.
A regular Joe consumer shouldn’t have to learn python to get thier foot in the door because some arbitrage purchaser has automated a ‘best deal’ purchasing bot based on real time pricing APIs.
I mean, I get it, saying you support something is technically a commitment. But my goal is to basically never open a chromium browser, so, I'm gonna do what I have to do. And you would be shocked, in the past 5 years I have only encountered ONE website which legitimately required a chromium browser. grapheneos.com, for the web usb installer.
The moment you openly publish information on the Internet, you have already given consent. There are other solutions to bandwidth usage.
User-agent discrimination should be illegal. All it does is further the control that Big Tech has, and help authoritarian governments with their control too.
Couldn't agree more. Nonsense like remote attestation too. If this discrimination persists, it will lead to the destruction of what little computing freedom we still have.
There are additional methods I chose not to document such as limiting access to logged in accounts that require double-opting-in to acceptable use policies and terms of use, not that most scrapers would give a toss. That it too much whack-a-mole for me personally. That method requires progressively adding friction to account creation and that comes with some pros and cons.
[1] - https://nochan.net/b/Internet-Crap/20260606-How-To-Block-Som...
"If you didn't want me to do this, you should had a fence/cameras/security guards. You shouldn't have dressed like that. You shouldn't have put your phone in that pocket."
Excusing trillion dollar corporations like low level criminals is embarrassing. Society shouldn't have to lock itself up because bad actors are spreading everywhere. The bad actors should just be removed.
If these scraping companies would respect some kind of X-Scraping-Permitted header, this whole process would be a lot better, but that's not going to happen in an industry that has already normalized using botnets.
It's like public photography, it's intrinsically legal, except when it's a Flock camera and then it's suddenly an invasion of privacy.
Fair. Scrapers should be polite and do their utmost to consume the smallest possible amount of resources.
> without my consent by masking your user-agent
Your consent is not required. It's my user agent. I set it to whatever I want.
> for the purpose of stealing data I didn't authorize you to have
Data can't be "stolen", only copied.
You set up an HTTP server that literally sends people the data when they request it. Don't do that if you don't want people to have the data. Secrecy is the only possible defense.
By the same argument I could say: If I send you an exploit and you execute it, don’t complain that your setup fell for it. Just don’t download and run random data from the internet.
In reality there’s a consent and expectation beyond the pure technicals.
I don't. I go out of my way to filter everything. Scraping is but one of the tools I use to do it. I want just the data that I actually care about, not people's javascripted hot mess websites full of malware-vectoring ads, fingerprinting and tracking.
I don't let my computers talk to strangers either. My servers don't respond to just anyone, they only reply to me, and only after I've cryptographically authenticated. When others try to talk to them it's like they're not even there.
But people want their computers to talk to strangers, don't they? They want to serve pages and pages of ads to massive audiences. Unlike your exploitation example, nobody's actively invading their computers and exfiltrating data. Breaking into someone else's computers and dumping their private databases is one thing. We're just requesting the exact same data that they're more than happy to send out to literally anyone who shows up with a browser, through the exact same channels even. So I really have no sympathy.
Here's some stuff on Adversarial Interoperability, which is an incredibly good thing that can reverse enshittification if it becomes more widespread: https://www.eff.org/deeplinks/2019/10/adversarial-interopera...
Though, scrapers can certainly steal capacity through conversion for their own use. When they do so, they permanently deprive the site owner and other users the beneficial use of that capacity at that time.
I'm speaking morally, not legally. Though, in the US at least, there are parallels with free newspapers. You're allowed to take one for free. It's not legal to clear the whole rack.