Cloudflare Turnstile requiring fingerprintable WebGL
hacktivis.me
hacktivis.me
I don't want to defend them, because they gate away a good chunk of the internet with their "bot protection", but unless you do PoW (which is also ecologically a nightmare), probably fingerprinting is the way to go - completely destroying the privacy of everyone involved.
Cromite, a privacy conscious fork of Chromium for Android, has constantly issues with CloudFlare Turnstile [2] because they (Cloudflare) try to fingerprint it in multiple ways in order to pass the challenge. The only way to get it to work would be to join the CloudFlare Browser Developer program - which requires signing an NDA. Rightfully so, the project maintainer didn't want to do it.
If you want to see the extent of what CloudFlare does to fingerprint the browsers, just have a look in the issue [2] and see which flags need to be disabled in order to allow CloudFlare to pass the challenge.
I understand both sides, but at least CloudFlare could be flexible enough to fall back to PoW instead of just blocking people from sending forms or accessing websites...
(That said, I still keep separate machines. One for doing "official" things, the other for everything else)
does it? same binary, same machine, same display, same 781 other heuristics.
I think this was as recent as 25 years ago?
Recently they added some new UI. There was and still is (I think) classic Profile Manager UI, which you can launch with
./firefox -ProfileManager
But you don't have to use any of those anyway - see my comment above (a response to parent).
Firefox with a non-default profile can be created like that:
./firefox -CreateProfile "profile-name /home/user/.mozilla/firefox/profile-dir/"
# For, say, cloudflare that would be:
./firefox -CreateProfile "cloudflare /home/user/.mozilla/firefox/cloudflare/"
./firefox -profile "/home/user/.mozilla/firefox/profile-dir/"
# For cloudflare that would be:
./firefox -profile "/home/user/.mozilla/firefox/cloudflare/"
- create a copy of it, say, /usr/bin/firefox-cloudflare
- adjust the relevant line, adding the -profile argument
Of course, "./firefox" from examples above should be replaced with the actual path to executable. For default installation of Firefox the path would be in /usr/bin/firefox script.
So, you can have a separate profiles for something sensitive/invasive (linkedin, cloudflare, shops, banks, etc.) and then you can have a separate profile for everything else.
And each profile can have its own set of extensions.
I'd argue, that for some, CLI path is actually cleaner.
You see, the way described above creates entirely separate points of entry, and you don't have to go to the central menu to launch specific profile.
It eliminates one step (Profile Manager, about:profiles or whatever) allowing you to get faster to the desired profile - same way you'd launch a default profile.
It's logical separation too. It's like separate browsers from UX standpoint (they do use the same distribution though ...unless they aren't - you can configure different distributions for different profiles - nothing stops you from that).
I'm just leaving the information about the gui option to other who may not be aware that it can be done from the gui too, and think its difficult to do in Firefox.
Edit: I RTFA'd, containers can't adjust `privacy.resistfingerprinting`. Boo
- Independent set of settings/about:config parameters.
You can't turn off, say, WebRTC completely for some profiles, while allowing it for other profiles.
Different history. I remember accidentally nuking history of a few years - that wasn't fun. Now, you reduce blast radius.
Proxy on/off or different proxies. Though, there's probably an extension that manages it on per-site basis.
Different userChrome.css, if you fancy that.
If you are on linux I can't recommend enough using a COW filesystem like btrfs and zfs with snapshots. I can't count the amount of times i have wiped or edited something by mistake and then restored it within seconds with it.
More to the point, these systems actually help scraping because proof of work unlocks essentially unlimited scraping, in my experience.
That said - from my experience on the other side, sure you can’t stop people like me or you, but you can stop 99% of the others. That’s more than worth it operationally.
It sure seems to keep me, the casual visitor, far away from just about any site they "protect". I have zero desire to alter my browsing configuration or use extra tools to get around turnstile, I'd rather not even visit the site in the first place.
Until your bank, airline, and tax ministry start using them.
Can you expand? I don't see a problem with some napkin math. 5W load for 2 seconds is 0.002Wh (we have to let smartphones pass and not by doing PoW for 10s of seconds). 8 billion checks a day for a year = 8GWh.
In any case, according to some napkin math done by Kimi 2.6 (which by itself is probably already consuming more than all of my PoW challenges for the upcoming 5 years) - the situation looks incredibly in favor of PoW: https://www.kimi.com/share/19e7ef40-a432-8912-8000-0000b4a71...
Which makes me wonder why CloudFlare isn't switching to this already
Just because you've never been in a situation to care about the benefit they are offering, does not mean it's not valuable.
And the position "you need a good excuse to have overly specific software" is extremely strange.
If some computation is "useless" but it serves it's purpose, it's not useless.
The reason why bitcoin network expends so much energy is down to tokenomics, not the system of PoW itself. At equilibrium we expect the power usage to be (blocks/hr) x (BTC/block) x ($/BTC) x (kWh/$), so it's a function of the BTC price and emission rate.
PoW in other context has way different driving factors. In this case, the marginal improvement of fetching the site again for AI bots isn't enough to cover the PoW cost. The PoW cost is outweighed by the net bandwidth cost of all the parties.
I think we have to expand the definition of stupid to include things that work but have net negative externalities. Not sure where PoW falls in that way of looking at things, but we should at least consider it.
(Thinking about it, Captcha is PoW, just theoretically work by the human)
I don't think I average even 2 captchas a day being terminally online, so 10 across every soul in the world sounds way too much for me. (we're ignoring bots it's meant to deter?)
> it's hard to see that such a modest energy cost would dissuade any attacks.
It's not against targeted attacks, but scrapping.
And not about energy cost, but available compute power -- it requires scrapper to use browser with JS (or time commitment to reimplement PoW outside of JS), limits their request rate by CPU core count.
You're mixing up checks, fingerprinting, and PoW with a captcha being triggered because those didn't pass. The less abnormal your setup is, the fewer captchas you'll get.
I agree with the rest of what you said.
Also I think you mean "scraper" and not "scrapper".
They also gate away a good many people with their "bot protection". I am extremely worried about how so many seem to have outsourced the control over who can access their websites to a company, with no second thoughts whatsoever.
I think the Web is on its last legs, anyway. Generative AI and LLM-instead-of-search has destroyed what little value remained.
I'm glad to have known the true internet before its demise. Truly one of the wonders of humanity.
How many people do you think are browsing with a weird enough config (eg. custom browser like OP, or some weird config like firefox with fingerprinting protection on a raspeberry pi) to trip cloudflare's protection?
We had all of our Devs Pixels get blocked, and after talking to CF, it was because Internet archive was rebooted their scraping farm, all the devices stampeded and overwhelmed the known bot safeguards, and those tags were added across the board. CF gives sites the tools to tune what is getting blocked, we bumped the sensitivity down to 25 and haven't had many complaints (despite having a very vocal community)
The most common complaint is users' IP address getting blocked because of compromised devices
In my experience what really makes it loop every single time though is JShelter. CF doesn't like having your fingerprintable data bits messed with.
There are legitimate uses for non-instrusive, ethical and legal scraping, but some of us have had to resort to extreme measures:
[1] - https://blawg.nochan.net/b/Internet-Crap/20260522-Maybe-AI-B...
And no, it wasn't due to a country-based block selected by site operator.
At least let me complete a "prove you are human" challenge or something, but don't outright ban my IP address?
It takes very little for CF to consider you "weird".
I know some actual users get blocked. But the amount of spam we get without it, the amount of bot traffic simply overwhelming the server... It is just too much.
Recently I also hard blocked all IPs from china Singapore India Pakistan Russia and whole of africa. Do I want to do it? No. But the amount of bot traffic and corresponding spam is a bigger problem :(
> I know some actual users get blocked. But the amount of spam we get without it, the amount of bot traffic simply overwhelming the server... It is just too much.
One of the core tenets of system design is Availability. If your service is not available - if your forms are blocking legitimate users - then why are you pretending to have a form submission feature at all? Just to frustrate users?
The service won't be available to anybody because of overwhelming unwanted traffic. Now it's available for most potential users. You're speaking econ 101 when everyone else has played out iterated prisoner's dilemmas.
Turns out that people have a tolerance for a non-zero amount of work, but still have a limit.
Suggesting "turn off your website" is does not account for the desire to also provide some access.
Treat people who host content as humans, just as we must treat users as humans. There are tradeoffs, suggesting "shut down your website unless you provide access everywhere" is worse on all fronts for everyone.
Maybe, maybe not.
If block-heavy websites shut down entirely, we lose some content, but other content moves to block-minimal sites and the average user might be able to access more.
Also if there's no blocking crutch, and people get pushed into shutdown and are mad about it, they might fight harder for anti-spam technology and legal enforcement, which could improve the situation.
Because those are the only two countries that we've ever in the life of our business, had a legitimate order from.
It prevents the majority of credit card testing, but it is tempting to apply it to the whole site to reduce traffic and server load.
If you are getting blocked by CloudFlare, you are most likely not our demographic.
And there's always email address given in form submission, so a couple of users (like less than 5), emailed about the block and I added rules for each of them.
Better than taking down the whole thing because of bots scraping the site 5x more rate than humans.
At least for China, I imagine most of the real humans might use a VPN anyway
Yesterday cloudflare blocked me from visiting the MX-Linux site ... including an old browser with -no- protections ...
I have to wonder - assuming these sites are paying CF for this 'service' - are they getting a list of all the fejected IPs?
No, we don't know. I honestly do not understand the problem. I run websites, both static and non-static. Granted, my sites aren't exactly the most popular internet go-to destinations, but I should be seeing this DDoS too, right?
I do see lots of requests. Nothing that any modern system can't handle. Computers are stupid fast these days. Unless you are doing something unreasonable, it's really hard to even notice this "extra load".
I understand there are sites for whom this causes problems, but I think these are rare and could be optimized not to do unreasonable things.
I think too many people are annoyed by AI companies (arguably understandable position), look at their logs and speak of "hammering", "DDoS" and "extra load", while in reality it doesn't matter much.
What matters most is usually how much there is to scrape. If you have like 5 pages that's nothing. For forum like websites where each thread, each user profile, etc. gets scraped that's when traffic increases. I just let them have at it with no issues though, computers are fast.
It's not a hostile DOS in the traditional sense (I've mitigated a few of those) - no "pay us to make it stop", no pattern to the requests other than "fetch every unique URL a few times".
It wasn't happening until financial incentives to gather large datasets for AI training appeared.
Bad actors (using residential proxies & claiming to be a real browser) mostly showed up after folk started blocking ones that identified themselves as AI scrapers.
It's obvious to blame AI training because there's a shortage of better explanations. Who else would be paying for these (expensive) residential botnets, only to use them to (eg) web-scrape wikipedia (which offers free downloads of its content in a structured format)?
The simplest explanation of the technical behavior is "a bot coded to follow every link it sees & save the results", and the simplest explanation of the motive to run such a bot is "to train a large language model".
"use Cloudflare to make it stop"
Cloudflare are merely the cheapest of the bunch.
The usage is extremely quick, and follows easy-to-spot patterns. We noticed a spike in bounce rate.
They never come from Google, and the bad programmed ones just crawl several pages at a time, faster than a user could do.
Then there's the crazy spikes in visits from specific countries, pretty much scraping the entire content. Often from pools of IPs. In some cases had 30% unexplained (meaning: it wasn't viral or a marketing campaign) random sustained increases in traffic.
There's also the fact they don't interact with the complicated widgets, so zero XHR requests other than analytics pings.
They also don't cause spikes in Google Analytics, so I assume it's blocked, but they show up in logs and in the internal analytics.
It's not enough to DDOS the website at all, but it's a lot of noise in statistics that we gotta learn to filter.
I’ve triggered this kind of “bot protection” right here on Hacker News many times. I did that by having a bunch of Hacker News pages open and then closing and reopening my browser. I’ve also triggered it by opening a bunch of links in the background too quickly. I’ve also triggered it by reading the article, then clicking back and upvoting/favouriting too quickly. I’m also located in Singapore, which people have started to advocate for blocking here recently.
A single non-bot legitimate user can easily trigger these kinds of heuristics just by using the site in a way you don’t expect. This can affect some users disproportionately more than others, e.g. disabled people who need to use assistive technology.
What I mean by "too fast" is opening 50 pages in the span of two or three milliseconds.
Either way, I'm not blocking. The CDN is handling the traffic alright.
A) you'd have to open >200 tabs, and B) if any tab solves the proof-of-work, any that are still waiting to do so reload in the background.
It's also not always easy to do. I run a small wiki which is fairly optimised, nearly every page manages at least ~3k rps on a small VPS. The only exception is the diff page which is ~150 rps. Optimising that while still giving good output isn't that easy, but the wiki doesn't have many users so that would be fine if it wasn't for the AI bots.
The AI bots ignore robots.txt and were initially hitting the site with ~1k rps crawling every combination. Even that would be manageable as there's currently ~150,000 combinations, except they kept re-crawling the whole lot each day. The server could manage it but it was a massive waste of resources.
They were using residential IPs and only sending 1 request from each IP making it impossible to block. In the end I gave up and put a Cloudflare challenge in front of it. I don't want to use Cloudflare but the alternative is forcing users to login to view diffs or remove them entirely.
I did try removing some of the links without success. I guess once they have them they just keep checking.
Similar to the one SQLite had: https://www2.sqlite.org/forum/forumpost/7d3eb059f81ff694?t=h
Each IP only makes ~1 request though so easy to detect after the fact.
I guess they will run out of IPs at some point so maybe if I had logged each one forever and shown a challenge only to them, it would have fixed it eventually. Just depends how big their pool of IPs is.
In any case, such labyrinths of expensive dynamically generated pages are no excuse for subjecting people requesting the start page to bot checks.
that's just ~17 req/sec
That's "cheap VPS running wordpress" level of traffic
It's phenomenal how inefficient the WordPress/WooCommerce stack is.
Though the main issue I'm seeing is credit card testing, not scraping.
And I'm ideologically opposed to using a CDN (because it shouldn't be needed for such a small site!) so it's somewhat a self-inflicted problem...
It's easier and better to just block 0.0.0.0/1 half of the time, and 128.0.0.0/1 for the other half of the time. Switch every day at noon.
Bot traffic will be cut by 50%, and humans are all treated equally! It's a total win!
Just block it all.
I adopted Cloudflare because it was getting DDoSed by the AI crawlers. I'm pretty sure all of them are vibe coding their crawlers and don't bother adding rate limiting as a requirement.
I've spent some effort on optimizing my sites, but most of the effort was focused on avoiding unreasonable (stupid) work. Do I need a session for every request? No, I don't! Do I need a database fetch for every access to my homepage? No, I don't! Is it a problem to actually load all of my static content in all supported languages (24) into memory and serve it from memory? No, it isn't!
I use Clojure behind nginx on the server for my sites. Oh, and I also pre-compress all static assets to Brotli, so anything that handles brotli gets a static file served directly from nginx. I also use immutable assets with unlimited caching semantics.
Really — the problem is that we've grown lax and our software has become bloated, slow, and with unreasonable code paths. If every page fetch does 12 database accesses and runs through a slow interpreter, that is surely going to be a problem.
If it was more static content it'd be easier, it's really the db being a bottle neck in a dynamic site.
Yes, the software could be better optimized but then I'd have to own the development of it. There is no reason a niche website should be getting millions of requests per day.
Not saying I'm not annoyed by the scraping; I am looking to block them, but I'm also not going to put the site behind the gatekeeper. If anything, Cloudflare must love AI scraping now for the same reason AV companies love malware.
Now, if you are running a PHP stack...yeah, maybe that's the problem right there.
I wonder if we should stop fighting this and instead create an API specifically for this purpose? Or, a central repository that you could send your data to and say to anyone wanting to scrape, "safe yourself some time and just get my data from this other place"
I’m with OP: I don’t like this but the alternatives all look like the death of the open web.
The person you're responding to already said they ran a modestly sized site. What actual scale opens one up to abuse? If only the top 1% of sites need it, then it seems silly to say "everyone" needs it.
And if you're thinking about blocking all of China, while this particular AI bot didn't use them, a bunch of other ones I've encountered use VPNs and hacked clients worldwide.
There are. They're not. They can't (without significant effort)
People shouldn't have to be experts or provision a larger server to run a UGC service that can withstand the sort of 30x more traffic I'm seeing from AI bots. Or rather, you didn't render the argument for why they should have to do that if they can just use CloudFlare's free tier.
Either way, it's easy to have all the answers when you've never had the problem.
I haven't had any problems with Firefox so far. Why do you say this?
Although the cynical part of me says that this will result in malicious actors trying to trick agents into giving out a bunch of micro payments. There are counter defenses that can help detect and compensate for that, but perhaps the best we will be able to do is prompt user with the default agent recommendation.
Never needed it. Just put the worst offenders in penalty bucket and that's usually enough
The bad guys don't pay that much. And sometimes the bad guys actually use the IPs of other people (botnets on residential IPs) and don't pay anything at all.
What would stop you offering someone a few tens of cents per GB to borrow any other token barrier you put up?
You can easily calculate which IPs/networks bots are using by looking at where most traffic comes from and who requests lot of pages with non-human speed.
By the way, what's your opinion about running a cryptominer on requests from datacenter and bot IPs?
So delegalize it. Strip searching everyone to paper over the fact that the societal contract has been broken only delays that.
As someone responsible for mitigating card testing "attacks", account harvesting, and DDOS attacks..
It is unfortunate, but the ISP industries(from telco up to transit) and CC industries aren't providing a lot of great options. This idea that people are doing things "without a second thought" is usually false when it comes to businesses.
1. If X% of the population gets wrongly branded with the scarlet letter B[ot], how do they appeal and get it fixed?
2. How will sites notice and know if their choice of "bot protection" is losing them X% of users/customers/job-seekers etc.? If it's a really robust system, they'll never even see the complaints either...
3. If everyone does detect that something is awry, will it be such a monopoly that there's no choice but to let it happen?
Bot protection with fingerprinting is just an illusion. Any signals like this which is on client side can be spoofed by an above average person. Fingerprinting is just way to consolidate the market for advertising business. Assigning Reputation to residential IP addresses and commercial blocks is is another approach to achieve the desired result. Providers would be a lot more careful to allow their IP addresses for misuses, however turns out that it would bring down the DDOS business on both sides, attackers and protectors.
Ironically, more than often its the same companies that invest in building their own bots and finding ways to stop bots from other companies.
At the upper bound, fraud can always be committed by paying real people with real accounts to perform the desired action in a way that is 100% truly indistinguishable from organic. There's fundamentally actual prevention technique at the limit.
So the entire game is only "increasing the costs until it's not viable ROI", not "holistically prevent", which is why fingerprinting is a relevant technique here.
As per cloudlare's own report, about 78% of the DDOS attacks are at the network layer where the fingerprinting technique is not useful.
DDOS is done against targets for certain reasons, most businesses are not even viable targets for everyone.
However letting everyone being fingerprinted on the pretext of solving the DDOS is where the privacy gets compromised (not much of it is left though). Some search engines did it indirectly by letting people use tag managers for free in their website and then utilize the data for their advertising business.
Relatively the end game is same, its just how these companies are approaching it.
Bots manipulate review scores, posting link spam to other users, crawl your database that isn't open to crawl, etc.
Tools are inherently amoral; only people can have motives we can celebrate or condemn.
Large companies and banks see >95% fraud on sign in / sign up flows. It’s a constant battle and the law of large numbers says even a tiny false negative rate can be catastrophic.
A bogus GCP or AWS or Azure account costs those companies hundreds to thousands of dollars. I don’t know what the average loss is on fraudulent bank signins, but probably on that order. And there are millions, sometimes billions of attempts per day.
I worked at a tech company that used an off-brand, truly awful captcha provider. Think “drag the mammal to the habitat it lives in, avoiding the wiggly lines”. When this awful provider went down (frequently), we fell back to recaptcha. Fraud rates were 100x higher in those minutes-to-hours outages. Though of course real users were also able to get in at higher rates.
turnstile is not a public good, it's a private product, promoted to private entities that want to achieve a certain outcome that is beneficial to them privately.
The mass surveillance is a side-effect - an externality that cloudflare does not have to pay for (but we as netizens pay collectively).
It is the role and responsibility of gov't to regulate away externality (or make those who benefit from it pay a cost somehow, to equalize said externality). Unfortunately, like with climate change, nothing has been forthcoming, and only a few people care about the actual damage enough to even talk about it.
So it will go on, and the masses do not have a say.
It's pretty clear that this is being done to solve an actual problem that they and their customers have. I'd prefer if it wasn't necessary, but I'll take this over solving challenges any day.
so as a toolmaker (presumably) you still have to answer for what you do.
your doctor seeing you naked does not destroy your privacy, it's your doctor sharing the photos with everybody that does. i.e. it problem here is that intermediaries like cloudflare don't work for you, they work for somebody else or sell the data themselves.
The WebGL fingerprinting thing is cute, too. I guess it'll buy them some time since off-the-shelf solutions are going to probably not handle this well yet. That said, as long as the reward for bypassing turnstile and other anti-bot protections remains high, these things really can't do much. A decently resourced adversary can probably come up with a dozen different approaches to make this less useful. Without really looking into it much, my kneejerk is you could probably tweak Mesa to have deterministically random behavior for whatever edge cases it looks for, but you could also just have lots of different GPU/driver combos to proxy to. The web gets less open, but in an asymmetrical way. If you really have an incentive to keep botting, you'll surely find a way.
The next step is to fully give up and just essentially implement WEI. And then the bot problem disappears?
Nope. Botting will still hold tremendous value, so likely there will be many crafty workarounds and bypasses over time. And there will be countermeasures for those and workarounds for that. Guess we'll start to find out who actually has the resources and incentives to keep botting in this environment.
So what's the real solution? Well the most obvious thing to do would be to make botting less valuable. Can we? I dunno. It may have been a mistake to move so many important things to the Internet after all. I mean, some of this is just threat actors catching up with what's possible and was inevitable to begin with. But, some of it is just trying to find solutions to problems that were unnecessary to begin with. Or failing to implement solutions despite an obvious need to do so.
There are a lot of threads to pull on, here. Account takeover still holds tremendous value to threat actors. Why? In my opinion, it's because passkeys were a tremendous failure, no matter what adoption shows. If we wanted to just improve security for users, I think we didn't need to restructure the internet around another authentication mechanism that of course, provides attestation capabilities, we could've just improved on passwords. For more secure handling of passwords, PAKEs exist. Password managers exist. For anti-phishing, TOTPs exist. What if you could have the exact same passkey experience, but in such a way that everything can gracefully fallback to just passwords and TOTP, because they're the real keymatter at the end of it? Add a web standard that lets browsers and browser extensions hook into the login process, standardize PAKEs as part of the web. Cross-vendor syncronization? A problem easily solved if we ever wanted to.
Instead of that, we got the dumbest possible world. Passkeys are sometimes available, but often not. Can you sync your passkeys across devices? Probably, maybe they have blacklisted KeepassXC by now so maybe I can't :)
But a lot of stuff doesn't even offer me the option to use passkeys, so they still use passwords. Can I enter my password to log in still? No, of course not. See, I will helpfully get the option to enter my password, in addition to the option to use email or SMS, the most secure authentication scheme known to Man, but if I actually select password and enter my secure password from my secure password manager, what I get to find out is that the password option is actually password and email or SMS and there's no option to use TOTP. Oh, and you randomly get logged out for no reason sometimes.
Some of the bots will probably disappear. Like, whatever bot is throwing me several terabytes of nonsense traffic every month will probably eventually disappear since they're wasting so much bandwidth on doing literally nothing. I have no idea what the point is, but I know it can't be terribly valuable for them, and it's not terribly expensive for me. I'd love to know who the hell is doing that and why, though.
But since the web is ran mostly by crap companies like Google, it will never get its shit together, and we will get solutions like WEI and identitity verification to solve problems that were entirely manufactured (or caused by a significant lack therefore of) in the first place.
By virtue of incompetent and ignorant Devs and middle managers. Our by virtue of greed and maliciousness.
Yeah yeah never attribute to malice what can be explained by stupidity... This time no. It's both.
I hate what the anti scrapper mechanisms have become but it really is the lesser evil. The alternative for many small operators is to just completely shutdown.
Well I mean maybe it wasn't useless 2 years ago, but in the age of AI it definitely is.
Those might ignore it, but there are always alternatives.
Cloudflare only exists in its current form because banks and such already enthusiastically accepted that trade off.
Most businesses don't have the luxury to be able to not care for the customer.
Cloudflare is a service provider for third parties, not the product I want to consume.
Only as long as legislation and law enforcement is off the table. Almost like we have those because everyone doing their own policing is not a reasonable way to run a society.
It's either proof-of-humanity (increasingly hard to get in this day and age, particularly if accessibility is a concern), proof odf identity (even worse) or proof of system integrity, which is the least bad out of all the terrible options.
Anything that amplifies the cost and effort required by the adversary by several orders of magnitude is worthwhile discussing.
For good reason. I've run that setting for ages but I kept having to disable it and add workarounds because websites would break in weird ways. Timezones in scheduling websites being messed up nearly made me miss a couple of appointments. There's no way to tell the user Firefox isn't broken without displaying a permanent banner like "if websites are broken in any way or you see weird glitches or your computer's time is wrong or fonts look weird or videos don't always work right, click here to disable fingerprinting protection".
Interestingly, Turnstile breaks with resistfingerprinting but works with fingerprintingProtection, I guess the latter takes this crap into account.
I somewhat expect breaking sites with strict settings, I don’t expect an still wide open tracking path.
That’s deceiving.
Websites already break often with the strictest protections enabled, adding a "super duper strict protections" mode will just lead to bug reports. Even more-than-bare-basic tracking prevention has HN threads full of comments like "doesn't work on <Firefox fork>" because they don't see the connection between fingerprinting protection, WebRTC/WebGL/WebGPU, and websites not working.
People who are willing to take that bet can enable it in about:config.
That’s what I‘m saying. They already break because of other effects of the strict settings, so what is the benefit of leaving resistFingerprinting turn off?
> There is no full immunity against fingerprinting.
There is 0 immunity if you don’t even try.
Strict means, do what you can, not do somethings strict other not so strict and others ignore completely.
Don’t call it strict if it isn’t strict
> Stronger protections that block more trackers, but may cause some sites to break.
That seems very reasonable to me. Anyone who wants more than that can turn on resistFingerprinting and live with the consequences.
People already expect sites to break, so why holding back?
There are three levels of protection:
- ETP Standard (see [0] for the latest improvements we rolled out)
- ETP Strict (we're working on things in Bugs 2036879 specifically this issue, 2037260, and more generally 2036786)
- Resist Fingerprinting (RFP)
These levels are something akin to "Wash your hands after using the subway", "Wear a mask on the subway", and "Wear a level B hazmat suit on the subway".
"people already expect sites to break, so why holding back?" - because the breakage is so severe, and people _don't_ associate that breakage with the setting they made. There are bug reports all over the internet proving it, here are some examples [01-4]. The protections we deploy in ETP Standard and Strict are calibrated to provide as much protections as possible while keeping the internet usable, and we're working fulltime on improving them.
[0] https://www.firefox.com/en-US/firefox/151.0/releasenotes/#no... [1] https://old.reddit.com/r/firefox/comments/fy6l1z/youtube_bor... [2] https://necromuralist.github.io/posts/mozilla-madness-resist... [3] https://support.mozilla.org/en-US/questions/1212634 [4] https://support.mozilla.org/en-US/questions/1322787
second, would it be possible to make RFP appear as an extension like uBO, where it suggests sites to allow-list, or hints that the page might be broken and asks if you want to disable RFP?
I'm more tech savvy than the average user, admittedly, but I've learned this pattern for uBO.
I can point you at a few things you could do if you wanted to pursue this:
1) We have a dev extension https://github.com/mozilla-extensions/fingerprinting-protect... that lets you enable/disable individual fingerprinting protections. We use to debug why something is broken.
2) granularOverrides allows you to enable/disable individual protections for a given website.
If you wanted this, you could go read https://docs.google.com/document/d/1FywogzvkWupoUoz4PcCp9nNd... ; then made an extension that made it easy to edit granular overrides (you couldn't directly set the preference, but you could produce the json you could copy/paste into the pref). You could do stuff with lists if you want. (Somewhere there was a FF fork that had a pretty impressive granularoverride list itself...) You'll be in this awkward spot where you don't have all the permissions to do what you want to do directly, but you can get yourself about.... 40 - 60% of the way there?
I would like to find a way to support power users while not making the problem worse (In https://ritter.vg/blog-telemetry.html I describe that the 'confused users think FF is broken' problem got so bad management wanted to just disable RFP entirely, but I was able to show that these users are a very vocal minority and the problem is not as bad as it seems) while also not giving myself a maintenance burden but... maybe there a path forward where this dev extension - that can do things normal extensions can't - could potentially get more functionality...?
Standard
Strict
Custom
To me custom is something I define between Standard and Strict and not the next level after Strict. Strict already mentions that sites can break, so I'm pretty sure people associate the setting with breakage.
> Stronger protection, but may cause some sites or content to break.
Additionally Strict says :
Firefox blocks the following:
Social media trackers
Cross-site cookies in all windows
Tracking content in all windows
Cryptominers
Known and suspected fingerprinters
It's confusing if Known and suspected fingerprinters doesn't include resist fingerprinting. resist fingerprinting isn't even an option in Custom so how do ordinary users know where to set that option. You know, those users you say won't associate the Strict setting with breaking pages depite the fact that it clearly says so. Some kind of Schrödinger's user? Too dumb to understand the warning, but smart enough to know special settings?
You may avoid unnecessary bug reports that way but maybe only because users don't recognice that they are tracked per fingerprinting. It's not like websites would tell them.
Feels like Mozilla traded their time for my privacy.
The reason for spoofing the time zone (to UTC) is that it is one of the many things used to fingerprint users. There is an unintended side effect however: a mismatch with the IP geolocation could out you as a VPN user even if no VPN is actually used.
When Youtube still supported trends, visiting the trends page from Poland with Safari set to English gave you really interesting results. Mostly intellectually-stimulating content from channels like Veritasium, with a smattering of reviews, trailers, music and focus soundtracks thrown in. Meanwhile, visiting that same page on (Windows) Chrome set to Polish gave you the typical "you won't believe what this man just did!!!" crap.
This stupid "war against bots" is going to lead to the downfall of the Internet and effectively turn it into another walled garden where only "approved" (anti-)user agents are allowed. Don't fall for the nonsense about "AI scrapers" --- it's just a way to manufacture consent.
Those images also used to crash all the early GUI irc and chat clients that showed inline images without size checks...
Once you added a redirect rule for the IP to apache you'd just check your log and see the IP that was hitting you every couple of minutes poofed for a good few hours.
That's nuts. I suppose you had Webalayzer on a minutely cron job. It might have been drawing more resources than Apache itself!
These are sad times we're living as far as openness of the web goes. People would have less of a scraping problem if their websites didn't ship with 20MB of JS.
Google bot is generally fairly well behaved, but this is not the case for all scrapers and it can cause significant traffic (and expense).
Imagine you run a company register for a local government. You want to let people look up companies by their registration number (which they must disclose in all communications to you) to see if they're legit and whether any warnings have been raised against them. You don't want unscrupulous marketers to just be able to `SELECT * FROM companies WHERE type='nail_salon' AND city='london'`.
If you aren't super strict about scraping, some shadowy business in Neverland, completely unconcerned with following your laws, will build that database.
Is this data not public for some reason? I think it will not hurt if there are multiple copies spread between public offices and private companies. What really hurts is a private company hammering your webserver for their own profit. They should get their own copy.
Reg_no, status, no_warnings_last_12m
I can't, because every request comes from a new IP!!!
Rate limits didn't work because they kept rotating IP addresses.
I'm pretty sure Turnstyle would allow more people through than my current solution, but this was quick and easy. I expect to have to ban more ASNs from other countries in the future but the worst bots are now gone.
[0]: https://konform-browser.codeberg.page/
[1]: Most? All? Without any telemetry, relying on user reports and our own testing here.
Another case of the much predicted downfall of freedom due to "people who hide themselves must have something to hide, so they are automatically suspicious"
They send these emails you know? "CF saved you XXX Gb of data and protected your from YYY attacks". I have few high load web sites which I turned CF on for a while. Knowing my traffic pretty well, I can say these "CF saved you XXX Gb of data and protected your from YYY attacks" is absolute bullshit with numbers greatly exaggerated.
Since wwe can't catch them on this lie, they can put any number they like to make their "service" attrractive.
If tech companies weren't waging a war against public resources in an attempt to get their grubby little hands on every bit of data they can, we wouldn't be in this mess.
If there were a more reliable "I'm not a bot" signal and maybe some reliable method of rate limiting, we could do away with Turnstile and just let people through again. Unfortunately, every well-intended privacy measure is abused by AI's war against the public good.
You were never entitled to it in the first place
I'll make sure to fail all cloudflare turnshit in the future.
Normally websites feature test and just skip using obscure disabled APIs, or more likely, websites don't use those APIs at all or only tracking scripts use it, which are already optional usually.
Problem with CF is that if you want increased security they'll prevent you from gaining it everywhere, even on sites they don't protect, or prevent you from accessing services even the ones you paid for. Browsers don't allow disabling APIs per domain, so you're either at risk everywhere or you're blocked from accessing a lot of things for no particular reason.
CF can't be bothered to feature test.
That pref is there for the Tor Browser.
Also enabled by default for Konform Browser and Mullvad Browser, which borrow many of the privacy- and security-related patches from Tor Browser.
WebGL finger printing is just one of many things you need to do if you actually want to stop automation. There is no way round it other than requiring ID of some sort.
Internet Archive passed?
PoW wouldn't be a big issue for them though since their volume is much lower.
Nevertheless even for these high value cases, you can still argue that it disincentivizes the business model, it becomes less efficient.
I don't use Anubis though. I just make my site not take five seconds to render a page so bots can overload it easily? It's not actually that hard?
ideally one would pick something a bit more forgiving than a linear function, to avoid penalizing too much users connecting from CGNAT
Some sort of decentralized trust web seems like another option, though less viable.
So it’s not quite as horrible as it sounds.
I have setting up Anubis for my own sites on my todo list. And I wish more people did it too. I don’t really mind waiting a little bit extra every now and then before the page loads. What I do mind is ReCaptcha asking me to click all the pictures with buses in them etc. And especially when I have to do it several times over before it’s happy. I’d rather wait a minute for a page to load than to ever solve a ReCaptcha again, if given the choice.
I don't know about you, but if a random webpage takes 60+ seconds to load, I just close it and choose to never interact with that site again (unless it's my bank, which is a real and annoying occurrence).
They don't now, but enough "high value to the bots" pages turning on JS or complicated redirects will simply result in the bot authors adding JS execution or redirect following so they can continue "botting" the sites they want to scrape.
It's a hole with no bottom. Each one-up on the anti-bot side will eventually be handled on the bot side.
And there are just not enough sites using Anubis for the people and companies running the bots to care to do that.
If you do care bypassing Anubis is trivial.
Anubis is active when a user agent looks like a web browser (e.g. contains the "Mozilla" substring every major browser uses). The reverse proxy serves an interstitial page that does a proof-of-work check, validated server side, setting a cookie if it passes.
This means a legitimate user won't constantly get the proof of work check, because they already passed it. But AI bots rotating through tons of residential IPs to scrape your forum or git forge or whatever will be slowed down.
Overall, I like the idea. It's unobtrusive, privacy preserving, and seems to be working out well for a lot of sites.
But in principle I agree that there's no good answer to this, scraping _is_ useful and I bet most of us here had scraped something, it is AI company and their use of human's material for training without consent and return that led us to this (I know botting exists in forum since forum is a thing but it is easily solved by human moderators and keyword filter)
What all security extensions do you run? After running into issues over the years, with extensions doing multiple things that fight each other, I switched to trying to block via ublock origin as much as possible, then prefer other extensions to just do one thing to extend coverage, like this one. Makes it much easier to troubleshoot/exclude/disable when it breaks something vs. fiddling in settings.
If randomized canvas stuff was cracked down upon as a bot thing but now everyone with a copy of Firefox is doing it, maybe Cloudflare should just “legalize” it?
I would get locked out of the account on all devices after saying these things until I compeleted their turnstile. For many accounts I just never used them again.
I could go more into this, but im highly suspicious of Cloudflare and of course X/Twitter in this regard. Ive been reccomend people to follow on anonymous twitter accounts for people I went to elementary school with and havent spoken to in years and have no digital connection to. Its very weird.
Nearly all of our sites are visiting by extremely tech literate folks, the exact type that may not be using Google Chrome or Firefox.
Also by default addons.mozilla.org is a privileged site so of course they include google tracking in it and they get the proper fingerprint no matter what you have configured.
Aside from general dev, could use a hand in bringing it to more platforms (mobile and flatpak are frequently asked) and taking a closer look at fingerprinting protections and what's currently tripping up the turnstile.
AMOs privileges are limited to (A) installing extensions with only one prompt (instead of two) (b) launching some sort of "UI Tour" feature that highlights some features of the UI and (c) extensions cannot, by default, operate on the site. That last one is an unfortunate trade-off we've made because of the massive waves of malicious extensions. You can re-enable extensions access to AMO on a case by case basis: https://support.mozilla.org/en-US/kb/quarantined-domains but I recognize this is an opt-in, non-default configuration.
I am saddened to hear we use Google Analytics on the site, but I can tell you with certainty that it is not bypassing any of Firefox's built-in fingerprinting protections or getting any privileged access that way.
So no real benefit for bot detection here. Just a privacy nightmare for everyone else.
The breadth of responses here about people who can't reproduce this (or can) is one of the most frustrating things about working on fingerprinting protection. I also cannot reproduce this behavior, and have to assume that there is some complicated, behind-the-scenes risk assessment that is being done and some people trigger it and some don't. If any Cloudflare devs want to chat, I would love to. While not a normal way to contact us (support requests will be ignored), I can be reached at security@mozilla.com
I use Cloudflare protection on all my website but only the account creation page uses Turnstyle.
Which, to be clear, is the entire problem: given how much of the internet goes through them, they should have enough alternative signals as to wether you’re not a bad actor that are stronger than this specific one.
However, this also presents the problem that there’s barely any users in their base with your exact configuration, so getting any actual solutions might just take forever.
Yeah, this needs to be burned to the ground.
Cloudflare's demo page still treats that as a pass, but complains about it. As is often the case with Cloudflare, I expect that they'll then take no responsibility for sites that use more aggressive settings.
So if you want privacy, you have to accept poor and sometimes insecure services.
Obviously this is terrible, but I think there's a possibility it's the least terrible option? Another option is IP reputation, which I think is worse. Or scanning a code with a non-rooted phone, which I think is even worse than that!
There isn't one, and pretending otherwise is nonsense because humans will always provide their credentials to something to act on their behalf.
In the limit you end up with Chinese phone farms.
You can forget about it. It is not possible. Simple as that.
Then when it's "processing", do them in bulk and prioritize slower users. There's huge opportunity do bot checks after checkout without affecting user experience.
Also on product launches you could add unique field which requires user to input, for example that way bots can't prepare for launches.
It's hard but it's not impossible. You can make it very inconvenient for scalpers. They need to poll at volume so their behaviour is very much detectable. A hard stance is required on IP rep, especially for more in demand concerts.
Which sounds extremely difficult to differentiate
Here's a more real-world projection of the cost and server impact - The Bandwidth Cost of AI Crawlers: What Scraping Really Costs Publishers - https://aipaypercrawl.com/articles/ai-crawler-bandwidth-cost
Cloudflare, Google Captcha, HCaptcha etc. are all shitty technical solutions because, as we are all discovering, it comes at the cost of our privacy (i.e. our personal data may monetise these services) and / or our computing resource and time. If current copyright laws aren't sufficient to prevent this, we have to acknowledge the system is broken. The answer could be enhancing it with some kind of Digital Millennium Copyright Act (DMCA) -like laws, but in favour of the creators against BigTech or rogue actors.
- Web-scraping and copyright law - https://www.neudata.co/blog/web-scraping-and-copyright-law
- Why DMCA Claims Against Web Scrapers Face Long Odds - https://capstonedc.com/insights/why-dmca-claims-against-web-...
> we have to acknowledge the system is broken
The system is broken. It probably takes, what, 10 seconds or less to use a residential or foreign proxy, 6+ months to internationally track and prosecute a single offender? So like a million times more effort going the regulatory route.
However, a big difference with crimes involving the internet is that they can be launched from anywhere. In the real world, I can't steal from someone unless I'm physically present in the same country as my victim. On the internet, the US could outlaw scraping and Russia would keep doing it.
So.. what made it go away, and how could learning about that help us make all the rest of it go away too?
As for issues like bots overloading websites or using too many resources scaling laws will take care of it quickly, it’s not like you can’t serve thousands of RPS from a Raspberry Pi these days.
The thing why Cloudflare got invented isn't AI scrapers. These are just the latest development... the original reason why Cloudflare got created and why it experienced such a meteoric growth is DDoS and botnets.
Yes. We need regulation in the AI space. But it will be useless as long as bad actors aren't held accountable - and a lot of the bad actors aren't in our jurisdictions. You got hacked devices all over the world in giant botnets, controlled by Russia, Chinese, Iranian and North Korean actors. You got Chinese AI scraper bots as China is heavily investing into training their own models. You got Indian, Filipino and Myanmar-based scammers.
And frankly I have no idea how to get all of that under control. As much as I'd like to see sanctions against both domestic and foreign enablers of abuse (which includes residential ISPs) - it's going to be one giant ass whack-a-mole game.
Who profits the most from it?
The services charging you to defend against it, that's who.
I'd like to hear from someone who worked on WebGL and how they feel about their ambitions being utterly subverted. Remember when the dream was playing games i. the browser?
I keep getting the turnstile and having to click the "I a human" button.
I'm not good at creating petitions but can happily sign it. Also with stop killing games and anti-chat control.
I can imagine this can get a traction, if it's explained in youtube video to "normal" people.
b. Accept Only Necessary Fingerprinting
And then legislation required those consent boxes back, so everyone built their own, instead of demanding that the default should be changed back.
Even simply changing the user agent was sabotaged at Firefox, and choosing one user agent per domain is wishful thinking.
I doubt politicians care much about fingerprinting, though. They're more afraid of actual businesses getting attacked by bots than they are about Linux users with weird setups not being able to access some websites.
this can mean WebContent process is crashing
>Turns out it's because Cloudflare wants to have a fingerprint of your device via WebGL, the only reason for doing this would be tracking.
> So Cloudflare just banned all WebKitGTK browsers as I guess they put an exception for Safari.
This is false. I ran firefox with:
* hardware acceleration disabled (so software renderer, nothing to fingerprint)
* resistfingerprinting enabled, including letterboxing with default window size
* webgl disabled
* VPN enabled
* In a Windows VM
By all accounts this should be the most suspicious fingerprint ever, but turnstile happily lets me through. If they want to track people, they're doing a pretty bad job. My guess is that OP's browser is getting banned because his WebKitGTK has a weird fingerprint, not because of webgl or whatever.
> Such things are blocked in WebKit, and have been for years. Meaning it's tracking so awful that even Apple would block it, and as far as I can tell it's not the kind of privacy protection you can easily disable in it.
This is also false. Webgl fingerprinting works just fine on Safari. They might try to mitigate it by adding some noise, but that's not so different than what firefox does, and is certainly not "blocked".
So why is Cloudflare saying the author got blocked because of WebGL?
> > Such things are blocked in WebKit, and have been for years. Meaning it's tracking so awful that even Apple would block it, and as far as I can tell it's not the kind of privacy protection you can easily disable in it.
> This is also false. Webgl fingerprinting works just fine on Safari. They might try to mitigate it by adding some noise, but that's not so different than what firefox does, and is certainly not "blocked".
While I don't have an iDevice to try, the assumption that they are special cased is fair... because they are: https://blog.cloudflare.com/eliminating-captchas-on-iphones-...
(Yes, this is basically WEI in a shinier package.)
No idea. I can't even reproduce the error OP got with webgl disabled.
fingerprintingProtection works fine on the other hand, but then again that's intentionally less intrusive.
Official Firefox can be leaky unless you build it yourself with some build-time changes or use a fork with such[0]. Am I guessing right that you still have Webcompat, RemoteSettings, and Nimbus enabled still? How do you know a compatibility intervention isn't causing your browser to open the kimono just enough to "unbreak the page"?
> My guess is that OP's browser is getting banned because his WebKitGTK has a weird fingerprint, not because of webgl or whatever.
My guess is a different flavor of the same: Not matching an expected fingerprint (simplified: whitelist vs blacklist approach) combined with other factors.
[0]: I'm currently aware of Tor Browser, Konform Browser (am dev), Mullvad Browser, and to a certain extent Waterfox, LibreWolf, and r3df0x doing that.
See my other comment, tor browser works fine too: https://news.ycombinator.com/item?id=48346659