avastel

Author here.

I first built FPScanner during my PhD around 2017–2018, when I was doing research on browser fingerprinting and bot detection. After that I did not really maintain it for years.

I recently decided to revive it because things changed a lot. Automation is much easier now. With free automation libraries like Puppeteer/Playwright/Selenium + headless Chrome and cheap proxies, you can build decent bots very quickly. At the same time, open source defensive tooling is still quite limited, or very basic.

To be clear, FPScanner is not trying to be a silver bullet.

It is a small, self-hosted library that focuses on deterministic client-side signals: - webdriver and automation flags - CDP-related artifacts - automation framework markers (Selenium, Playwright, etc.) - JS cross-context inconsistencies: main JS context, iframes and workers

It also generates a JA4-inspired fingerprint ID for clustering sessions, and includes encrypted payload + simple anti-replay protections.

There is no ML here, no "AI detection", and no claim to block 100% of bots. The idea is just to expose strong, explainable signals and make automation a bit more expensive. I assume attackers can read the source code.

I tested it on different devices and browsers to avoid JS errors and obvious false positives, but I’m sure there are edge cases. If you try it on less common browsers or hardened setups and it breaks, please let me know or open an issue.

Happy to answer questions or discuss design choices / limitations.

I wrote a blog post recently about the technique used by LinkedIn to do extension probing, as well as other ways to do it with less side effects

https://blog.castle.io/detecting-browser-extensions-for-bot-...

Since I was also tracking this proxy network as part of my side project, I wrote a short blog post + give access to 16m+ proxy IPs IoCs that belong to this proxy network: https://deviceandbrowserinfo.com/learning_zone/articles/insi...

Note that even after the disruption, I'm still able to route millions of requests/day through IP IDEA's network

Since I was also tracking this proxy network as part of my side project, I wrote a short blog post + give access to 16m+ proxy IPs IoCs that belong to this proxy network: https://deviceandbrowserinfo.com/learning_zone/articles/insi...

I recently wrote about the limits of these kinds of fingerprinting tests. They tend to overly focus on uniqueness without taking into account stability. Moreover sample size is often really small which tends to artificially make a lot of users unique

https://blog.castle.io/what-browser-fingerprinting-tests-lik...

(Author here) TBH I don't think the main goal of Google was to make bots undetectable. It was more a side effect of preventing side effects while reviewing errors in the devtools: https://source.chromium.org/chromium/_/chromium/v8/v8/+/e08e...

It was the same when Google released the new headless Chrome a few years ago: https://antoinevastel.com/bot%20detection/2023/02/19/new-hea... It made vanilla/naive bots more realistic/detectable by default.

Interesting article. I’ve been curious for a while about how residential proxy IPs are collected too. Many come from shady browser extensions or mobile apps, especially free VPNs (wink wink Hola VPN). People often don’t realize they are turning their device into an exit node.

Some time ago I started to track this as a side project (I work in bot detection and was always surprised by how many residential proxies show up in attacks). It started just out of curiosity. Now I collect proxy IPs, which provider they belong to, and how often they are seen. I also publish stats here: https://deviceandbrowserinfo.com/proxy-api/stats/proxy-db-30...

For example, in the last 30 days I saw more than 120K IPs from Comcast and nearly 100K from AT&T.

I also maintain an open IP (ranges) blocklist, mostly effective against data center and ISP proxies. Residential IPs are harder since they are often shared with legit users: https://github.com/antoinevastel/avastel-bot-ips-lists

Even if you can’t block all of them, tracking volume and reuse gives useful signal.

Author here: I work in bot detection, and wrote this post to explain why privacy-conscious users (VPNs, Brave, LibreWolf, etc.) often get flagged or blocked by anti-bot systems.

I’ve seen a lot of frustration in threads here, so I wanted to offer a technical perspective on why these false positives happen, and how detection systems interpret signals from non-mainstream setups.

Author here: There’ve been a lot of HN threads lately about scraping, especially in the context of AI, and with them, a fair amount of confusion about what actually works to stop bots on high-profile websites.

This post uses TikTok’s obfuscated JavaScript VM (recently discussed on HN) as a case study to walk through what modern bot defenses look like in practice. It’s not spyware, it’s an anti-bot measure designed to make life harder for HTTP clients and non-browser automation.

Key points:

- HTTP-based bots skip JS, so TikTok hides detection logic inside a JavaScript VM interpreter

- The VM computes signals like webdriver checks and canvas-based fingerprints

- Obfuscating this logic in a custom VM makes it significantly harder to reimplement outside the browser (and so to scale an attack)

The goal isn’t to stop all bots, it’s to push attackers into full browser environments, where detection is more feasible

The post covers why simple solutions like "just require JS" don’t hold up, and why defenders use techniques like VM-based obfuscation to increase attacker cost and reduce replayability.

Reposting a similar point I made recently about CAPTCHA and scalpers, but it’s even more relevant for scrapers.

PoW can help against basic scrapers or DDoS, but it won’t stop anyone serious. Last week I looked into a Binance CAPTCHA solver that didn’t use a browser at all, just a plain HTTP client. https://blog.castle.io/what-a-binance-captcha-solver-tells-u...

The attacker had fully reverse engineered the signal collection and solved-state flow, including obfuscated parts. They could forge all the expected telemetry.

This kind of setup is pretty standard in bot-heavy environments like ticketing or sneaker drops. Scrapers often do the same to cut costs. CAPTCHA and PoW mostly become signal collection protocols, if those signals aren’t tightly coupled to the actual runtime, they get spoofed.

And regarding PoW: if you try to make it slow enough to hurt bots, you also hurt users on low-end devices. Someone even ported PerimeterX’s PoW to CUDA to accelerate solving: https://github.com/re-jevi/PerimiterXCudaSolver/blob/main/po...

Yeah, not (too) surprising after a few years in the anti-bot industry. Last week I looked into a Binance CAPTCHA solver that didn’t use a browser at all, just a basic HTTP client. The attacker had reverse engineered the entire signal collection and response flow, including how the CAPTCHA was marked as solved. They were able to forge the expected telemetry despite some obfuscation. https://blog.castle.io/what-a-binance-captcha-solver-tells-u...

This is pretty standard now in bot-heavy spaces like ticketing or sneaker drops. CAPTCHA often just ends up being a protocol to collect signals, and if those aren’t tightly bound to the browser/runtime, they get spoofed.

Also not surprised PoW isn’t holding up. Someone reverse engineered the PerimeterX PoW and converted it to CUDA to accelerate solving: https://github.com/re-jevi/PerimiterXCudaSolver/blob/main/po... At some point, it’s hard to make PoW slow enough for bots without also killing UX for humans on low-end devices.

Author here. A few weeks ago, someone posted a link on Reddit to an open-source CAPTCHA solver made for Binance’s slider challenge. It’s written in Python and works without using a browser. Just a custom HTTP client, some image matching, and basic reverse engineering.

I was curious and decided to dig into it. I wrote a long breakdown of how it works, how it solves the challenge, and what this says about how bots are built today. Many bots use headless browsers, but this one doesn’t, and it still gets through.

One of the main takeaways is how effective this kind of non-browser approach can be when CAPTCHA is deployed in isolation, without other layers like continuous behavioral checks.

Happy to discuss or answer questions.

Hi, author here. I wrote a blog post where I analyze Hidemium, a popular anti-detect browser. I break down the techniques it uses to spoof fingerprints and show how JavaScript feature inconsistencies can reveal its presence. Of course, JS feature detection isn’t a silver bullet, attackers can adapt. I also discuss the limitations of this approach and what it takes to build more reliable, environment-aware detection systems that work even against unfamiliar tools.

Using a 4G dongle makes it easier to hide in the crowd indeed. Since your traffic will go through heavily shared mobile IPs, probably with thousands of users behind them, anti-bot vendors won't/shouldn't block per IP, but per fingerprint/session cookie instead.

The DB contains different types of proxies: - Residential - ISP - Data center

I don't include mobile proxies since they're heavily shared, so knowing that an IP address was used as a proxy at some point is basically useless.

Regarding your remark, indeed, there are several shared residential IPs, including IPs of legitimate users who may have a shady app that routes traffic through their device. That's why I don't recommend blocking using IP addresses as is. It's supposed to be more of a datapoint/signal to enrich your anti-fraud/anti-bot system. However, regarding the block list, I analyze the IPs on bigger time frames, the percentage of IPs in the range that were used as proxies, and generate a confidence score to indicate whether or not it is safe to block.

I am working on a curated database of proxy IP addresses frequently used by bots: https://deviceandbrowserinfo.com/product/proxies-ips

So far I have ~ 3M distinct IP addresses per 30 days, with a lot of fresh proxy IPs, 1.7M. The DB contains only verified IP addresses through which I've been able to route traffic. It DOESN'T rely on 3rd party/open-source data sources.

I also made an open-source proxy IP block list based on the data: https://github.com/antoinevastel/avastel-bot-ips-lists

Author here: I've been in the bot industry/bot detection field for ~ 10 years. I frequently see strong opinion about bot detection on Reddit and HN, in particular why it doesn't make sense for bot detection companies (I won't name who, but you will guess), to treat you so differently based on your user agent, and why it shouldn't matter when it comes to bot detection.

That's why I wrote a blog post about the role of the user agent in bot detection. Of course, everyone knows that the user agent is fragile, that it is one of the first signals spoofed by attackers to bypass basic detection. However, it's still really useful in a bot detection context. Detection engines should treat it a the identity claimed by the end user (potentially an attacker), not as the real identity. It should be used along with other fingerprinting signals to verify if the identity claimed in the user agent is consistent with the JS APIs observed, the canvas fingerprinting values and any types of proof of work/red pill

Author of the post: bot detection and bot-related topics are often discussed on HN, particularly lately with the increase of AI scrapers trying to scrape protected websites to gather training data for LLMs.

Most of the time, people just recommend using CAPTCHA or implementing rate limiting. In the article, I try to cover the most popular bot detection approaches, such as CAPTCHA, IP-based rate limiting, geo-blocking, and static JS fingerprinting, and discuss their main limits.

Castle (https://castle.io) | Senior ML Engineer | Senior R&D Detection Engineer | REMOTE (Europe/EMEA) | Full-time

Hi HN! I’m Antoine, Head of Research at Castle. We’re a YC-backed Series A startup helping companies like Canva, Atlassian, and Rockstar Games fight fraud and abuse. We're building modern, AI-powered bot detection and abuse prevention tech, with a focus on real-time protection and behavioral intelligence. We’re a small, fast-moving team looking to add two technical roles:

Senior ML Engineer – Fraud & Bot Detection You'll work on real-time ML pipelines, anomaly detection, and adversarial ML techniques to stop bots and malicious actors. We’re combining traditional fraud signals with modern AI—LLMs, behavioral clustering, and vector-based search—to stay ahead of attackers. You’ll own models end to end and deploy quickly, with direct access to rich, structured behavioral and fingerprinting data. Ideal for someone with strong experience in the anti-fraud/anti-bot field who loves cutting-edge ML and wants immediate impact.

Senior Detection Engine Engineer (R&D) You’ll build and scale the core infrastructure behind our detection logic—rule engines, fingerprinting pipelines, and high-speed signal ingestion. We’re integrating AI-driven detection and auto-adaptive logic that reacts in real time. If you enjoy real-time systems, security, and automation, this is a highly technical role that’s at the core of our detection stack. Perfect if you’ve worked in security, data pipelines, or real-time analytics and want to outsmart fraudsters.

We pay US salaries globally | Fully remote from Europe/EMEA timezone | Flexible hours | Unlimited PTO | Paid parental leave We're actively hiring and committed to responding to every qualified applicant.

Drop us a line: antoine.vastel@castle.io

Author here: This is the 2nd blog post in a series about anti-detect browsers.

Anti-detect browsers are browsers that integrate features that aim to bypass anti-fraud/bot detection systems, such as: - native proxies integration;

- anti-fingerprinting/fingerprinting randomization;

- multi-profiles management (to have separate identities linked to different fingerprints and locations)

In this blog post, I analyze the Undetectable anti-detect browser, the JS script it injects to modify the fingerprint, and lower level changes done at the c++ level within the browser. In the end, I show how you can craft a simple JS challenge that exploits side effects of Undetectable to detect it.

Disclaimer: I know anti-detect browsers can also be used for privacy purposes. The purpose of this article is to cover the main mechanisms used by these browsers that can be used to bypass security systems and how websites can protect against these techniques.

I build a website related to fraud (detection), bot (detection), fingerprinting, email intelligence. You can test your browser fingerprint here https://deviceandbrowserinfo.com/info_device and get information about disposable emails https://deviceandbrowserinfo.com/data/emails/disposable/inde...

Author here: most websites that have valuable data targeted by scrapers implement some sort of anti bots mechanisms. In particular they will force the user to execute different JS proof of works and run different browser fingerprinting tests to verify the nature of the browser.

https://antoinevastel.com/ Mostly about bot detection and browser fingerprinting.

It doesn't. I can access my passwords locally.

"These API routes, like any API that pulls data from a database, are necessary to the day-to-day functioning of Stack Overflow. To protect routes like these from DDoS, here’s what you can do:

- Insist that every API call be authenticated. This will help identify malicious users. If having only authenticated API calls is not possible, set stricter limits for anonymous / unauthenticated traffic.

- Minimize the amount of data a single API call can return. When we build our front page question list, we don’t retrieve all of the data for every question. We paginate, lazy load only the data in the viewport, and request only the data that will be visible (that is, we don’t request the text for every answer until loading the question page itself).

- Rate-limit all API calls. This goes hand-in-hand with minimizing data per call; to get large amounts of data, the attacker will need to call the API multiple times. Nobody needs to call your API a hundred times per second. Filter malicious traffic before it hits your application. HAProxy load balancers sit between all requests and our servers to balance the amount of traffic across our servers. But that doesn’t mean all traffic has to go to one of those servers.

- Implement thorough and easily queryable logs so malicious requests can be easily identified and blocked."

When using HAProxy you can also try to leverage TLS/JA3 fingerprints (https://haproxy.com/blog/announcing-haproxy-2-5/) While it can still be spoofed using several packages , e.g. https://github.com/Danny-Dasilva/CycleTLS it can still provide a meaningful/easy-to-manipulate signal.

Moreover, most bots conducting L7 DDoS don't use real/headless browsers in order to be able to scale their attack, so it's highly likely they'll have a discriminating/inconsistent TLS fingerprint.

This can also be done directly in Fastly using e.g. https://developer.fastly.com/reference/vcl/variables/client-...

Another approach to proactively flag malicious IPs is to scrape free proxies. Indeed, most DDoS leverage lot of cheap/know bad IPs. It's frequent to see these free proxies in these attacks.

I did a demo a few years ago based on their paper: https://antoinevastel.com/browser%20fingerprinting/2019/03/2...

This is used by companies like Cloudflare, Discord. Facebook also has their own picasso like canvas fingerprinting (at least the last time I checked)

DataDome | NYC / Paris | Full Time | Onsite or Remote

https://datadome.co/

https://docs.datadome.co/

Bot hunters is a full time job! With our SaaS cybersecurity technology, we detect and manage non-human traffic in real-time to protect our clients’ websites and APIs.

Named as a Strong Performer by Forrester, DataDome is a growing cybersecurity pioneering in bot detection and protection for content publishers, eCommerce websites and advertisers across the world. We handle and protect more than 4 billions requests daily below 3ms (99p).

Our stack:

  - Stream data processing: Kafka / Flink / Elasticsearch
  - Real time detection engine: Java / Scala
  - Webservers modules (nginx, haproxy…): C++ / C#, nodeJS, lua
  - Infrastructure: 1000 VMs (AWS/GCP/Azure/baremetal)
  - Dashboard: Angular7 / Symfony3

We’re expanding our team and are looking to hire:

  - Cyber Security Analyst / Threat researcher
  - Senior Software Engineer : Java, Scala, C++
  - Data Engineer / Data Scientist : Scala, Java, Python
  - DevOps/SRE Engineer
  - FullStack Web Engineer : Angular7 / Symfony3

Join us: apply@datadome.co or https://www.linkedin.com/jobs/datadome-jobs-worldwide

I worked on browser fingerprinting and countermeasures during my PhD. In one paper, we showed that in the case of an anti canvas fingerprinting extension (canvas defender I think) we were able to extract the seed used to randomise the canvas on each. Since by default the seed was constant, you could use it for tracking: https://hal.inria.fr/hal-01820197 (the paper also talks about other anti fingerprinting techniques).

There is also a shorter version in a blog post: https://antoinevastel.com/tracking/2018/07/01/eval-canvasdef... (Evaluating the privacy implications of a canvas fingerprinting countermeasure)

Hi, I am the author of the post. I run some fingerprinting tests sometimes for research purposes. Nevertheless I don't see which of the tests would ask for any audio or photos permissions. Which browser were you using?

You're right, I changed the code.