BotOrNot (2021) | Hacker News Reader

BotOrNot (2021)

incolumitas.com

Kind reminder Google broke almost all TCP fingerprints (i.e. JA3) when introducing grease values and other stuff such as random order in the ciphers in Chromium. I work in a bot detection company and we put a lot effort on it, currently have an advanced implementation that ignores some extensions, order the fields, remove the grease values etc.

I have an open source implementation of normalized JA3 fingerprinting at https://github.com/fidraC/canary (Includes CreepJS and other stuff to be added / WIP) it’s not very complicated but I did waste some time before realizing that grease was everywhere, not just extensions.

Note: That is my secondary account for school. I’m not stealing someone else’s work. I use a different name as a sort of mental compartmentalization between different spheres of my life.

Google's plan is to break every type of client fingerprinting and then funnel everyone into using their browser attestation.

iOS isn't any better and already sends a unique "Private Access Token" with requests to Cloudflare.

link to bot detection page - https://bot.incolumitas.com/ - is down

I just get a 403. I assumed it was because of my VPN but even without it I get the same result.

It's been down for months, sadly. I was working on some dummy Puppeteer human-behavior script back then and the site helped a lot. Hope the guy's doing alright :/

Not down for me.

   echo 167.99.241.135 bot.incolumitas.com|sed -i -e1r/dev/stdin -e1N /etc/hosts
   FTPUSERAGENT= tnftp -4vdo/dev/stdout https://bot.incolumitas.com
   links -no-connect https://bot.incolumitas.com
   sed -i 1d /etc/hosts

   curl --resolve *:443:167.99.241.135 https://bot.incolumitas.com

False. It's up.

Alas, does not accept TLS1.3.

     printf 'HEAD / HTTP/1.0\r\nhost: bot.incolumitas.com\r\nconnection: close\r\n\r\n'|openssl s_client -connect 167.99.241.135:443 -ign_eof
     printf 'GET / HTTP/1.0\r\nhost: bot.incolumitas.com\r\nconnection: close\r\n\r\n'|openssl s_client -connect 167.99.241.135:443 -ign_eof
     echo|openssl s_client -connect 167.99.241.135:443  -showcerts|openssl x509 -text

Well I get a 403 when trying to load the detection page (in Firefox) from behind a VPN. Guess that's a fail then?

What are the current SOTA open source projects for bot detection?

Some parts are outdated now. Bot detection vs web scraping is an ever evolving arms race.

Anyway, incolumnitas blog is a great source of information about bot detection and techniques to avoid being detected. We have implemented many successful methods based on it in https://scrapingfish.com/.

Ugh, just remembered hotornot circa 2004. Those neurons haven't been exercised in a while...

Reply on news.ycombinator.com