Some of these so-called "advanced" techniques:
* We use our own mobile emulation software (similiar to bluestacks). Turns out, mobile helps with a lot of things (below).
* We use mobile IPs only. Mobile LTE data users are behind CGNATfor IPV4. You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point.
* All you need is a new useragent and browser fingerprint; combined with emulation + mobile IPs, there's really no easy way for companies to block this.
* With the advent and ease of virtualization; we avoid using any headless browsers. Seriously, if you can, never use headless. This should be close to rule number one for anyone looking to operate any kind of scrapers. All of our scrapers are run in isolated virtual instances with full mobile browsers.
* We can easily reset our device identifier, device carrier, simulated SIM information, and especially important is the Google advertising ID that is set per device; the list goes on. The key here is #1, our mobile emulation software.
* Our automation scripts are a combination of human recorded set of actions which we then perfected and can run in certain loops (for some of our data).Also, does this work only for browsers or also for mobile apps? I have always assumed that it is always theoretically possible to get data from browsers (very extreme resort is save the browser page / (screenshot + computer vision)); but it can be impossible to get data from apps (especially ios). Are my assumptions correct?
Also can you explain mobile IPs more? If they are such a big vulnerability, why is there no potential solution to them?
Once we're all on IPv6 we can go back to blocking IPs. But then IPv6 creates its own problems.
Basically what's different about mobile and laptop IPs?
Your home or local coffee shop might have 10-20 users max behind a single IP. A mobile network on the other hand might put most a small-medium city behind 3 or 4 IPs.
> You can't block one ip without possibly blocking hundreds of innocent IPs using the same exit point.
This is similar how Tor is supposed to work in practice. Make everyone look like the same user so these companies can't tell who's who.
In this specific case, the people wanting to detect bots want to avoid having their signals burned, the scrapers don't want the defenders to know which signals they're able to cloak since it will spur new signals development. So what gets disclosed publicly is just the really simple stuff.
It's kind of sad. There's a huge pipeline for getting people up to speed on security engineering, since there's a lot of incentive for everyone in the ecosystem to share information as publicly as possible except for relatively short responsible disclosure windows.
In contrast, the only way to learn abuse engineering is to happen to work in an organization with an abuse problem (and live with the frustration of an abnormally long ramp up period), or to go black hat. And likewise it's quite hard for the good guys to actually learn from each other, since they're spread across so many companies and it's thus hard for them to exchange information on what works and what doesn't.
When treated like a puzzle it can be really interesting. So I thought I'd share a few tidbits.
1) We did a simple 'speed' test, how many queries per second were coming from an IP, and auto-ban on the limit being exceeded, started at 100qps and watched as the traffic moved down to 99.5qps. Pushed to 10qps and watched the traffic follow it down. Even at 3qps you would get traffic at bit over 2 qps trying to limbo in under the limit.
2) At that time, lots of people who highjacked browsers with toolbars sold scraping as a service to third parties. Their toolbar would check in to see if it should do a query and it would launch a query and return the results without the user even knowing. One company, 80 legs, was pretty up front about their "service", SEO types would use it to scrape Google results to see how their SEO campaigns were doing.
3) The majority of the traffic had criminal intent, looking for metadata on web pages to indicate they were running an unpatched version of some store software or had sql injection bugs. These would often come from PCs that had been compromised for other purposes or "zombie" PCs. We could rapidly map out these networks when we got 100 queries from 100 different IPs looking for "joomla version x.y",p=1 through "joomla version x.y",p=100. We briefly played around with sending them official looking SERPs but all the links went to fbi.gov though an obfuscator.
One of more effective strategies was to field a "black hole" server, basically it was an http server that answered like you had gotten hold of it but then it never sent any data. With some simple kernel mods these TCP connections were silently removed on our end so they took no resources and the client would wait basically forever. We ack'd all keep alive packets with "Yup, we're here." so they just kept waiting and waiting.
It really was a never ending game. We mass banned an entire Ukranian ISP because out of billions of queries not a single one was legitimate.
Mailinator would do a similar thing with their custom email server hardware. Since they didn't really use sockets in the traditional sense, they were happy to give slooow replies and never disconnect "bad" connections.
If you don't mind, can you elaborate on the 2nd point? Specifically, what do you mean by hijacking browsers with toolbars?
Lots of people did it, even Blekko (although we stopped after we figured out it was just a scam), and the way it worked is toolbar company X would approach your internet site and say "we can send you a lot of traffic, just use this toolbar, we'll even pay you every time one gets installed."
Anyway, the toolbar would hook itself into the address bar and search selection hook in the browser and redirect every search to where ever it was told to. The nefarious part is that these tool bars were often shipped to the company as binaries not source, so you didn't really know everything they were doing. Once we figured out they were scumbags we also found out that they did some really scumbaggish things.
We stopped using them but lots of people did and finally the browser makers re-wrote the browsers to make what they did either impossible or easy for the user to revert and those guys rolled up their shops and went on to become some other type of scam.