Hacker News Reader

Comment by cortesoft | Hacker News Reader

> The crawl function reads a URL from the SQS queue, then Pyppeteer tells Chrome to navigate to that page behind a rotating residential proxy. The residential proxy is necessary to prevent Google from blocking the IP Lambda makes requests from.

I am very interested in what a 'rotating residential proxy' is. Are they routing requests through random people's internet connections? Are these people willing participants? Where do they come from?

View on HN

Check out Luminati for example. They have a huge network of true residential IPs to exit traffic from, and you have to pay a hefty premium per GB of traffic to do so ($12.50 per GB for rotating residential IPs, but requires a minimum $500 commitment per month). The reason they can offer this is because they're exiting traffic through the users of the free Hola VPN Chrome extension.

It looks like that's basically what they are (https://smartproxy.com/blog/what-is-a-residential-proxies-ne...).

A residential proxy is listed as an "IP address provided by an Internet Service Provider", but I still don't really understand how they get access to them. ISPs have to be selling them access, right?

My guess is free VPN services and browser extensions which resell your residential network in tiny chunks.

Yes, to all your questions.

https://luminati.io/

Providers of the 'free' Hola vpn.

I think you can make a reasonable argument that Hola VPN is largely exploiting users who don't actually understand and consent to having their IP address and connection used as a proxy.

How awful.

"80M+ Monthly devices hosting Luminati's SDK" & "100% Peers chose to opt-in to Luminati's network" (https://luminati.io/network-details)

There is a 0% chance that 80M+ are agreeing "I am OK with Luminati selling access to my home internet connection to any party able to pay", which seems like an honest description of their business model. More likely Luminati is paying unscrupulous app developers to include this SDK in their apps, and some put some legalese into 10,000 word install-time agreements that no one reads.

People who use the network also participate in the network themselves.

Reply on news.ycombinator.com