Hacker News Reader

HNHacker News

Top New Best Ask Show Jobs

Comment by vivzkestrel | Hacker News Reader

Full thread

vivzkestrel·

- can someone with expertise give us an overview of the architecture involved doing this

- let us say you ran yt-dlp inside python aiohttp

- surely your ll run a limit soon as your ip address will be flagged

- what solutions do we have to auto rotate proxies in python

- are there better, faster and more reliable ways to go about downloading a 100 million videos without getting your ip address blocked?

View on HN

Use Tor to proxy. Every time you get a connection error, send a SIGHUP to the Tor daemon, which will tell the Tor daemon to establish a new connection, thereby giving you a new IP. You may not get as good results as a residential proxy, but these proxies are prohibitively expensive if you're doing this for your own personal or academic use, and the Tor method works surprisingly well.

- am not familiar with tor except for the fact that it uses something called onion with multiple hops and is very hard to trace making it privacy friendly

- are we talking here about using tor programmatically?

- what I am referring to on my end is how to go about programmatically downloading a 100 million videos one after the other on s3

- what kind of bot detection / evasion / ip address / proxy rotation mechanisms are needed

- are you familiar with libraries in the pypi ecosystem capable of tackling some of these concerns?

- is aiohttp / asyncio python enough for this paired with multiprocessing or do we have better options?

- Use the python Library stem to control the Tor daemon and send sighup commands when necessary

- yt dlp has builtin support for tor

- write the retry logic using your favourite coding AI

- spin up a cheap cloud instance and let it run

Dude, Tor's exit nodes are flagged everywhere.

I downloaded the Laion Disco dataset this way.

Google may supply cache boxes to ISPs which can be manipulated to let download any video you want

Reply on news.ycombinator.com