Pretty cool, but I recommend anyone wanting to do this kind of thing to check out the source Puppeteer library. You can do some really powerful stuff and make a custom crawler fairly easily.
I haven’t looked into it, but I imagine it has a pretty clear fingerprint as well. So it would be easier to block than stock chrome in headless mode.
0 - https://github.com/GoogleChrome/puppeteer/issues/850 1 - https://github.com/cretz/chrome-screen-rec-poc/tree/master/a... 2 - https://bugs.chromium.org/p/chromium/issues/detail?id=706008
[0]: https://github.com/yukinying/chrome-headless-browser-docker
Let me quickly add instructions here, first you need to install some dependancies, add the following to dockerfile:
RUN apt-get install -y gconf-service libasound2 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
browser = await puppeteer.launch({
args: ['--no-sandbox'] /*, headless: false*/
})