[flagged]
1. For individual use (or company use but each user is on their device) typically the traffic is drown out in regular user activity since we use the same browser and no particular measure is needed, it just works. We have options for power users.
2. For large scale use, we offer tailored solutions depending on the anti-bot measures encountered. Part of it is to emulate #1.
3. We don't deal with "blackhat bots", so we don't offer support to work around legitimate anti-bot measures such as social spambots etc.
One thing to note, there are different "tiers" of websites, each requiring different counter-measures. Not everyone is pursuing the high competition websites, and most importantly as we learned in several cases scraping is fully consensual or within the rights of the user. For example:
* Many of our users scrape their own websites to send notifications to their discord community. It's a super easy way to create alerts without code.
* Sometimes users are locked in their own providers, for example some companies have years of job posting information in their ATS they cannot get out. We do help with that.
* Public data websites who are underutilized precisely because the data is difficult to access. We help make that data operational and actionable. We had for example a sailor setup alerts on buoys to stay safe in high waters. A random example[1]
My original point was not necessarily about bypassing anti-bot protections, and rather to offer a different branch of browser automation independent of incumbent solutions such as Puppeteer, Selenium and others, which we believe are not made for this purpose, and has many limitations as TFA mentions, requiring way too many workarounds as your solution illustrates.
We avoided the fork & patch route because it's both labor intensive for a limited return on investment, and a game of catching up. Updating the forked framework is challenging on its own right, let alone porting existing customer payloads to newer versions, locking you de-facto to older versions. I did maintain a custom fork at a previous workplace that was similar in scope to Browserless[0] and I can tell you it was a real pain.
Developing your own framework (besides satisfying the obvious NIH itch) allows you to precisely control your exposure (reduce the attack surface) from a security perspective, and protects your customers from upstream decisions such as deprecations or major changes that might not be aligned with your customer requirements. I also have enough experience in this space to know exactly what we need to implement and the capabilities we want to enable. No bloat (yet)
> you will burn the user's fingerprint depending on scale
It's relative to your activity. See my other comment about scale and use cases, for personal device usage this is not an issue in practice, and users can automate several websites[1] using their personal agents without worrying about this. For more involved scenarios we have appropriate strategies that avoid this issue.
> we fix leaks and bugs of automation frameworks
Sounds interesting! I'd love to read a write up, or PRs if you have contributed something upstream.