Hacker News Reader

Comment by jasonwcfan | Hacker News Reader

I mentioned this in another comment, but I know from experience that it's impossible to reliably differentiate bots from humans over a network. And since the right to automate browsers has survived repeated legal challenges, all vendors can do is make it incrementally harder to weed out the low sophistication actors.

This actually creates an evergreen problem that companies need to overcome, and our paid version will probably involve helping companies overcome these barriers.

Also I should clarify that we're explicitly not trying to build a playwright abstraction - we're trying to remain as unopinionated as possible about how developers code the bot, and just help with the network-level infrastructure they'll need to make it reliable and make it scale.

It's good feedback for us, we'll make that point more clear!

View on HN

> but I know from experience that it's impossible to reliably differentiate bots from humans over a network

While this might be true in theory, it doesn't stop them from trying! And believe me, it's getting to a point where the WAF settings on some websites are even annoying the majority of the real users! Some of the issues I am hinting at however are fundemental issues you run into when automating the web using any mainstream browser that hasn't had some source code patches, I'm curious to see if a solution to that will be part of your service if you decide to tackle it.

Don't take this the wrong way, but this is the kind of unethical behavior that our industry should frown upon IMO. I view this kind of thing on the same level as DDoS-as-a-Service companies.

I wish your company the kind of success it deserves.

Why is it unethical when courts have repeatedly affirmed browser automation to be legal and permitted?

If anything, it's unethical for companies to dictate how their customers can access services they've already paid for. If I'm paying hundreds of thousands per year for software, shouldn't I be allowed to build automations over it? Instead, many enterprise products go to great lengths to restrict this kind of usage.

I led the team that dealt with DDoS and other network level attacks at Robinhood so I know how harmful they are. But I also got to see many developers using our services in creative ways that could have been a whole new product (example: https://github.com/sanko/Robinhood).

Instead we had to go after these people and shut them down because it wasn't aligned with the company's long term risk profile. It sucked.

That's why we're focused on authenticated agents for B2B use cases, not the kind of malicious bots you might be thinking of.

> they've already paid for.

That is the crux, rarely is it a service being scraped that they paid for

Depends on the use case. Lots of hospitals and banks use RPA to automate routine processes on their EHRs and systems of record, because these kinds of software typically don't have APIs available. Or if they do, they're very limited.

Playwright and other browser automation scripts are a much more powerful version of RPA but they do require some knowledge of code. But there are more and more developers every year and code just gets more powerful every year. So I think it's a good bet to make that browser automation in code will replace RPA altogether some day.

Many times it is scraping aggregators of data that those aggregators also did not pay for.

Reply on news.ycombinator.com