Benchmark: Puppeteer vs. Selenium vs. Playwright vs. WebDriverIO | Hacker News Reader

Benchmark: Puppeteer vs. Selenium vs. Playwright vs. WebDriverIO

blog.checklyhq.com

118 pointsragog29 comments

I'm really excited by what Playwright has been up to on the python side recently with the pytest-playwright package. Browser-testing a Django app can be as simple as:

  def test_homepage(live_server, page):
      page.goto(live_server.url)
      assert page.innerText('#splash-quote') == 'our splash-quote'
  
  $ pytest -k test_homepage --browser chromium --browser firefox --verbose
  tests/test_browser.py::test_homepage[chromium] PASSED                [ 50%]
  tests/test_browser.py::test_homepage[firefox] PASSED                 [100%]

It would be neat if I could get coverage from these tests using python, but other than that I agree. Playwright is great.

It’s been some time since I’ve used Selenium, but I don’t remember it being any more complicated than that to test arbitrary websites.

What is that code doing that’s special?

Not OP. For one thing, Selenium was not super reliable, at least for me, especially when trying to run multiple instances. Do note that I used the official docker swarm method.

When I was exploring that option, Playwright was still windows only. Super excited to see it is available on Linux and Mac.

Playwright has always supported all platforms: Win, Linux and Mac. Is there something that didn’t work for you?

Maybe it didn't have python support? Sorry, I may have misremembered. I do remember something missing that I couldn't use it in my existing dev setup. As solo developer, I prefer to keep my dependencies minimal, mostly python.

While interesting, performance is a tiny concern here. Most browser automation is going to have a lot of implicit and explicit waits that dwarf the differences in performance between libraries. Moreover, someone may choose Puppeteer over Selenium for functional differences (for example, Selenium is easier to catch and block)

It’s only a “tiny concern” because the results seem so similar. But if one was drastically slower, then it is important.

However I don’t believe this testing methodology shows the true performance impact of the different frameworks. Only once you have a significant number of assertions and page interactions then to we really see the performance impact of the architectural differences.

Sure. In my experience (I've done a ton of this kind of work) well designed flow can take at least 30 seconds, up to several minutes depending on what you're automating. A difference of 5-10 seconds tends to not matter in that situation, and you'll be more interested in things like code ergonomics, memory leaks over time, etc.

It is indeed hard to have an all-encompassing comparison that will take everything into account. Unfortunately, to have precise answers in feasible time you have to narrow the scope. Here we looked at one parameter of a complex equation.

Interesting.. I've chatted with Tim and learned a lot of people run Puppeteer on Lambda.. there is an easier way (spoiler - I work on the open source project OpenFaaS) - https://www.openfaas.com/blog/puppeteer-scraping/

Hope it's of some use to folks who may be struggling to compile lambda-chrome or with local testing.

I used Selenium a lot in a past life, so this post is interesting to see, but the UX and maintenance cost is also going to need to be factored in with bigger suites of tests.

This is certainly a good approach if you need that extra performance or plan on heavy use and that outweighs the extra ops. In my own experiments with Chrome and Puppeteer on Lambda, it has been rather slow as you also noticed.

With that said, if someone does want to use Lambda, it's easier than outlined in the article. Use one of the layers listed at https://github.com/shelfio/chrome-aws-lambda-layer and then `require('chrome-aws-lambda')` works fine, making it easy to throw something together even in the AWS console's own editor.

The combination of headless browsers and any form of FaaS / Lambda / serverless solution is immensely powerful. There are all kinds of businesses popping up around this tech stack, of which my business is one.

OpenFaaS is a great alternative that we will surely test drive.

AWS now supporting containers on Lambda will hopefully make shipping browser binaries to Lambda also easier.

Take this benchmark as a grain of salt. Playwright > Puppeteer from technical point of view. Playwright is well maintained by some developer who started puppeteer and there are less bugs on playwright than on Puppeteer.

It may be faster but that depends on browser too right? Which browser version pinned to same? There are many other factors too.

Why playwright > Puppetter

1. Playwright supports cross browser properly. You don't usually see errors on firefox in playwright 2. Maintenance (Just watch recent commits, issues etc) 3. Less bugs on playwright

The scope of the benchmark is narrower than that, as stated in the article. It is not our aim to talk features, community, reliability (etc.) in this one article - that rather sounds like a book, and a long one at that.

I've put browser automation code examples in ava, cypress, jest, playwright, puppeteer, testcafe and webdriver here: https://github.com/umaar/learn-browser-testing/tree/master/1..., if ever you wish to compare them with each other.

I knew before even looking at these that Playwright and Puppeteer would be the best performers. However, they didn't really go into a lot of details about system configuration and browsers which can also have a significant impact on performance.

Author here - I'd be keen to go into further detail in a follow-up article if people are interested. Out of curiosity, what details would be most interesting for you about system and browser config?

I'd be curious for you to try performance of Puppeteer and Playwright running in a lightweight Alpine container using Chromium on Linux so that you get native support. Let me know if you need assistance.

I wonder how much difference the scripting language would have made to these benchmarks if any. OP if you are up to it, I would suggest using Python, Go with Chromedp and also rod[1] next time for same tests.

[1]https://github.com/go-rod/rod

The simple code examples on the websites of both Puppeteer [1] and Playwright [2] are impressive in their conciseness and power.

It's been a long time since I did browser automation but it was always much uglier than that.

These new browser drivers look (dare I say) downright fun to write stuff in. It's really tempting me to think of some browser automation side project to play with over a weekend...

[1] https://pptr.dev/ [2] https://playwright.dev/

No cypress?

If you include Cypress, you need a log scale to fit its awesomeness on the same chart as the others.

In the near future.

Why are all the selenium extensions so janky? Selenium and Katolan recorder are infuriating to use when they work and they break so often that I might as well code my tests by hand instead of try and fail to use the recording tools again and again...

The variation between puppeteer and playwright is surprising. I thought playwright was a fork of puppeteer with a little bit of added functionality mixed in.

That's a popular misconception. It is just the shape of the API that makes them look alike, other than that they are fundamentally different at this point. We've put so much effort into Playwright during the last year, probably 3x we put into Puppeteer itself!

There's obviously a place for browser testing. But I'm still not convinced most of the huge effort that goes into it couldn't have been better spent building stronger unit and integration tests elsewhere in the tech stack.

As long as there are buttons to click, people will want robots to click the same buttons.

It is surprising, though, how much engineering is required to click a button.

Reply on news.ycombinator.com