Detecting and Handling Flaky Tests in Playwright | Hacker News Reader

Detecting and Handling Flaky Tests in Playwright

ray.run

23 pointslucgagan16 comments

Something that makes your end-to-end tests a heck of a lot more resilient is mocking your API so you're not hitting the network, and so your API responses are deterministic. Doing that also makes the tests run a lot faster. MSW is my preferred library for doing that, and there's a Playwright integration (https://www.npmjs.com/package/playwright-msw).

Mocking the API makes it really easy to response with failures too - having a test that checks the app handles a 500 error or a 401 Unauthorized is really useful, and a total pain to do without mocks.

You can test the API is working and returning what you'd expect with a different set of tests. :)

If you mock out one end of the test, can it really be considered end-to-end anymore? Isn't it just UI testing now ?

We had this discussion at work too, and decided it was worth keeping our e2e tests e2e (no mocking, pointing the UI code at our integration environment). However, I think there's value in also having a set of tests that do mock the dependencies while testing UI flows because of how long it takes to run the real e2e tests. so "both" is the dream, with e2e tests being more critical.

can't you have both and just run the mocks if not specified else like in a ci or whatever

Yes you can. Our tests are layered as follows: - unit tests - mocking tests - integration tests (i.e. service + db) - system tests (full api to db level)

integration starts providing flaky tests already sadly, i.e. with kafka on our end… (i.e. calling the listener twice or thrice even tho it should only once..)

It's hard to name them I suppose. They're more than just UI tests because there's usually a decent amount of logic in the frontend apps I build (significantly more than just driving the UI), so I call them end-to-end or integration tests.

It's a hermetic end to end test.

The ideal is to mock out the API with some sort of recording mode to autoupdate the mock.

Do you have any example of this you can share? Keeping mocks up to date can be hell

do both!

It's still not loading for me (redirect loop) but archive.org has it already so I could still read it at https://web.archive.org/web/20230703090352/https://ray.run/b...

I really wanted to read this, but I think we may have hugged your site to death.

Its back online and just added Cloudflare in front of it

It seems to be doing a redirect loop

Currently in a redirect loop:

  > curl -D- https://ray.run/blog/detecting-and-handling-flaky-tests-in-playwright
  HTTP/1.1 200 Connection established

  HTTP/2 301
  date: Mon, 03 Jul 2023 10:43:44 GMT
  content-length: 0
  location: https://ray.run/blog/detecting-and-handling-flaky-tests-in-playwright
  ...

Looks like it is back for some users, but not others. Probably just the time it takes to propagate the DNS. What a bummer! Thank you for helping to troubhelshoot

Just got ERR_TOO_MANY_REDIRECTS

Reply on news.ycombinator.com