Hacker News Reader

Comment by mcv | Hacker News Reader

> "And when we look at what Cypress allows you to do (which is write many useful tests that have very little flake) then you wanna write more end-to-end tests."

Very little flake? That's not my experience with Cypress. The Cypress e2e tests on our build server sometimes fail for no discernible reason. Then you run them locally, and different tests fail, sometimes for understandable reasons (but then why didn't they fail on the server?) or sometimes also for no clear reason.

I'm spending a lot of time maintaining and fixing our Cypress tests, and the work they add seems far worse than the help they provide. Most of the time, a failing e2e test is not a problem with the code, but a problem with the test.

Honestly, I find myself longing for the days of Protractor. Horrible to set up perhaps, but in my memory of it, it was reliable once you got it working.

View on HN

I've found 2 reasons that e2e tests are flakey (whether it be selenium, puppeteer or cypress):

1. poorly developed tests. There are nuances that go into writing browser interaction tests that the less-experienced will have to work through.

2. There's a weird functionality implemented in such a way that makes testing difficult. This could be something like a modal that has a fly-in animation and if you try to click it while it's animated it won't fire the onClick action.

If you want e2e tests to be successful, you have to write testable front end code, which is pretty dang hard for a lot of reasons.

I’ve seen the same at several different companies. There seems to be just too much that can go wrong during an e2e test like this: network failures, slow CPU causing timeouts, etc. At one company, the entire suite of 700 tests was re-run 3 times nightly, taking 5+ hours, and some tests still failed randomly.

I have found that the majority of errors can be caught by unit and integration tests on a single service, and that the extra burden of maintaining Cypress just isn’t worth the small amount of coverage.

> The Cypress e2e tests on our build server sometimes fail for no discernible reason.

You'll probably be interested to hear that Cypress are... uhm... addressing this by adding "flaky test detection" and some amount of automatic retrying in v5.

We use Cypress at work, we have flaky test problems, but I'm not convinced much of it is actually Cypress's fault. As others have mentioned, writing front-end code to be testable by E2E is not necessarily straightforward, and when you're trying to add tests to a big existing codebase I think you kinda have to accept some flakiness and retrying.

What I've been wondering about recently though is: if you construct applications in a certain way do you still need real E2E tests? For example, React is declarative, and with a state-management system like Redux shouldn't you be able to get the same kind of assurance that E2E gets you without needing an actual browser? Maybe things like animations, visibility etc are where that comes unstuck?

As I remember it Protractor was just a wrapper around WebDriver with some glue code to wait for Angular doing its thing after data changes. If you're using e.g. React or anything else that does not do dirty checking, you should still be able to use WebDriver.

(That said, that's certainly still flaky enough.)

The cool part was that all of the extra JS Protractor used (sometimes for native WebDriver tasks) made things _less_ reliable.

Reply on news.ycombinator.com