Hacker News Reader

Comment by kodablah | Hacker News Reader

Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue.

Also it sounds like the solution is just a bit higher than limited integration testing which does have value of course. Sounds trite, but if you don't test end-to-end you aren't going to catch bugs that only appear end-to-end (which also happen to be the ones customers see making the e2e suite a decent place for high level regressions assuming you maintain test performance of course). This is especially true in environment-specific scenarios.

View on HN

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests.

They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e you get the more likely you are to have flaky tests.

Flaky tests are an indication of non-determinism either in your test or your system. If you have non-determinism in your system, then you can't confidently test it regardless of the flavor of tests you use. Non-determinism in your tests should be minimized; if you can take a random-seed as an explicit parameter, do so, so that you can reproduce the flaky failures. Test failures (flaky or not) are always indicative of a bug either in the test or in the system, and should be investigated as such. Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments.

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments.

...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties?

(I know my answer, but I'm curious to hear yours.)

I always mocked out 3rd party tests in my tests. I've never actually had a problem with some third party changing their API. That's the whole point of a versioned API anyway. I think when people talk about e2e tests, it's more about testing only integration between contracts that you own.

I’ve run into this a few times with some upstream package breaking and showing up in tests. I try to avoid mocking as much as possible in tests these days.

One thing I've done is adding the ability to run tests both with a "mocked" and a "real" version. The mocked version is fast and can be run quickly, the real version is much slower, but tests the actual real service. It's not that much extra effort to make in most cases, and I've caught some bugs when my mocked version made assumptions that were false, didn't cover some edge case, or whatnot.

That said, I too avoid mocks unless there's a specific good reason to add one.

I like the idea of testpoints in code that can be switched on or off, an idea originally from the hardware side. Modifying the testpoints to allow switching between different test implementations is a useful generalization of the idea.

It’s valid to say you’re e2e testing your system, just not e2e testing the “full system”.

This is why the classification of the test into e2e, integration, and unit can cause confusion. I like to try to encourage people to avoid bucketing and instead say “this test should be more integration style than it currently is”, “this test should be more isolated than it currently is”. At the end of the day all testing mocks out the user and things like old web browsers or other factors that are a part of the real world system you care about may not be simulated in your test, so the way to get ”real” e2e verification is probably monitoring real users, if you consider that the user is a part of your “system”

This system didn't rely on third party services, so "not applicable" I guess?

That's monitoring, not testing.

Of course, both have the same form, you run the system and verifies if the results match the expected. But monitoring is done constantly during the lifetime of your infrastructure, and verifies the entire infrastructure; while tests are done episodically, and verifies your program or a component. Tests also often block some procedures, while monitoring doesn't (but it certainly starts some).

You can try to monitor that an endpoint responds quickly but how do you monitor that it responds correctly? At the end of the day both tests and monitoring are forms of verification

Some people run (subsets) of their tests in production as a form of monitoring. Sometimes monitoring does not pass or fail and is instead qualitative like a dashboard or raw logging, without alerts

I’d say there is a grey area between monitoring and testing, it is more precise to ask if you’re verifying pre production, post production, or both

Generally, I think tests are used to validate changes to your service code (often as a gate to release it to production). Whereas monitoring is used to detect issues external to your code (often operated in production).

Edit: That is to say, what distinguishes testing from monitoring isn’t content, but purpose.

Monitoring can catch issues in the code. For example if an event is dead lettered or the application crashes unexpectedly, it triggers an alert, which may make you aware of some edge case you forgot to test. Both tests and monitoring can encompass validating code is running correctly, some even run their tests against production at regular intervals as a form of monitoring, for example see “datadog synthetic tests”, which could be characterized as both a test and a monitor. Many companies opting not to do traditional e2e tests actually still have them, they’re just running them against production instead of blocking CI (with the rationale they will prioritize fast detection and mitigation rather than trying to prevent bugs from entering prod)

Third-party services should be mocked for integration and end-to-end testing. Error conditions with respect to these services should be something that is monitored and alerted on when appropriate.

Our solution was two test suites.

End-to-end (which I will fight for being the highest value test site, and it's not close) had no external dependencies.

And a separate test suite that touched external services, split into two components: one that tested our integrations, typically against a remote testbed (if the 3rd party was competent enough to have such a thing), and a second chunk that attempted to see if remote api behavior had changed. Which it does with annoying regularity.

There are two things that can be tested here, not one: whether the upstream service conforms to the contract / API promise, and whether your code behaves correctly with respect to what the API promises.

So that gives you a number of options for testing the second one of those. Recording sample traffic and replaying it in the test suite is one approach. Actually running an instance of the service (if it's open-source - there's still value in paying someone to competently run an OSS service) in your test suite is another, as is running some clone of the service (e.g., if you're talking to S3, there are probably a hundred S3 API-compatible clones that are good enough to run in your test suite, even if, again, you are happy to pay Amazon to competently run production).

You also want to pay attention to the first one of those, but that's not a job for your test suite. That's the job for some balance between their test suite, your monitoring or production logging, and your business relationship with them.

There's basically no value in having tests against third-party code anyway, because all the test is going to do is tell you that they broke their interface. And by then, production is already broken.

I agree this is often the case but disagree it is always; “testing” against the api can be a canary for your new usage of their third party api not working the way you think it does.

I don’t know, I think having your tests assume that the 3rd party data looks a certain way is helpful. If that ever breaks in prod, then something needs to change, and your tests can change if the interface changes.

Flaky tests are an indication of non-determinism either in your test or your system.

Yeah, my first though upon reading the article was: If their E2E tests produced non-deterministic results due to asynchrony, how can they have any confidence that their production data ever becomes 'eventually consistent'?

All end to end tests are non-deterministic due to asynchrony. At some point you have to trust the discrete states of your software.

I should be able to test that this usually works though, right?

You can test these things, sure. But if you're using other people's software (linux, vms, chromedriver, capybara) on other people's hardware (again, vms), you have to tolerate the fact that you can't control everything if you want to actually get work done. A little electrical, magnetic, or gravitational anomaly here, a little memory access blip there, some competition for cpu time elsewhere... I suspect there are probably only a handful of completely controlled environments on the planet and even those are suspect.

Test suites are sort of an eventual consistency problem themselves...

Right, but the key word here being “usually” - if I can’t just run the test three times and assume 2/3rds of the time it’s good, how can I know it usually works in production?

Is the right solution really to throw up your hands and not test end to end ever? I guess the argument is more convincing if it’s not that it’s impractical, it’s just too expensive relative to the returns.

You can if it's BSD-licensed.

If you use other people's software and hardware, and those things don't perform the way your software assumes they perform, knowing that would be useful, right? There's always a limit to how much you want to handle, but if you are having a test fail even a large fraction of 1% of the time, then there's probably some underlying behavior that you should account for in production as well.

No, that test doesn’t give you any useful information, because all it told you was that your expected answer wasn’t found in the configured time interval. You have no way of knowing whether or not your expected behavior would be satisfied if you ran for t + 1 seconds.

After some time you have to consider the test is failed and investigate, even if it would have succeeded had the timeout been 1 second larger. I cannot believe they do not have quality of service requirements. Testing those requirements is of course not easy. It may take to much time to run on every release or may be considered out of the scope of E2E tests and compliance is checked with telemetry results.

However pick any response time mandated by the QOS requirements, multiply by an appropriate x and use this as the pass/fail timeout for your test. Take a value large enough that can easily be considered a bug (because e.g. the customer would think the operation failed and would hit refresh or back). You then have an issue that is definitely worth investigating. You may actually have reproduced a rare issue that is part of the long tail of your telemetry.

Right have a timeout measured in minutes. The timeouts have zero effect on a clean run, so large timeouts have no effect on time to deploy if you require a clean run of tests for deploying.

This is what’s known as “counterintuition.” You would think that you could, but you are wrong.

I’m not saying you can’t write a passing end to end test. Of course you can get it to pass some times. But they are inherently non-deterministic.

I mean the exact output given a certain set of inputs may be slightly different due to asynchrony, but given a set of inputs, there should be a finite set of correct outputs and check for those.

To use a stupid example: if listAnimals returns [cat, dog, mouse] some of the time and [cat, mouse, dog] other times, if your passes on the former and not the latter, then your test is broken and you should fix it. If it sometimes returns [cat, dog, mouse, tree] then your system is broken and you should fix it.

A more accurate way to look at this based on your example is that, sometimes listAnimals returns [cat, dog, mouse], and sometimes it returns null.

It’s not that the result is nondeterministic, it’s that _whether or not the result is returned within the timeout of the polling mechanism_ is deterministic.

Presumably that happens in production as well, and the test can determine that the system does the proper thing when that happens?

> Flaky tests are an indication of non-determinism either in your test or your system

Or in the system that runs your tests. That can itself be non-trivial.

> Sounds like this specific e2e suite was poorly optimized and was killed instead of rewritten/optimized due to a perceived notion that inefficiences are inherent in all e2e suites. If you maintain speed and strict curation of such a suite, most of the bullet points against are not an issue.

At least for web applications, all end to end test suites are slow and flaky. This is not an exaggeration - all of them. There are no magical optimizations. This is something that every project runs into, over and over again.

I will never willingly write an end to end test ever again. Unit / module tests + targeted integration tests are the only hope that we have.

> I will never willingly write an end to end test ever again. Unit / module tests + targeted integration tests are the only hope that we have.

What are your "integration" tests that are not "end to end" tests like, how do they differ from end to end tests?

Integration tests may integrate smaller-than-the-whole groups of subsystems. It definitely gets fuzzy. A lot of people treat end-to-end and integration tests as equivalent, but piecing together everything-but-the-frontend and testing it is also an integration test, but not an end-to-end test.

If we consider tests as existing at and covering different scales, unit tests are at the smallest scale and integration tests run the gamut from 2 units to the entire system.

Not OP, but testing groups of the "units" you unit tests can have a lot of value.

Picture a test that doesn’t involve any clicking of UI elements. That’s a start. So right there you’re avoiding the complexities of UI rendering, you just call the commands that are invoked by clicking directly.

Also, you can test frontend components together, and backend components together, but not cross the client-server boundary. Faster, more reliable.

That leaves a very small amount of e2e tests that you even want to write, and by that point I’m totally fine with manual smoke testing or automating them. But they’re the vast minority of tests.

Sounds like you’ve been subject to some pretty poor test setups. I’ve experienced good ones. My cynical take is that well maintained e2e tests aren’t a product priority in environments where they’re flaky and slow so they come as an afterthought. Not that they can’t be good. Usually product wants to ship code yesterday and doesn't care if there are bugs… so good test hygiene is nowhere to be seen.

This argument does not account for the fact that all e2e tests are non-deterministic, so the quality of your “setup” is not relevant.

You must have worked on some incredibly bad software to have non-determinism dominate your life. If a request fails then retry it just like a user would. If it keeps failing there's a problem. If it works then move on. I doubt your bank just throws in the towel and says "whelp software systems are inherently non-deterministic so we'll just forget some transactions here, allow the wrong amount over there, forget tests they're hard we can handle a little chance in our payment flows". The closest thing I've heard to that is amazon very occasionally shipping multiples of the same item because it was allegedly more expensive to implement immediate consistency than to ship a few duplicate items.

I never had any problems writing reliable end-to-end tests. They are super useful for catching serious subtle bugs before the system goes into production. Not having solid end-to-end tests is a massive red flag for me.

I flat out don’t believe you.

Well it is true. The fact that you don’t believe it tells me you have a lot to learn. Writing good end-to-end tests is a skill you need to learn. Don’t assume that software developers can do it without proper training/learning. It is hard to do well.

My disbelief of you is from my own experiences writing them, and talking to dozens of colleagues across different companies.

Every single company has to heavily parallelize their e2e tests, and pays a huge CI bill on top of effort maintaining an overly complex CI config.

Even after this, every company has to have a retry mechanism for their e2e tests because at least one fails at least every test run. It’s also the first thing I ask on interviews. I have many dat points on this one, across many teams and companies.

It’s an abomination.

Maybe you’re not talking about interactive web applications. Or maybe you’re not talking about the scale where you have thousands of e2e tests written over multi-year projects. If you’re talking about a toy project, sure you might not run into nondeterminism. These problems present themselves in aggregate.

I am talking about very large scale systems with web + mobile + desktop clients in an Enterprise environments. Think airlines and airports (operations and pairing/rostering). However it sounds as if your experience has been pretty bad. So I understand where you are coming from. My experience is that yes it is hard to do right but also that it is worth doing. YMMV of course.

I've never had any issues with the E2E tests in my company either, and I'm unsure where the flakyness would even come from. Using Cypress with great success.

+1 on this comment. We had issues with poorly written selenium tests and after rewriting new tests with cypress and better test practices, the e2e test are reliable enough to be used as canary testing in new environments without false positive flaking. It ultimately comes down to how much you're willing to invest in writing good tests.

If you're suffering from seriously flakey e2e tests results, more often than not it's from either outdated tech, poor testing practices, and not enabling a selective retry on failures.

You hit the nail on the head, using your own words.

> selective retry on failures

This is why your test suite passes, not because of an avoidance of outdated tech or poor practices. You rerun your flaky tests until they pass. That’s bad engineering, and the definition of non-determinism.

> At least for web applications, all end to end test suites are slow and flaky.

Then you should be asking why are they flaky in the test environment? Its probably because your services are running on very slow servers.

No, it’s because end to end tests are non-deterministic with respect to execution time. They are deterministic only in their discrete states.

This is why all end to end testing involves polling to wait for asynchronous operations to complete, which is by definition non-deterministic.

No not really, what you are saying is that you dont how long a test will take to complete, it can take 1 min or it can take 1 hour. If it sometimes takes 1 hour then you have to put on your detective hat and go look in the logs to see which service is slowing the e2e flow.

The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them.

It’s much easier to build a testing pyramid from the bottom up. The skills maturity comes from the bottom of the pyramid, not the top, and thinking about the end game stunts your growth.

Often E2E tests have such sunk costs involved that they materially affect the project roadmap.

> Often E2E tests have such sunk costs involved that they materially affect the project roadmap.

I've seen this first hand. When the e2e tests take hours to run, are flaky on a good day, and are only really understood by one or two people on the whole team, they can be a major roadblock to new features or even just moderate refactors.

That definitely happens. The E2E tests tend to make assumptions about how the app works (encode not just the requirements but also the architecture) and some features change the design. In order to add this feature we have to fix dozens of other tests. I’ve seen people on multiple projects team up to fix these, take over a day working together, and still not be done. They always try to tweak the tests but the test assumptions fight them.

Meanwhile if we add a feature that invalidates a unit test, you just delete the unit test and start over. Unit tests are cattle, E2E tests are pets.

I should add as well: after that day, day and a half working together on old tests, those engineers look beaten down. They are not having a good time. It’s miserable work.

It must be some sort of Stockholm syndrome that people in this state still defend the tests. Even after they’ve invested more time and energy into fixing them than we ever would just manually testing that part of the code in perpetuity.

While I agree that writing unit tests are a lot harder, and you develop good skills in attempting to write them, I must say that in the projects I worked on most bugs were caught by integration tests (technically not E2E tests), and not unit tests.

I've also had projects with only unit tests, and almost no bugs were found by it, and there were plenty of bugs.

Ideally, I would like both. But if I had to have only one, I'd go with tests at a coarser granularity than unit tests.

My experience is that most bugs are found by randomisers / fuzzers. I’m consistently surprised they aren’t used more often, because they’re insanely good value for the time spent writing them.

Eg, a b-tree has a bunch of invariants: Leaves have equal height, data is sorted, nodes have between N/2 and N values, they contain everything that was inserted and not deleted, etc. So write a test which makes random changes to a b-tree in a loop, and makes those same changes to a simple sorted list. Every iteration, verify the invariants hold and values match. Every 1000 iterations, throw out the object and start again with a new seed. If the test ever fails, print out the seed for easy reproducibility.

In your unit testing suite, run this fuzzer for about 100ms or something. This catches lots of bugs. And occasionally leave the randomiser running overnight looking for rare bugs.

This sort of thing is so humbling, for the sheer volume of “obvious” bugs you find in otherwise working code. It’s hands down the best value testing code I’ve ever written.

> The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests. The E2E are a magical security blanket that covers over all of the mistakes you’ve made leading up to them.

Either alone is insufficient. Both together aren't necessarily sufficient.

> The main problem I see over and over with E2E tests is that they keep people from getting good at unit tests.

I'd view that as a win then. Unit tests are next to worthless for anything but tightly bound domains, like libraries, in most cases.

They're actively harmful in things like application-level service code, where they're used to turn perfectly well-written code into a chockablock mess to satisfy "testibility".

Unit tests are worthless. End-to-end tests are 100% needed. You need tests that cover all use cases end-to-end including testing error cases. That’s the absolute minimum I would expect from a well-engineered system.

You're both kind of right. You need to E2E your product, and what your product is is what matters.

You can’t test all error cases end to end. If you can you have shitty error handling.

Clock skew between servers? Drifting clock skew? Disk space exhaustion? Disk space exhaustion at each possible failure point? There are so many of these and you’re going to inject most of them in unit tests.

My original point was that if you can’t write good unit tests your e2e tests are also going to be lousy, and you will never get good at either, let alone both, if you fixate on more coverage with E2E tests.

They’re also just too damned expensive even if they were qualitatively as good. Which they are not. They are less numerous, sure, but that’s false economy because they are usually 3 orders of magnitude slower.

Of course you can. Part of the end-to-end test is to setup the test scenarios you want to test (including limited HD space etc.)

You're going to spin up a vm with the wrong system time, and then advance it between two operations that take 200 ms on a live system?

Bullshit.

What a BS straw man answer. There are much easier ways to do that kind of testing.

They are only slow if you do it wrong.

I'm not sure that the idea of e2e being relatively inefficient is just "perceived".

E2E tests in all orgs I worked at have always been the slowest and flakiest part, especially when simulating UI work and when working with systems that go beyond a handful of services.

I have seen efficient e2e suites, often built by and having a BDFL who had the same experiences as you. They have enforced best practices like "no sleeps", "no time-based tests", "every test must be concurrent and isolated", "refactor liberally", "bootstrap/share expensively allocated resources", etc.

I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are considered second-class pieces of software and only get the attention of QA engineers or developers who are not applying the same level of effort as their runtime code.

> I don't know how to say it humbly, but the biggest problem I've witnessed in slow e2e suites is that they are considered second-class pieces of software and only get the attention of QA engineers or developers who are not applying the same level of effort as their runtime code.

I replied in two other places on this thread before seeing this comment. It's very true. Since tests don't get shipped to customers, tests don't get the same level of effort. But when your tests are known to be of poor quality, people stop trusting them, and when people don't trust the tests, they stop adding any value.

It's a fair point, which shows the underlying problem with e2e: if you don't have a BDFL who's willing to fight on this hill, the system will eventually break down. This implies a huge amount of constant friction that I don't believe is sustainable over the long term.

Most engineering organizations don't have "excellent" leadership, and so most orgs are well served by having team dynamics such that they don't depend on that. A bunch of additional integration tests and a bit of formalization of the difficult parts that e2e tests (the sort of async message-passing stuff that has unpredictable bounds) seems like a far better alternative for most orgs.

I have a friend that works on a team whose whole job is writing e2e tests. Before them the tests were slow, buggy, and couldn't be ran in parallel. Now they can be ran in parallel and there are few-to-no false positives.

There's still challenges with this model (such as tracking changes on other teams, helping ensure that UIs are testable), but it seems to have worked out much better for their company than expecting every developer to write and maintain them.

Yeah but what's the point then, in that case you can just take back the old QA team and delete the gazillion lines of e2e test code and save yourself the liability of all that complexity. If it's cheap and simple to make a manual test, why replace that with something that complex, expensive and hard?

> If it's cheap and simple to make a manual test, why replace that with something that complex, expensive and hard?

Because you don't want to make _a_ manual test, you want to make _hundreds_ of tests.

If you are a human you have judgement and can determine which tests are the most critical and most relevant, so you don't have to always execute all of them. "It's just one line of css change to fix the styling, ok deploy". Second, a team of QA can very well make hundreds of tests in a day. And more importantly, they can really easily make decisions and draw conclusions such as "it's a bit slow sometimes, but overall acceptable", or "the animation is displayed correctly" or "there was a glitch in the rendering, but it's fine now", or "it works but the stying has moved slightly off center" etc etc, which take test expert programmers forever to try, and fail, to create deterministic automated tests for.

> is that they are considered second-class pieces of software and only get the attention of QA engineers or developers who are not applying the same level of effort as their runtime code.

Another way to say this is that efficient e2e tests require significant continuous investment in top-tier engineer time. The question then is how much engineering time is worth being spent in that way.

It may be that, yes, you can have fast e2e suites, but doing so is too expensive to justify the cost.

You have to compare that against what was done instead. Their solution was to employ a few engineers to create a new contract-based test framework, which will also have to be maintained. I believe that counts as "significant" investment too, but the calculus has to be whether that is less costly than improving their E2E tests.

> The question then is how much engineering time is worth being spent in that way.

Well, since it brings more value than testing in a lower level, I would say, more than any other kind of test (except, maybe, for monitoring).

Another good question is, is there any kind of tests that gives you good results without investing good-engineers time? If you find any, I'd ask you to share (but I would understand if you consider the information a market differentiator and won't).

I have generally found that integration tests with well-mocked external dependencies achieve 80% of the things E2E tests do with a quarter of the effort.

How do you run tests in parallel if part of the logic you are testing is a sql statement?

Do you just test them separately? For example, mock out the db when testing the app and then sequentially test the db to make sure the sql statement works as expected. However, this explicitly doesn't test the integration.

As one example, Django handles test parallelism by creating N test databases (on the single test database server) and dividing tests into N runners. https://docs.djangoproject.com/en/3.2/ref/django-admin/#envv...

You could also have multiple Docker containers running DBs.

Thank you! That's so crazy!

Not so crazy, it's very feasible to roll it yourself! Postgres has a "copy database" feature that's very useful (`CREATE DATABASE xxx WITH TEMPLATE yyy`).

I saw a project on HN a while ago focused on "managing isolated PostgreSQL databases for your integration tests", never used it but looks like a good idea: https://github.com/allaboutapps/integresql

In addition to the database pool approach, you can also write tests so that they are inherently independent. Each test creates and (optionally) deletes its own data, without making assumption about what else is in the database. That's not ideal, as it's hard to know you're not making a hidden assumption.

One option I’ve used only works if there’s some natural partition of the data like a customer ID. Every test starts by creating a new customer account. Since by design customers can’t see each other’s data, therefore tests can’t interfere with each other and can run in parallel on a single database. After all, in production all your customers are going to be using the database at the same time right? So it needs to work anyway.

Cool. I think I can use this strategy when I add communities.

Here our tests are written in BDD (behavior-driven development) style, mimicking user actions and data expectations. During development, these are run against mocks (either in-memory DB or a mock repository). Individual small scenarios are also combined into realistic long-running processes, for example cases from opened to closed taking various paths.

The suite runs in parallel fast and frequently alongside unit tests. Then occasionally like before PR merges, the same scenarios are run against a clone of the production environment to catch any mismatch with the run-time environment, also in parallel (connections) to simulate multi-user usage. Any technical issue prompts an improvement of the mocks and rarely resurfaces.

Running these as scripts also doubles as a fake data generator to play with for manual testing, reporting, etc. Last we proceed with some manual testing to validate new changes and pick-up UI-related issues - we don't do UI automation.

Your comment matches my experience very well. I had the same experience as GP and OP with low-quality e2e tests at my job. I got fed up four years ago, started something new from scratch, and now I'm the BDFL you mentioned, for a bunch of teams working in a common testing framework.

The main thing is indeed enforcing high quality standards even when individual engineers aren't very invested. You've identified some good practices right in your post, but it can take some time for people to learn these principles. And they can be reluctant if they see it as a waste of time. "These are just tests, I need to do my real work!"

For me, the crucial thing here is to avoid building things that are just for testing. If you tell someone that sleeping here is not good enough, and they need to build something more elaborate - then it's much more compelling if you can figure out how to build that so it's not just useful for a test, but also useful in production. This can be things like more flexible configurations, recovery tools for emergencies, new monitoring scripts and systems... all kinds of stuff.

If you stay focused on building things that are flexible enough to be used for both testing and production, then your life gets harder in some ways, but you can be much more strict about requiring high-quality work.

(btw, I'm hiring for the team building this infrastructure: http://catern.com/tsint_job.html )

I'm someone who had to build some mocked services to do end-to-end testing (well, as much as we can). The stuff I work on involves making two DNS requests (to different providers) and a possible HTTP request (for notification) and these three end-points are not under our control (as far as the department I work in are concerned). The two DNS requests are made concurrently [1] and management wanted to test the following scenarios:

* A returns, then B;

* B returns, then A;

* A returns, B returns late [2];

* B returns, A returns late;

* A returns, B never returns;

* B returns, A never returns.

I had to implement a side channel from the testing program to the mocked DNS servers (because a program like bind is just overkill for this---seriously) to implement artificial delays in the responses. Kind of hard to justify that for a production server (and yes, there is an active bug where B returns but A doesn't and the wrong information is returned, but it happens so rarely in production [3] that it was deemed acceptable for now).

The other component, the notification via HTTP, required ensuring that a notification that wasn't supposed to happen, didn't happen. [4] Again, I had to implement a mock with a side channel to the testing program to inform if it was to expect a request or not, and then report after all the tests were run how many requests were actually made. If the value between the testing program and the mock didn't match, it's an error. Oh, it's also useful to inform the mock what HTTP status code to return for the test. Such fun.

Management doesn't seem to think these mocks are a waste of time, but it seems like you might.

[1] At least for now. In the past, there were cases were we were to only contact A; some cases where we contact B, then maybe A; and some cases where we contacted both. This was done to save money at the time because all queries cost us money.

[2] we have some real time constraints on handling queries from our customers, the Oligarchic Cell Phone Companies.

[3] Excessive KPI logging for the win here.

[4] Proving a negative---lovely. Thanks, management!

Same experience, I'm sure you can design a large e2e test which isn't slow and flaky but for that you need some very very strict set of rules & care. I've personally never experienced one like this though.

Given that E2E tests should run in an environment that is more controlled than production, if you can't get an e2e test to perform reliably then it's a strong indication that your system won't perform reliably in production.

If an e2e test is not performing reliably not because it can't, but because the test is half-assed, then that needs to be treated as a bug in the test, and the test should not be used to assess the quality of your software. Developers (including me!) have a natural tendency to treat bugs in tests as lesser than bugs in the product, but given that bugs in tests will mask bugs in the product, this is a problem.

True story: a new e2e test was failing randomly. For 6 months nobody looked at it because "it was just a flaky test." A manager found out and insisted that someone fix the test, and it turned up the test was fine, it just found a (non-deterministic) bug that had been in the product for over a decade.

> Given that E2E tests should run in an environment that is more controlled than production, if you can't get an e2e test to perform reliably then it's a strong indication that your system won't perform reliably in production.

"Reliably" isn't a binary indicator, but a spectrum of how frequently certain classes of bugs may appear in a system.

In the example that you were mentioning, it would appear that the amount of effort needed to maintain the e2e test suite was simply not worth it. How many man-hours were spent by your manager and staff ignoring the test suite? How critical was the bug (it would appear not much)? How much effort would have to be dedicated to get the e2e suite working will that won't be spent doing other classes of tests or feature development?

I'm not saying a well-maintained e2e suite doesn't work well or help to catch a lot of interesting production bugs. But I am saying that I think that for the vast majority of systems it's just not a good use of your time. Save your efforts and put more thought into the system design to avoid certain theoretical classes of errors and devote the rest of your time to better integration tests and that will likely serve more orgs better.

> How many man-hours were spent by your manager and staff ignoring the test suite?

I'm not sure what you mean by that. Ignoring the test-suite isn't something that you spend time doing. It was 6ish weeks with a team of 8ish people, so you could say "48 man-weeks" were spent ignoring it, but they were doing other things in that time, not just sitting at their desk proclaiming "I'm ignoring this test."

Once the manager forced someone to fix the test it took less than one man-day to find the bug, and about 5 minutes to fix the bug once it was found.

Yes, this is true in my experience as well.

They are, however, extremely useful when they aren't flaky. A well-built E2E test can be a huge timesaver when debugging interactions between components.

Of course they're the slowest and flakiest - they include the most sources of slowness and flakiness that you have. But, if you want to make your production fast and reliable, they're pretty good gauges if you're headed for success.

I have a Rule of 8 for the testing pyramid that has been roughly stable across three programming languages.

Each level you crawl up the testing pyramid increases run time for good tests by a factor of 6-10. If your functional tests are taking more than 10 times as long as your unit tests there is something wrong that is worth investigating. Usually I set a default “slow” time equal to multiples of 8 over a good unit test and round off to a whole number to invite fewer questions.

But it also means that if your unit tests are running in 10ms apiece, your integration tests should run in about 640ms and your E2E tests in under 5 seconds. Getting most people to make 3 second end to end tests is at least as hard as getting them to push them down the stack.

You need more tests as you go down, but that generally takes about a 5:1 ratio, meaning you still get a 30% improvement in run time for every test you can push down, and sometimes we are using end to end tests to do unit test work, which is going to be 4 to 500 times faster depending in how many cases were really missing in the unit tests.

Yeah I agree with you, and it's because writing a test suite to simulate the users and verify (all) the use cases of your system, is really complex. I think the test advocates make it way too easy for themselves when they always just say "the first rule of testing is that your tests should always be fast" See, you broke the first rule, that's your problem! Well.. how do you execute a large number of complex operations and verifications, quickly? There's a how lot of actual practical solutions missing here, and just a lot of obstinate principle belittling "rules" and deflecting from providing an actual solution, which in practice, is hard.

One of the things that I love about Bazel is it thinks of a binary that obeys a contract as a test. This means you can have things like `sh_test` which just runs a shell script in a sandbox and gives you all of the benefits Bazel has normally for test execution. You get automated caching, parallelization, and remote execution of tests for free.

A great talk about this: https://www.youtube.com/watch?v=muvU1DYrY0w

You can often get situations where integration tests (that cover large features) take less than 30 seconds, only ever execute your tests when it is possible for the outcome to change (a dep has changed), and you can run your tests on a fleet of machines rather than one laptop or CI runner.

> If you maintain speed and strict curation of such a suite

This seems to be the hard part. Any tips for maintaining speed and strict curation, especially at scale (in terms of developers)?

At the very least, it seems that E2E tests are a tool that's easy to misuse. Not sure of the best way to mistake-proof it.

Automatic linter completely banning sleep(). Present reasonable alternative functions instead, like untilServiceAvailable.

Sounds silly and obvious but look into your test suite and i'm sure you will find sleep everywhere.

Majority of all flaky tests are due to miscalculated sleeps in my experience. Sleep is also the biggest contributor to slow runs, with longer and longer sleeps being used to combat the flakiness. Both factors being the biggest pain points of E2E tests.

I've been thinking about the benefits of only writing E2E smoke tests which cover a small number of critical paths quickly. Seems like most of their problems came about because they wrote more tests than they needed, with higher coverage than was necessary.

If you are only allowed to have a single test in your project, it would have to be the E2E smoke. I've seen systems where master don't even start up for weeks but project is still proud to present unit tests are green with 100% coverage. One single E2E smoke outweighs all those tests, catching everything from faulty configuration, infrastructure, interface assumptions, integrations, libraries and code bugs. After this it becomes more of a runtime vs coverage balance.

What you need to be aware of is that even one single E2E test can require a significant investment in bootstrapping your test-environment. If you are clever you will use this to improve the quality of your production code as well. For example, if you only have a single DB for production and now need to spin up an new instance for test, don't do it manually, instead refactor it into infrastructure-as-code and you've now turned this component into cattle instead of sheep, allowing you to scale both prod and test and giving your ops team a much easier life.

this.. I have been doing that in products from hardware to ai models. It is effective and gives reasonable feedback.

I even gave a baiting tech talk at one of the companies about how useless unit test really are. Useless is understating it, they are often counter productive. Especially "junior" people, who try and grind the testing mindset, will come and write a little test for everything. Good luck making small inconsequential changes in the future.

If you have well written unit test (high level apis) then they become worthwhile - but once you arrive there, you have to just do one little step further to move into integration test and get good cover from failures/changes/bugs in cloud/infra provider.

> if you don't test end-to-end you aren't going to catch bugs that only appear end-to-end

Points 2, 4 and 7 from the assessment expose why sometimes this is not achieved even in E2E tests.

Well of course e2e suites are not a panacea. That doesn't support killing them. With regards to flaky or hard to debug tests, those are implementation-specific issues that should not be used dispel the entire concept of e2e testing (and can usually be solved by high code-quality tolerances and tracing respectively).

> That doesn't support killing them.

If you have to wait hours, sometimes days for a queue to run tests that catch 1 bug in 1000 runs and you still end up with bugs in production, I believe this supports killing the current e2e process, if you find ways to guarantee system integrity between services.

I think you are assuming accidental complexity, but hard to debug tests could also be a symptom of inherent system complexity.

Nubank is a gigantic proponent of clojure and is regarded to have high standards of code quality, so I think that we can give them the benefit of the doubt in this aspect.

The entire article is about the reasons they ditched the test suite and replaced it with a different practice. Does it need to be more specific about the tradeoffs between fixing/rewriting the e2e suite vs. doing something different?

Should a BMW test-driver take a car out on the test track, when a engineer/designer is tweaking the glove compartiment handle ?

BMW still does crash testing on finished products (end-to-end tests)... which would cover glove compartment too and how it affects overall safety... (perhaps it breaks up into pointy objects on crash, maybe it opens randomly during driving causing safety issues...)

You wouldn't build a car without doing test drives at the end... or crash testing... or certifications.

Yes. The latch might not be strong enough to handle the centrifugal force when driving hard, or vibrations, etc.

You don't need to go out to the track once per tweak of course. You could very well do a few laps to test out the whole system once in a while.

I’ve personally seen way more e2e regressions than isolated regressions (mobile dev). It seems to make sense from a high level: it’s easy to test finite/internal behavior (unit test, UI test, or manually), but there are exponentially more cases when integrating any bit of code with any other bit of code.

Agree. Other organisations can effective run very large end-to-end tests. So it is most definitely possible. If they don’t have the in-house skills to do it then they should hire somebody who can.

What’s incorrect about my comment? Clearly other organisations can write successful end-to-end tests. The fact that you can’t do it doesn’t mean that nobody can.

Reply on news.ycombinator.com