Comment by aidenn0 | Hacker News Reader

Comment by aidenn0 | Hacker News Reader

Parent Full thread

Right, they talk about fighting for a queue. Firstly, a good test-suite can be run (a configurable subset) on the developer's workstation. Secondly it needs to run on commits in a reasonable amount of time. This is just as true of E2E as of unit tests.

They also mention flaky tests. If there is a spectrum between unit tests that can run on a single function and e2e tests that need a complete system, the closer to e2e you get the more likely you are to have flaky tests.

Flaky tests are an indication of non-determinism either in your test or your system. If you have non-determinism in your system, then you can't confidently test it regardless of the flavor of tests you use. Non-determinism in your tests should be minimized; if you can take a random-seed as an explicit parameter, do so, so that you can reproduce the flaky failures. Test failures (flaky or not) are always indicative of a bug either in the test or in the system, and should be investigated as such. Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments.

Flaky tests are an indication of non-determinism either in your test or your system.

Yeah, my first though upon reading the article was: If their E2E tests produced non-deterministic results due to asynchrony, how can they have any confidence that their production data ever becomes 'eventually consistent'?

All end to end tests are non-deterministic due to asynchrony. At some point you have to trust the discrete states of your software.

I should be able to test that this usually works though, right?

You can test these things, sure. But if you're using other people's software (linux, vms, chromedriver, capybara) on other people's hardware (again, vms), you have to tolerate the fact that you can't control everything if you want to actually get work done. A little electrical, magnetic, or gravitational anomaly here, a little memory access blip there, some competition for cpu time elsewhere... I suspect there are probably only a handful of completely controlled environments on the planet and even those are suspect.

Test suites are sort of an eventual consistency problem themselves...

Right, but the key word here being “usually” - if I can’t just run the test three times and assume 2/3rds of the time it’s good, how can I know it usually works in production?

Is the right solution really to throw up your hands and not test end to end ever? I guess the argument is more convincing if it’s not that it’s impractical, it’s just too expensive relative to the returns.

You can if it's BSD-licensed.

If you use other people's software and hardware, and those things don't perform the way your software assumes they perform, knowing that would be useful, right? There's always a limit to how much you want to handle, but if you are having a test fail even a large fraction of 1% of the time, then there's probably some underlying behavior that you should account for in production as well.

No, that test doesn’t give you any useful information, because all it told you was that your expected answer wasn’t found in the configured time interval. You have no way of knowing whether or not your expected behavior would be satisfied if you ran for t + 1 seconds.

After some time you have to consider the test is failed and investigate, even if it would have succeeded had the timeout been 1 second larger. I cannot believe they do not have quality of service requirements. Testing those requirements is of course not easy. It may take to much time to run on every release or may be considered out of the scope of E2E tests and compliance is checked with telemetry results.

However pick any response time mandated by the QOS requirements, multiply by an appropriate x and use this as the pass/fail timeout for your test. Take a value large enough that can easily be considered a bug (because e.g. the customer would think the operation failed and would hit refresh or back). You then have an issue that is definitely worth investigating. You may actually have reproduced a rare issue that is part of the long tail of your telemetry.

Right have a timeout measured in minutes. The timeouts have zero effect on a clean run, so large timeouts have no effect on time to deploy if you require a clean run of tests for deploying.

This is what’s known as “counterintuition.” You would think that you could, but you are wrong.

I’m not saying you can’t write a passing end to end test. Of course you can get it to pass some times. But they are inherently non-deterministic.

I mean the exact output given a certain set of inputs may be slightly different due to asynchrony, but given a set of inputs, there should be a finite set of correct outputs and check for those.

To use a stupid example: if listAnimals returns [cat, dog, mouse] some of the time and [cat, mouse, dog] other times, if your passes on the former and not the latter, then your test is broken and you should fix it. If it sometimes returns [cat, dog, mouse, tree] then your system is broken and you should fix it.

A more accurate way to look at this based on your example is that, sometimes listAnimals returns [cat, dog, mouse], and sometimes it returns null.

It’s not that the result is nondeterministic, it’s that _whether or not the result is returned within the timeout of the polling mechanism_ is deterministic.

Presumably that happens in production as well, and the test can determine that the system does the proper thing when that happens?

> Flaky tests should be removed from the production testing system just like code that fails tests should be removed from production deployments.

...then how do you know when third-party upstream services are obeying their contracts to your service, if not by testing how your service interacts with those third-parties?

(I know my answer, but I'm curious to hear yours.)

I always mocked out 3rd party tests in my tests. I've never actually had a problem with some third party changing their API. That's the whole point of a versioned API anyway. I think when people talk about e2e tests, it's more about testing only integration between contracts that you own.

I’ve run into this a few times with some upstream package breaking and showing up in tests. I try to avoid mocking as much as possible in tests these days.

One thing I've done is adding the ability to run tests both with a "mocked" and a "real" version. The mocked version is fast and can be run quickly, the real version is much slower, but tests the actual real service. It's not that much extra effort to make in most cases, and I've caught some bugs when my mocked version made assumptions that were false, didn't cover some edge case, or whatnot.

That said, I too avoid mocks unless there's a specific good reason to add one.

I like the idea of testpoints in code that can be switched on or off, an idea originally from the hardware side. Modifying the testpoints to allow switching between different test implementations is a useful generalization of the idea.

It’s valid to say you’re e2e testing your system, just not e2e testing the “full system”.

This is why the classification of the test into e2e, integration, and unit can cause confusion. I like to try to encourage people to avoid bucketing and instead say “this test should be more integration style than it currently is”, “this test should be more isolated than it currently is”. At the end of the day all testing mocks out the user and things like old web browsers or other factors that are a part of the real world system you care about may not be simulated in your test, so the way to get ”real” e2e verification is probably monitoring real users, if you consider that the user is a part of your “system”

This system didn't rely on third party services, so "not applicable" I guess?

That's monitoring, not testing.

Of course, both have the same form, you run the system and verifies if the results match the expected. But monitoring is done constantly during the lifetime of your infrastructure, and verifies the entire infrastructure; while tests are done episodically, and verifies your program or a component. Tests also often block some procedures, while monitoring doesn't (but it certainly starts some).

You can try to monitor that an endpoint responds quickly but how do you monitor that it responds correctly? At the end of the day both tests and monitoring are forms of verification

Some people run (subsets) of their tests in production as a form of monitoring. Sometimes monitoring does not pass or fail and is instead qualitative like a dashboard or raw logging, without alerts

I’d say there is a grey area between monitoring and testing, it is more precise to ask if you’re verifying pre production, post production, or both

Generally, I think tests are used to validate changes to your service code (often as a gate to release it to production). Whereas monitoring is used to detect issues external to your code (often operated in production).

Edit: That is to say, what distinguishes testing from monitoring isn’t content, but purpose.

Monitoring can catch issues in the code. For example if an event is dead lettered or the application crashes unexpectedly, it triggers an alert, which may make you aware of some edge case you forgot to test. Both tests and monitoring can encompass validating code is running correctly, some even run their tests against production at regular intervals as a form of monitoring, for example see “datadog synthetic tests”, which could be characterized as both a test and a monitor. Many companies opting not to do traditional e2e tests actually still have them, they’re just running them against production instead of blocking CI (with the rationale they will prioritize fast detection and mitigation rather than trying to prevent bugs from entering prod)

Third-party services should be mocked for integration and end-to-end testing. Error conditions with respect to these services should be something that is monitored and alerted on when appropriate.

Our solution was two test suites.

End-to-end (which I will fight for being the highest value test site, and it's not close) had no external dependencies.

And a separate test suite that touched external services, split into two components: one that tested our integrations, typically against a remote testbed (if the 3rd party was competent enough to have such a thing), and a second chunk that attempted to see if remote api behavior had changed. Which it does with annoying regularity.

There are two things that can be tested here, not one: whether the upstream service conforms to the contract / API promise, and whether your code behaves correctly with respect to what the API promises.

So that gives you a number of options for testing the second one of those. Recording sample traffic and replaying it in the test suite is one approach. Actually running an instance of the service (if it's open-source - there's still value in paying someone to competently run an OSS service) in your test suite is another, as is running some clone of the service (e.g., if you're talking to S3, there are probably a hundred S3 API-compatible clones that are good enough to run in your test suite, even if, again, you are happy to pay Amazon to competently run production).

You also want to pay attention to the first one of those, but that's not a job for your test suite. That's the job for some balance between their test suite, your monitoring or production logging, and your business relationship with them.

There's basically no value in having tests against third-party code anyway, because all the test is going to do is tell you that they broke their interface. And by then, production is already broken.

I agree this is often the case but disagree it is always; “testing” against the api can be a canary for your new usage of their third party api not working the way you think it does.

I don’t know, I think having your tests assume that the 3rd party data looks a certain way is helpful. If that ever breaks in prod, then something needs to change, and your tests can change if the interface changes.

> Flaky tests are an indication of non-determinism either in your test or your system

Or in the system that runs your tests. That can itself be non-trivial.

Reply on news.ycombinator.com