Hacker News Reader

Comment by locknitpicker | Hacker News Reader

> The isolation comes from the test implementation not the framework. There isn’t any framework out there that can guarantee/give you isolation.

I'm not sure what you mean by that. All the frameworks I ever used were designed with test isolation as the primary design goal. Even when you set shared test fixtures and setup/teardown code, all they provide is a way to share code across tests, which are by themselves independent and isolated. BDD-style frameworks are the notable exception in breaking away from the pattern of having isolation as a fundamental design trait. In fact, the whole reason why BDD-style tools require special support from testing frameworks is that they need to work around test isolation in order to share context across steps.

View on HN

In JUnit, tests run sequentially in a single thread by default [1].

Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread.

Setup and teardown methods can be used to create and clean up test-specific state. However, developers must still ensure that test data is unique to the test so that concurrently running tests do not interfere with each other. This uniqueness is unfortunately called "isolated", and has led to much confusion like in this thread. Certainly, the Test Execution Framework cannot guarantee data-isolation.

Parallel and randomized test execution can also help expose application-side problems involving shared or order-dependent state. Such failures may only appear when tests happen to exercise the application under the relevant ordering or concurrency conditions. When I was a junior developer teaching myself Java Servlets in 2000, I had to learn this lesson the hard way. A Test Execution Framework would not be able to guarantee any "isolation" of test data and of workflows if the server-side state is mis-managed by the tech stack and/or by the developer.

[1] https://docs.junit.org/6.1.3/writing-tests/parallel-executio...

This

> Parallel execution must be explicitly enabled. When enabled, JUnit uses a fork-join thread pool, so tests may run concurrently on different worker threads. Because these threads are reused, a ThreadLocal value left behind by one test could be visible to a later test that happens to run on the same thread.

I don't understand your comment. JUnit docs describe parallel execution as an experimental feature that still has gotchas that you need to be mindful to avoid. So you have a test framework designed with isolation in mind for a specific execution mode, but when you go out of your way to try another execution mode that is described as experimental then you can stumble upon very corner cases where isolation is not ensured. What point did you wanted to convey?

I disagree so hard with the notion that isolation is somehow done for you, that I don't even know how to begin arguing.

I'll admit to starting this confusion by reading what I thought would be written, not what was actually there. TFA does phrase it to my satisfaction that the test is isolated. I assumed when reading it that Kent thought he had isolation but did not. By analogies: a naive programmer does not perceive the existence of race conditions; a naive web app developer doesn't perceive the existence of CAP; a naive backend engineer thinks they have full ACID guarantees when they're only guaranteed READ_COMMITTED. I thought I was reading another writer just wishing isolation into existence through ignorance.

(So I accidentally straw-manned Kent's argument, but maybe that's not such a problem, because it seems you're OK with the straw man version?)

Anyway, maybe a question moves things forward.

JUnit's been around for decades. Every developer has had multicore dev machines for decades. Why would parallel execution not be the default, or a simple toggle that everyone would turn on immediately?

All I mean that there is no way for a framework to prevent your tests from interfering with each other, or to solve isolation for you.

It’s the implementation of each test that is responsible for its isolation. Every time you write or make changes to your tests. You have to think, can they mess with each other.

> All I mean that there is no way for a framework to prevent your tests from interfering with each other, or to solve isolation for you.

What? By design that's exactly how virtually all test frameworks are designed to work.

> It's the implementation of each test that is responsible for its isolation. Every time you write or make changes to your tests. You have to think, can they mess with each other.

Nonsense. I've been writing unit tests for years in multiple programming languages and using multiple frameworks, and not once did I stumbled upon any scenario whatsoever where I had to be mindful about isolating tests. Because that was assured and a given from the very start.

I wonder what kind of convoluted mess you are doing with your tests.

You’ve never ever had to be mindful about isolation of tests? Also, Unit tests aren’t the only tests that exist.

You say assured? You think it’s impossible to write 2 tests that interact each other?

All you gotta do is google “test isolation”. But if you want to pretend it’s not a thing the I have no idea what to tell you?

Here’s one example since junit came up

https://martinfowler.com/bliki/JunitNewInstance.html

> You’ve never ever had to be mindful about isolation of tests? Also, Unit tests aren’t the only tests that exist.

No. Frameworks take care of it out of the box, and you need to go way out of your way to mess that up.

> Also, Unit tests aren’t the only tests that exist.

True, but that's immaterial to the discussion.

> All you gotta do is google “test isolation”.

Yes,and if you follow your own recommendations you will eventually discover that virtually all testing frameworks are explicitly designed with test isolation in mind, to the point that you need to go way out of your way to design custom test fixtures to share context across test runs.

> Here’s one example since junit came up

I seriously do not think you even read the article you are citing. Did you even bothered to open the link?

¯\_(ツ)_/¯

You can believe whatever you want to believe. Or maybe you live in a magical world where all your tests are pure and side effects nor IO exist.

In the other thread you think running tests in parallel doesn't count because it has documented gotchas...

Reply on news.ycombinator.com