Hacker News Reader

Comment by __MatrixMan__ | Hacker News Reader

I know it's unconventional, but I like to keep my (anything larger than unit) tests in a separate repo so that I'm not running old versions of the tests when I bisect:

- find bug

- write new test to expose it

- bisect commits, running new test each time

- find culprit commit

- stop bisecting, fix bug

- when the test passes, you know bug is fixed.

View on HN

How do you handle tests against new features that would definitely not pass on commits before the feature is introduced?

Some test frameworks allow you to mark these tests as xfail (expected fail). When you fix the issue, you can remove the xfail in the same commit.

This was my preferred approach for several years. The tests were in a repo separate from the code under test, and each declared its prerequisites, either as a feature or a version pattern.

If a test started failing, I could annotate it as an xfail with an open-ended version pattern. If I noticed that an xfail started passing, making it a upass (I think these were blue in our tests results), I put an upper bound on the xfail version pattern.

The nice thing about this was that I could always run the latest and greatest suite against any version, although it made it a little more complicated to run them as they existed when they were first run.

I'd typically start bisecting only after (or at the changeset) the feature has been introduced. At least with hg (and seemingly git as well) you can tell bisect to use a commit range where one end is good and the other is bad.

If I'm bisecting, I usually already have a single test in mind which exposes the bug I'm looking for, so I'm explicitly running that test only.

In general, I want to store one or more services and a first/last valid commit for each one as metadata for each test so that my framework knows which ones to run based on the environment under test. Haven't added this feature yet though.

My preferred variant of this would be

- write new tests to expose it, but not commit the tests

- bisect commits, running new tests each time

- find culprit commit

- stop bisecting, fix bug

- Look for similar bugs nearby, and ways to improve the new tests, now I know the root problem.

- Figure out the best series of commits for the tests and fixes together, that will be easiest to review by my colleagues/future me/alien archaeologists.

- Send PRs/commit to main.

Reply on news.ycombinator.com