Hacker News Reader

Comment by alkonaut | Hacker News Reader

Recently git bisect gained got a new capability that was mysteriously missing before: bisect —first-parent. With that, it became possible to bisect also in repos where all commits on feature branches may not be buildable or passing tests, but commits on the development/main/integration branch does.

A feature I’m still missing is to bisect using a new test. That is, I have discovered a bug in the program that there is no test for, so I author a failing test. Now I want to find out when this failure occurred but I can’t bisect because even the previous commit is green as the test didn’t exist. It’s possible to work around by placing the test outside the repo root and bisecting with a script that “authors” the test (e.g copies the file) at each step, but it’s cumbersome and feels like something that could be possible to add to the bisect command, e.g an option to merge a commit (the new test) before testing each step of the bisect.

View on HN

For the latter I use often use `git stash` for that; stashing off the test case/data, then at each step of the bisect, you can do `git stash apply` (rather than `git stash pop`) to reapply the same changes multiple times.

Before moving on, a quick `git reset —-hard` will get back to a clean flip

For ruby projects where I could simply put the "new test" in a new file I do this:

1. Create new test (e.g. "spec/reproduce-bug_spec.rb")

2. Ignore it locally: `echo "spec/reproduce_bug_spec.rb" >> .git/info/exclude`

3. Run bisect (something like: `git bisect run rspec spec/reproduce-bug_spec.rb`)

If running the test gets more complex (e.g. installing dependencies as they might change travelling through history), I usually create a wrapper script (and ignore it) to bisect-run-it.

Just put the test in a new untracked file?

That works but only for languages/build-systems where the mere presence of the test is enough to make it run. In others you might have xml project files, test lists etc which makes the addition of a test require a modification of existing files.

The test subcommand of bisect can do anything you want, so it's easy to make a wrapper that does that prep work before starting that actual tests. Just have to remember to pass back the tests' exit code after cleaning up the repository for git.

Good point, though I don't like the idea of a setup where your test file's only executed if you remember to do that personally (if the tooling doesn't already do similar, I'd at least want to generate the list you describe by globbing the relevantly named/located files).

You could just copy that entrypoint into another untracked file, have its list include (or be only - quicker) the new untracked test file, and execute that instead of the normal one while bisecting.

To bisect with a new test, you have to do this:

  $ git bisect [good|bad]

  $ git cherry-pick new-test

  $ ./run-the-test

  $ git reset --hard HEAD^

  $ git bisect [good|bad]

I sometimes use the stash for that: cherry pick the test as a working tree uncommitted change. "git stash" it before every bisect step, then "git pop".

Yeah, since it's more or less a standard cumbersome recipe, it seems it would be a good idea to create a high level git switch for it.

git bisect run --cherry-pick abc123 [....]

to do a regular bisect but at each step cherry picking the commit abc123

I have saved a little script to use with git bisect and a stashed test: https://gist.github.com/progers/998452a15180ae369fee338bb393...

Have you tried duplicating the test temporarily, which should turn the duplicate into an untracked file so you wouldn’t have to copy it at each step?

This might be a good argument for keeping tests as a separate repository. Which would bring a lot more problems of its own, though...

Reply on news.ycombinator.com