Hacker News Reader

I've successfully used "git bisect" in situations in which, as part of every test, I had to "git stash pop" some local changes, then do the test, then remember to "git stash save" them before continuing to bisect. (Alternatively, "git apply" the stash, then "git reset --hard" to blow away the changes.) That's how you can handle situations where test code has to be added to the program to demonstrate the problem, and that test code doesn't exist in old versions.

Yeah, not a bad method. And I suppose if I've already commited my unit test, I could use a "git reset --soft" to put the test in a stash. Next time I use bisect, I'll try to do this. Thanks

The whole idea of bisection is that you have an objective behavior that has changed. If the test (which is, by definition, the tool used to measure that objective behavior) is not constant across the range of changes, then you simply don't have a problem that can be solved by bisection. You're breaking rule one ("This bug" isn't a single behavior).

Obviously in practice you'd probably just use whatever the most recent version of the test is.

Really I think you're problem is that you're overspecifying rule 2. It's not about identifying a "unit test" per se, it's about identifying any specific process to detect the bug. If the tests in your tree work, great. If not, you may have to do a little development to make one that is automatable.

I think you misunderstood. The test is brand new. As in, you find a problem that nobody has identified before and want to see how/when it was introduced.

Easy way to do this is put your test external to the tree. Then, bisect as normal for git, and run your test.

This is probably not uncommon for kernel tests, as you often have a test which could be "does this random laptop resume from sleep correctly?" Not really easy to automate that. And doesn't live in your tree.

> "does this random laptop resume from sleep correctly?" Not really easy to automate that.

that's what rtcwake is for

Right, my point was more that often everything will be working great for the main developer's laptop. I did not think it was uncommon to have someone come out of the weeds with a specific laptop that doesn't work. Or worse, it works if you sleep using the "sleep" button, but not if you close the lid.

So, yes, you may be able to automate it later, once you fully understand the specifics that cause it to not rewake. Possibly specific to how it went to sleep with the lid close. Until then, it is nice to be able to do ad hoc tests when necessary.

In the past, I've committed the unit test like you, branched, rebased the unit test back to some appropriate point in time, and then run git bisect. This avoids applying/resetting at each step, and allows for some trickiness like changing your test to deal with an interface that mutates over the range of commits you're testing. The hash of the errant commit will be different after rebasing of course, but it's easy enough to match it up with the one in the original branch.

I wish there were a guide or some official "this is how to manage your git repository along with your tests so bisect will work wonders" - IIRC, bisect gets tripped up by commits that won't build and also some merges... would be nice to know what exactly to do about those once they're in your history.

In cases like this I usually copy the relevant test file to somewhere outside the repo, and refer to that path directly in the bisect cmd. That way you don't have to worry about anything conflicting with the new test that you need for the bisect.

E.g. for a Rails app, I might end up using a command like `rspec ~/Desktop/my_new_test_spec.rb`.

I think it works ok if you make a new branch from a known good commit, add the test, then rebase the new changes on top. You could do that with

git checkout -b testing git rebase -i HEAD~50 (or whatever) move test commit to bottom, then save.

That's a great idea. I'll do this next time.

The best part is I should be able to resolve any conflicts before I start bisecting.

Where's the "accept as the right answer" check mark? :)

You generally create the test in a new file that won't be affected by checking out older code. If you need to change some part of your build system or test system, have the test be a patch that can be applied, run the build, run the tests, and check the results.

As far as having other uncommitted changes; just stash those. "git stash" is a tool you should use very often for any time you have uncommitted work that you need to clear out before doing some other git operation.

> I know you can automate running a shell script, but I don't write my automated tests in shell script.

Your tests aren't a shell script, but certainly you can compile and run your tests from a shell script.

You don't need to commit your unit test... you can run any script from any path.

git bisect run my_script arguments

Sounds challenging. If it's something that happened a lot, you could keep the test suite in a separate repo with API levels, such that tests need newer APIs go in a higher numbered folder (think database migrations in web frameworks) and create a simple run script.

The idea of bisect is that you only need to do log_2(n) steps to find a bug in "n" commits. Even if you have to do manual testing, it's still normally pretty fast, unless you have hundreds and hundreds of commits.

You're not considering the time it takes to manually test.

  * How much time does it take to do a clean build?
  * How much time does it take to do a deploy?
  * How much time does it take to reproduce the issue manually?

Add those together and multiply them by the number of commits bisect makes you check. It can add up.

Absolutely, and I'm not claiming that bisect is useful for debugging every regression. But it's saved me a LOT of time on a number of occasions, and shouldn't be dismissed out of hand just because it relies on manual testing. Even if going through 6 or 7 bisect revisions takes a few hours, it can sometimes be completely worth it.

Perhaps you could commit the unit test in a separate commit, then `git cherry-pick` it as a part of the script?

Will bisect not get confused if you change the commit range it's testing as part of running it? Also cherry-picking might still generate conflicts.

No: you would cherry pick the change needed to bring in the test code, run the test and decide whether it is "good" or "bad". Then before running "git bisect good/bad" you would eliminate the cherry-picked change with "git reset --hard HEAD^".

You can also "git cherry-pick -n" to only apply it to the working directory, with no commit, and then "git checkout -f" is an easy cleanup.