I see the author posted this.
What are your thoughts on automated/unit tests used to guarantee expectations do not change in future revisions (a regression suite) and also used to present example inputs usages of the API that was designed?
What are your thoughts on automated/unit tests used to guarantee expectations do not change in future revisions (a regression suite) and also used to present example inputs usages of the API that was designed?
To be honest, this is one idea I was toying around with. As in, have a test suite that generates a "state" file for your program and then, in any given patch, list the items in the state that you'd expect to change.
That way, one can basically catch a lot of "bugs" which often boil down to "this change I made here to affect X is also unexpectedly affecting Y".
My main problem with this are:
1. I see nobody doing this, and I'm not sure why. 2. The tooling for this doesn't really seem to exist, and I'm not sure how easy one could bring it into existence and/or if a generic version of it could be written. 3. This breaks down with a lot of software where a tiny change can affect, well, everything (e.g. modifying a random seed or changing an error-checking constant)
What do you mean here?
Do you mean you would run a specific function with specific inputs and record specific outputs and compare them?
E.g. given a program with variables
a1 = array a2 = int a3 = string
Running might get the state flow:
a1 = []
a2 = 10
a3 = 'abc'
a3 = 'dd1'
a2 = 200
a1 = [200,'dd1']
Then, if a change is made, the programmer can say "I expect this change to only affect the state of a2" and if you get the state flow:
a1 = []
a2 = 11
a3 = 'abc'
a3 = 'dd1'
a2 = 500
a1 = [200,'dd1']
Then the test passes
If you get the state flow:
a1 = []
a2 = 11
a3 = 'abc'
a3 = 'dd1'
a2 = 500
a1 = [500,'dd1']
The test fails with error "The change you expected to only affect a2 also had an effect upon a1"
But again, the actual state you capture here and the way you check for equality (e.g. to you check for equality among all the states during the whole runtime, or just in the final state of the program ? If the former how is state-change ordering handled ? If the later, what about relevant state that is out of scope by the time the program finishes running ?)
e.g.
[TestCase(null, null)]
[TestCase(10, 11)]
[TestCase('abc', 'abc')]
[TestCase('dd1', 'dd1')]
[TestCase(500, 500)]
[TestCase([200, 'dd1'], [200, 'dd1'])]
public void StateChanges(input, expected) {
actual = DoThing(input);
Assert.AreEqual(actual, expected);
}Also, un-tenable with any code that is non-deterministic, like multiple threads or external I/O.
It is very clear that you have not worked on large, complex systems.