Hacker News Reader

> To be honest, this is one idea I was toying around with. As in, have a test suite that generates a "state" file for your program and then, in any given patch, list the items in the state that you'd expect to change.

It is very clear that you have not worked on large, complex systems.

The problem is what do you do when you change the code? System state is different; how do you know it is correct?

Also, un-tenable with any code that is non-deterministic, like multiple threads or external I/O.

> As in, have a test suite that generates a "state" file for your program

What do you mean here?

Do you mean you would run a specific function with specific inputs and record specific outputs and compare them?

Yes, or rather, run your whole program with specific inputs then (e.g. by using a debugger) capture as much of it's state as you can and compare that with the state from the next run of those same inputs.

E.g. given a program with variables

a1 = array a2 = int a3 = string

Running might get the state flow:

a1 = []

a2 = 10

a3 = 'abc'

a3 = 'dd1'

a2 = 200

a1 = [200,'dd1']

Then, if a change is made, the programmer can say "I expect this change to only affect the state of a2" and if you get the state flow:

a1 = []

a2 = 11

a3 = 'abc'

a3 = 'dd1'

a2 = 500

a1 = [200,'dd1']

Then the test passes

If you get the state flow:

a1 = []

a2 = 11

a3 = 'abc'

a3 = 'dd1'

a2 = 500

a1 = [500,'dd1']

The test fails with error "The change you expected to only affect a2 also had an effect upon a1"

But again, the actual state you capture here and the way you check for equality (e.g. to you check for equality among all the states during the whole runtime, or just in the final state of the program ? If the former how is state-change ordering handled ? If the later, what about relevant state that is out of scope by the time the program finishes running ?)

How is this different from a set of unit tests where you give the function under test each of those inputs and have the expected output be the same for each entry as the given list of outputs?

e.g.

  [TestCase(null, null)]
  [TestCase(10, 11)]
  [TestCase('abc', 'abc')]
  [TestCase('dd1', 'dd1')]
  [TestCase(500, 500)]
  [TestCase([200, 'dd1'], [200, 'dd1'])]
  public void StateChanges(input, expected) {
      actual = DoThing(input);
      Assert.AreEqual(actual, expected);
  }

Because this is only checking the individual function, not the entire systems state. If I have a function that adds two ints I could test it with [TestCase([2, 2], 4)] and this would pass but if it also changed the value of an object there could be an issue a unit test (and indeed integration/e2e test) wouldn't catch.