But yes, even if there is no test, this should have been caught in code review or latest when testing the OS.
You test a UI by basically throwing sequences of interactions at it. Some of the properties you'd want to assert:
* Given two interaction sequences that only differ in what they do to the password hint field in the UI, the result should only differ differ in the returned password hint. (Or alternatively neither should finish the UI dialogue.)
* As a dual: two interaction sequences that share the same interaction with the password hint field should yield the same password hint field. (In this case it is acceptable for them to differ in whether they actually finish the dialogue.)
Those two are fairly generic, so you can imagine setting that up as a general framework for all your data input fields in your UIs. It should work backwards as well, eg to assert that eg the UI should look the same no matter what password (/ password hash) is stored, to make sure you are not leaking any information.
See eg https://fsharpforfunandprofit.com/posts/property-based-testi... for more background, and http://hypothesis.works/articles/incremental-property-based-... for a real world example.
As for 'should have been caught in code review': yes, but humans are fallible and they should get all the help we can give them. To see for yourself, have a look at the example (a simple runlength encoding) at the top of http://hypothesis.readthedocs.io/en/latest/quickstart.html and see whether you can spot the obvious error just by reviewing the code.
My go-to example for property based testing isn't addition, but idempotence. Idempotence is a concept well known from REST apis to the general programming public, and also in sorting or clean up data. And it's easy to state in code, eg: sorted(sorted(x)) == sorted(x).
To give a better answer : I’m not knowledgeable of the whole source for the framework, but there are only two cases :
- either the outputs of your critical function are testable and then just test it.
- or they’re not ( most often because of side effects), in which case you should extract the critical part in purer function, and test that.
It’s a great benefit of unit test : they force you to isolate side effects from business logic, to be able to test the latter.
In that case, if the function isn’t testable, then maybe the creation of this intermediate minimal dictionary should belong in its own function that just does the data mapping.
No doubt this basic test was done on the command line API, which works correctly. It’s likely the problem here is they created a separate StorageKit API just for Disk Utility to use and got sloppy with the unit test for that. A good example of why it is a good idea to try to have GUIs and command lines use one code path whenever possible.