224 karma · joined October 25, 2009
If you have time, I have another round of questions:
1. Did you try formal methods like TLA+ on the client? I think that the logic covered by CanopyCheck may be a nice target.
2. Do you have some tests with several clients running at the same time on a shared directory? In particular, I think of the termination invariant where the clients are fighting because several users have reorganized the directory by moving a lot of stuff, and each client is trying to converge in a different direction (ie they are making operations that cancelled the ones made by other clients).
3. The article says "In the Nucleus data model, nodes are represented by a unique identifier". Does it happen that a node has to change its identifier? For example, in a scenario like this one:
(ada) $ offline
(grace) $ offline
(grace) $ mv ~/Dropbox/shared/TODO.txt ~/Dropbox/private-grace/
(ada) $ mv ~/Dropbox/shared/TODO.txt ~/Dropbox/private-ada/
(ada) $ echo 'foo' >> ~/Dropbox/private-ada/TODO.txt
(grace) $ echo 'bar' >> ~/Dropbox/private-grace/TODO.txt
(ada) $ online
(grace) $ online1. I'm a bit surprised that you don't persist the scenarios of the tests after a failure, only its seed. Does it mean that when you want to replay it, you have to redo the minimization phase? Or, do you have a way to find a seed for the minimized scenario?
2. Do you have some tests where you generate a set of operations and play them twice: one time with the mocks and one time on the real servers to check that they have the same results?
3. The article says "Note also the importance of the commit hash, as another type of “test input” alongside the seed: if the code changes, the course of execution may change too!". How are you ensuring that a commit really fixes a bug, and not just change the execution path to a happy path where the conditions of the bug are not met? By playing again a lot of tests, or do you write a new unit test that exhibit the bug to ensure the reproductability?
4. Do you think we can say that CanopyCheck is applying randomized testing at the unit tests level and Trinity is applying it at the integration tests level?
[1]: http://erlang.org/doc/apps/dialyzer/dialyzer_chapter.html [2]: http://elixir-lang.org/docs/stable/elixir/typespecs.html
Not sure it helps, it's on my todo list.
"When you upload or otherwise submit content to our Services, you give Google (and those we work with) a worldwide license to use, host, store, reproduce, modify, create derivative works (such as those resulting from translations, adaptations or other changes we make so that your content works better with our Services), communicate, publish, publicly perform, publicly display and distribute such content." -- http://www.google.com/intl/en/policies/terms/
EDIT: We are a team of 8 people using it, 2-4 per project.