Comment by susi22 | Hacker News Reader

Comment by susi22 | Hacker News Reader

Parent Full thread

Monte Carlo Simulations (MCS) weren't invented to do software testing and I wouldn't really recommend to use it as such. There is better ideas. MCS are used daily for processes/algorithms that are so difficult that a analytical solution is hard or impossible to find. You just throw in some data in your black box and put a "probe" at your variable that you're interested in. Do it many times and you see the pattern (eg. plot a histogram when probing a variable of 1 dimension etc).

If you have an algorithm that you understand and is not too complicated, then I'd argue that MCS are NOT the way to go (maybe in addition but NOT solely). The reason is that due to the law of large numbers it keeps simulating the nice cases instead of the edge cases. Unless you change your input distribution.

I really expected the article to conclude with a link or explanation to Quickcheck [1] since this is exactly what the author is trying to get to (Ie automatically check the edge cases and automatically minimize the input which fails it). Many languages have been porting these ideas. The Scala/Clojure ones are pretty nice.

[1] http://en.wikipedia.org/wiki/QuickCheck

Original paper of 2000:

http://www.eecs.northwestern.edu/~robby/courses/395-495-2009...

Testing with randomized inputs is a reasonably well established engineering technique:

http://en.wikipedia.org/wiki/Fuzz_testing

Certainly it is not going to replace many other kinds of testing necessary for building reliable software, but it has its place I guess, this is an interesting case study:

http://blog.chromium.org/2012/04/fuzzing-for-security.html

I think "does my software fail on random input" is an extremely important test to run, it doesn't matter if a naive MCS doesn't hit the edge cases if it's finding failures in the middle of the problem spaces on the regular (I've had jobs where (somebody else's) code works fine on hand-selected test cases, but as soon as you try something a little bit unexpected it goes all to hell.)

You can say that 'oh you should have caught that earlier' and yeah, ok, whatever.

Depends entirely on the system. This is akin to thinking communication will break down if we all switch to different languages. Certainly true, but far from a likely thing to worry about.

That is, "random input" is not nearly as important as "likely input."

So, the question here would be if you could use a simulation to cover "likely bad" values. Specifically, based on values that have caused previous methods of similar types to fail. I would think certainly. This is, essentially, boundary value testing. (Or am I wrong?)

And yes, I realize there are plenty of libraries that help with this. ScalaCheck/QuickCheck and the like.

Reply on news.ycombinator.com