Monte Carlo Simulations and Fibonacci: Why Developers Still Need The Basics | Hacker News Reader

Monte Carlo Simulations and Fibonacci: Why Developers Still Need The Basics

blog.smartbear.com

48 pointsBaustin15 comments

Monte Carlo Simulations (MCS) weren't invented to do software testing and I wouldn't really recommend to use it as such. There is better ideas. MCS are used daily for processes/algorithms that are so difficult that a analytical solution is hard or impossible to find. You just throw in some data in your black box and put a "probe" at your variable that you're interested in. Do it many times and you see the pattern (eg. plot a histogram when probing a variable of 1 dimension etc).

If you have an algorithm that you understand and is not too complicated, then I'd argue that MCS are NOT the way to go (maybe in addition but NOT solely). The reason is that due to the law of large numbers it keeps simulating the nice cases instead of the edge cases. Unless you change your input distribution.

I really expected the article to conclude with a link or explanation to Quickcheck [1] since this is exactly what the author is trying to get to (Ie automatically check the edge cases and automatically minimize the input which fails it). Many languages have been porting these ideas. The Scala/Clojure ones are pretty nice.

[1] http://en.wikipedia.org/wiki/QuickCheck

Original paper of 2000:

http://www.eecs.northwestern.edu/~robby/courses/395-495-2009...

I think "does my software fail on random input" is an extremely important test to run, it doesn't matter if a naive MCS doesn't hit the edge cases if it's finding failures in the middle of the problem spaces on the regular (I've had jobs where (somebody else's) code works fine on hand-selected test cases, but as soon as you try something a little bit unexpected it goes all to hell.)

You can say that 'oh you should have caught that earlier' and yeah, ok, whatever.

Depends entirely on the system. This is akin to thinking communication will break down if we all switch to different languages. Certainly true, but far from a likely thing to worry about.

That is, "random input" is not nearly as important as "likely input."

So, the question here would be if you could use a simulation to cover "likely bad" values. Specifically, based on values that have caused previous methods of similar types to fail. I would think certainly. This is, essentially, boundary value testing. (Or am I wrong?)

And yes, I realize there are plenty of libraries that help with this. ScalaCheck/QuickCheck and the like.

Testing with randomized inputs is a reasonably well established engineering technique:

http://en.wikipedia.org/wiki/Fuzz_testing

Certainly it is not going to replace many other kinds of testing necessary for building reliable software, but it has its place I guess, this is an interesting case study:

http://blog.chromium.org/2012/04/fuzzing-for-security.html

as a mathematician, i grimace at the level of resistence I get from programmers as being too "theoretical".

I try arguing, "but all your libraries do at low level is math calculations, don't you want any control over that?"

The typical response is "no"... people are quite happy with the boundaries they feel their API's are setting for them.

The purpose of having stuff in libraries that have been tested and work is that you don't have to worry about it. If we had to reinvent the wheel every time we would never get anything done.

This is a typical rebuttal from programmers when they don't know how something works underneath.

You are correct about "why" libraries are created.

Since, I'm assuming you're talking about Open-source libraries: what happens when theres an issue you're having and wondering why the library isn't doing what you want?

I'd end up reading the documentation and understanding why I was using the library incorrectly. Failing that, I'd go look at how the library was implemented.

There's a distinct difference between looking at something when it's broken for the sake of fixing it and implementing every library on your own because you don't want to use the abstractions created by the programmers who came before you.

Think of it as Lazy evaluation. Without the need to solve some specific problem poking around a library may vary well be a waste of time. Granted, if that's what you want to do in your free time feel free, but work is about getting things done not having fun.

If it's a much-used library with a reputation for quality, like FFTW, I'm going to assume the fault isn't exactly in my stars here.

I seem to recall a statement to the effect that the number of people that actually understand how a modern Intel cpu works can be counted on one (slightly mutilated) hand.

Don't you want to know how your cpu works? What happens when you're having an issue and wonder why your code doesn't work as expected?

Depth of knowledge is good, but I think the rational choice is to know the most about the domain you're working in -- because the entire domain is already too big for any one person to fully grasp.

The point isn't reinventing the wheel. It is understanding why the wheel was created, why it is better than some of the alternatives, the pros and cons over using one kind of wheel vs another, and when it is time to stop using a wheel and invent tank treads instead.

I have an inner desire to understand things full circle (or full stack here) but often cause paralysis in case I get stuck [1]. I really dislike the blind ignorance that can cause 'acute library calling' but at the same time I'm very impressed, even envious at times because of [1], by people that can really abstract and build on top of systems they barely get.

As an example SQL was black magic to me until I made a big tangent off into the functional programming world where name binding and capture is something almost first class, which made SQL almost obvious now that I have a better idea of what and how they could do underneath. It also leads to hacking opportunities otherwise I'd never be able to even think about it.

As a mathematician, until I have reason to suspect libraries, I'm happy to let them be. (and for a lot of the high quality well-tested ones out there, I need a lot of evidence to believe it's them, not me).

Perhaps they see it as reducing their task to a set of previously solved problems.

Reply on news.ycombinator.com