Ask HN: What tools and methods do you use for software testing?
Other than unit testing, what tools and methods do you use for software testing? Do you even do any other type of testing? Do you use commercial tools?
We then also do full build performance tests using the grinder at night. This is currently not very automated at all, and involves doing processing in the morning. These are short runs, multiple samples averaged. It would be good to know how others are running there performance testing automatically, and also how they are monitoring the systems during the exact times of the test.
We also prior to release run some serious stress testing, again using grinder. Then soak tests using grinder at expected load for 1 week. This gives us the data we need for a release.
Do you mean metrics from your program under test, the program testing it or the system?
We have the test metrics from the tester, the logs from the software under test, but it's working out how to properly monitor the machines and tie them to specific test runs we haven't managed yet. The tests start up from around 8 at night on a seperate 10 machine subnet. We are looking in to setting up munin in a controller-slave setup on one machine to take aggregate data from the other 9 test machines and somehow tie it to a specific test run and provide us with report graphs using both the throughput data from the grinder/jmeter. This is the bit I can't seem to find any information about doing. I've seen a lot of documentation and tech talks telling how you must monitor the system as well as the software, and they show throughput against memory usage and throughput against cpu usage, file IO etc. We do a lot of this stuff manually, after we see a spike in system usage stats during the night we will go digging in logs comparing time stamps finding out what was running and what went wrong. It's painful and it's slow.
It would be great to read how some other people solved this problem.
Most of our stuff is custom code for a number of reasons, the biggest external library being HttpClient. I'm not sure how other people have solved this particular problem, I just got to the point that clean EC2 instances were generating pretty comparable data and then set to putting out fires in the code (:
To test the framework itself, we mainly use Selenium to test the actual HTML and javascript output. To test things that Selenium can't easily handle, we've also used the Watir library (http://wtr.rubyforge.org/).
Our real unit tests are basically built on top of JUnit, though we have our own extensions to it that are specific to our platform.
To run the tests and report results, we have a custom application developed on our web framework and the rest of our software stack. Originally we used CruiseControl, but we outgrew that fairly quickly. It does a lot of stuff for us, but the key abilities are 1) parceling the tests out to a farm of servers and 2) assigning test breaks to users (basically a guess based on who changed what files in source control since the last successful test run); if you have more than a few people on the team, without actually making the test breaks one person's problem to resolve you end up with massive diffusion of responsibility where no one thinks it's their break.
We also have an internal performance testing harness for running load tests, which we write using the same framework as the smoke tests I described above (though in that case the clients are naturally remote).
I find nose particularly useful both for running tests (including unit). I have also found (fire)wait to be useful for driving a browser.
Watir (ruby-scriptable driving of a browser) for web UI testing.