This!
I cant agree with this more, unless you approach me with an experiment that would put most science to shame I do not value your use of Jmeter/Tsung/Siege/Rando-Python-Script. If you test plan does not have pauses in it, and you are not engineering a nuclear reactor I probably wont even start the conversation.
....
At $DAYJOB, when we were putting together a new search engine we spent some time upfront putting in a lot of tooling to worry about and test performance every step of the way. Some of this tooling was built around jmeter and focused on generating realistic test plans (which we could derive from the already operational existing search engine). It was a neat infrastructure it tracked everything by commit, captured all manner of metrics and used huge datasets, mined from the logs, built out with markov models. Everything we reported into an install of the software that runs http://speed.pypy.org, we thought we had it conquered.
Yet it was so flawed! Data would get stale, models would break, caches invalidated datasets and much much more.
It was funny at the same time I wrote a tool that we now know as darklaunch, this tool is stupid simple, it captures incoming traffic to the production system with pcap, and then (after a little mussing with it) sends it through to a always on test cluster. Yet its this tool that produces all the load, has all the weird user behavior, never gets out of sync and can run load, soak, endurance and impact tests.
Guess which one we use to test out new software, or new features, or to tune a garbage collector in as close to production as possible ?
I have not started Jmeter now in over a year, its been no great loss; sure we have microbenchmarks where they matter but for everything else you often hear the words "darklaunch it"
Now for sure we have the luxury of having real traffic to use and abuse, but if you don't have real traffic then realise you are using a scrying device anyhow and I will bet you good money your assumptions are wrong.