* It allows me to focus on latency versus throughput independently
* It has tools for testing performance on pipelined requests (very important in my space)
* It supports scripting complex authentication processes (like magic headers and oauth2)
* It supports scripting fuzzing (random values) for URLs and POST requests
[1]: https://github.com/wg/wrk
For wide-area load testing, I simply buy some advertising since that gives me cheap access to millions of simultaneous users.
I love your 2nd approach (advertising). Any pointers to how to get started?
[1]: https://github.com/tsliwowicz/go-wrk
If you want to purchase advertising for this purpose, I'm happy to help with this. My contact details are linked through my profile.
- Your HTTP endpoints will need to be whitelisted with traffic quality groups and creative scanners (This can take a week or so). I recommend using a separate dns domain name for this since adblockers tend to subscribe to these lists.
- Your JavaScript should not be abusive. There are subjective tests for this. Generally doing an image request or some POST data of a few kb isn't going to be an issue. Some ad servers (like sizmek) download 2mb "test images" to exercise their bandwidth from some locations (like the US).
- Costs vary based on device, region, the web page the ads are on, and silly things like demographic. Some of these factors are interesting to benchmarkers but others are not. Thinking about what you actually need to test can help keep your costs efficient.
It includes the good ones like Vegeta, Siege, Goad, Hey, Httpref, Gatling, among others.
WCAT: https://www.iis.net/downloads/community/2007/05/wcat-63-x86 (pretty old and quite manual, but found it really effective, can control many clients distributing the load)
West Wind's WebSurge: https://websurge.west-wind.com/ Bloody good bit of kit and very easy to user
Bombadier: https://github.com/codesenberg/bombardier Very fast Go HTTP(S) benchmarking.
And Netling: https://github.com/hallatore/Netling I just really like it.
JMeter is indeed a tad heavier, but I find it's much more user friendly as in, you spend a couple of hours to build basic tests, or afternoon and have learned most of its functionality. It also has a variety of ways implementing this randomized user behavior [1].
When it comes to Protocol Level Testing, those two tools are still pretty much the best bang for buck, and by far the most popular open source load generators.
If the target you're hitting is a web app and you want to create a more realistic load scenario without crafting individual requests, check out a tool called Flood Chrome, which uses thousands of instances of Google Chrome[1] to generate load. It uses the same scripting language as SeleniumJS, so it's quite easy to learn.
Also on the topic of Flood [2], they're a very good platform for abstracting away the server orchestration and results collection you'll face when you try to scale up from 1 to N servers running either Gatling or JMeter.
[2]: https://flood.io/
[3]: https://gatling.io/
However the smarter apps can have problems too. I've pinned all available cores on fairly beefy machines because they are gunzipping bodies (behaving more like a real browser).
If you are at a scale where you need really good testing I recommend that you think of wrk results as comparable to a unit test and something like jmeter as comparable to an integration test. It's not worth the effort for most companies but where I work we serve a decent amount of traffic and downtime costs 6 figures per minute so good performance engineering and good business overlap.
Shameless plug:
I have developed a small tool called vex [2] ( using PHP ) for this purpose but it's not feature complete.
[1] https://github.com/rakyll/hey [2] https://github.com/vamsiikrishna/vex