Stormforger – Cloud-based Load Testing as a Service | Hacker News Reader

Stormforger – Cloud-based Load Testing as a Service

stormforger.com

47 pointswhereismypw29 comments

I have no idea, why this ended up here, but if you are interested in some technical details what SF does: http://blog.stormforger.com/2014/05/27/load-testing-an-inter...

Wow, this sounds really impressive, especially:

> Here are a few numbers: StormForger did a total of 1,213,583,187 requests in over 50 load test runs to the Quizduell system and a total of about 2.21 TB of data was moved. The error rate was at about 0.000000216% (1 error every 4,624,616 requests).

You are referring to the error rate? :) We actually did about 800M additional requests two days after I've written the post.

Sounds useful for doing end-to-end from a cloud provider to another cloud provider, but there's more lessons to learn when you have 20 different ISPs and a couple dozen different clients sending you traffic in different ways.

For example, for free services without customer support, usually you don't notice the bugs from all these different devices trying to access your service under load. Once you start charging and provide support, suddenly people are calling you with all kinds of edge cases that affect a minority of users, and you have to invest time in tracking them down or refund the money (which is a bad idea since that builds a reputation that you'd rather dump customers than help them).

What other services like this are out there ?

I know of loader.io but I am specifically looking for ones which I can script to simulate real users.

http://blazemeter.com/ ?

https://wondernetwork.com/loadtesting ??

Also http://blitz.io

http://loadimpact.com/

I've used loadimpact.com before. Pretty good scripting capabilities.

Impressive. It's not their fault and they seem to be good and talented, but I can't stop thinking about a possible marketing line: "DDoS as a Service".

Apparently it is not DDoS enough as the load test was mitigated by Google ;-)

Normally services like this make you add a DNS record and/or file to the target to confirm you are the owner.

this is especially challenging actually. just because you own the domain name, doesn't mean you own the infrastructure you are pointing your DNS records to!

So adding DNS records is pretty useless to provide proof that you own the resources under test.

Yes but at that point you are theoretically traceable and can be found. It isn't like a botnet. Someone had to buy the domain.

Okay, that's true. But the question was about confirming the target's ownership. And that is something you cannot ensure via DNS entries.

Something I have been really interested in is being able to replay the actual traffic our site is experiencing. So, we press record and everything that's going to our server is recorded for playback at a later date. Does anyone know any solutions like that?

There are a couple solutions that do that - none of them I find sufficient. If you want to clone live traffic, that's probably okay, but recording traffic and replaying it in a usable fashion is a non-trivial task. e.g. systems tend to be dynamic so your recordings would need to be dynamic too in order to be realistic.

The api looks alot like tsung. I'm almost willing to bet that either they got inspiration from it or are using it on the server. Anyway, tsung is great and these services are great too.

if you look at the 'your data - your results' piece of the carousel, it has a log which has 'pid' as a column and, e.g., '<7038,69,0>' as a value. So you're probably quite right.

I can't recommend Redline13 (https://www.redline13.com) highly enough. They act as a service that coordinates load testing using instances within your own AWS account.

Really easy to use, and I can run tests that used to cost upwards of $5,000 for under $5. I don't know how they aren't better known amongst DevOps types.

Load testing as a service- thats a brilliant idea!

But why tie yourself to Ruby? (My initial impression is "gross!", but I can't say I'm much of a language purist as I spend most of my time in JS)

Would services like Stripe, Mailgun and Twilio have succeeded if they had forced their customers to use a Ruby DSL?

I saw basti's talk on stormforger at last month's interactive cologne – and I think he switched to JavaScript for the test definitions. Maybe the website is a bit outdated. :)

Indeed, the DSL is now in JavaScript.

Heroku started with just Rails. It's better to start with one language and do it well and then expand than half-ass it with everything from JS to Erlang imo. The important part is the quality of the service and the information it provides, not the DSL.

How much better is this than a program like WRK on a couple laptops on different IPs?

WRK is a nice tool that can bring you quite far, for sure. But what about the orchestration of complex sessions? As I understood that blog post mentioned in another comment StormForger was actually able to play a game. A simple one, but there is still a lot of complexity involved.

But even if you can build something like this with WRK (which should be possible, due to the Lua support) you still need to setup your test environment (reproducibly), collect all the logs from the machines involved, monitor the test generators, create useful metrics from the test results and make it possible to interact with them. And I'm pretty sure I forget a whole bunch of stuff that needs to be done.

And I know, that I don't want to do all this stuff by myself. With a service providing all this, I can concentrate on the real work: Interpreting numbers and fixing problems ;-)

BTW: I'm pretty sure under the hood there is some load testing tool deployed. Cannot imagine someone runs such a large scale test without such thing.

what happens if I use cloudflare, can it mitigate such massive traffic?

Some load testing services ban Cloudflare testing.

Reply on news.ycombinator.com