Ask HN: please review our online load testing service - loadimpact.com
loadimpact.com
loadimpact.com
Why would I want to pay you $40/mo, if I'm paying $3/mo for shared hosting that I know can handle 250 users?
And its the same way through out. Even at the highest level, it'd be cheaper for me to lease a second dedicated server to spread the load, than it is to pay you for your service.
And I don't think this is something that should be a monthly service. I only want to check my load once in a while to see if its doing fine, so I think by asking so much money on a monthly basis, you are driving users away. I think you'll be better off doing it as one off results. i.e. the $499/mo option, should be a $29.99 one time fee for 3 reports
First - the people on $3 /mo shared hosting are not in the target market for this service. If you are on shared hosting, you don't need loadtesting.
Second - I think subscription is fine. As a subscription, I recognize that I -should be- load testing on a regular basis. As a-la-carte, I am much more likely to say "umpteen dollars?I can do my own loadtest for cheaper then that."
But I guess it depends on how you use the service. For people who just want to verify performance occasionally, what you write makes sense, and we're going to create a one-time (or maybe "one-day") offer also that you can buy every time you need to run a test.
If the higher-end plan ends up being popular with a certain type of business (?), perhaps you could include a few "mega-tests" per month for the low plan. I would mind fewer 250 tests if it meant I could perform a couple of 500 or 1000 tests.
Who is your target customer anyway?
But, seriously, this should be strictly an opt-in service. Only if there is a loadimpact.txt on the server, you run your tests. Otherwise - refuse and explain that the server admin has not consented to the stress testing using your service.
I'm sure you have some DoS provisions in place, but in the end it really boils down to the fact that if you screw up, then it's my server that gets DoS'd.
Allow-Stress-Testing: *
I love being able to test this out right away, so I'm in favor of keeping it pretty much how it is. I was looking for this exact service two weeks ago and was surprised I couldn't easily find someone who did this with transparent pricing/plans.
Keep the transparency and ease of trying it out-- there are too many companies in this space saying "Call us for a free quote" with "account execs" when this doesn't need to be that complicated of a service.
We want the service to be really user-friendly and easy to get started with, and think we have reached a fairly good level of compromise where security is "good enough" without sacrificing usability.
What we could, and should do, however, is be more informative about all the security measures we have taken to prevent abuse. Because we have put a lot of man-hours into that lately, and we will continue to build more, hopefully non-intrusive, security measures all the time to try and make the service unattractive to would-be abusers.
Just keep in mind that for an average hosting provider it is far easier to null route your subnet than to sit there and assume that you will not screw up.
Also, as a side note, even if you are tracking per-IP statistics of your tests, I suspect you are not doing any detection of multihomed machines. For example, my company has a dozen of websites served from a single box, each on its own IP address. Do you seriously expect us to let your service anywhere near our boxes ?
(Its a DDOS in a can if you have a few dozen people capable of reading directions.)
You probably don't want to do this because it would make the process much harder, but you can allow it in reverse; claim to deny load tests for a domain.
I'm guessing that you want to give a "live demo" to people, but I'm sure you could just show a demo report of a couple sites that have already been tested.
Example: the test starts by simulating 10 clients. The average page load time measured is 500 ms. The test then moves on to test using 20 clients. If the average page load time goes above 1 second (twice what it was originally), the test will be aborted.
We have a general security philosophy that is based on different levels of trust. An anonymous user is trusted the least, a registered but non-paying user is trusted a little bit more, and a paying user is trusted even more. The level of trust determines how often you can run tests, how much data your tests are allowed to transfer, how often you can place load on individual destination servers (IPs), what settings you can make for your tests, etc.
First of all, you're admitting that it's okay for anonymous people to inflict a significant, measurable delay in page load time for my entire website. That is completely unacceptable.
Also, while I'm sure your system is well designed, it probably isn't perfect (because nothing is). If your monitoring service malfunctions then the tests might never cut off.
There should only be two levels of trust: people who own the server (and can test it), and people who don't own the server (and can do nothing to it).
I went through your registration process, and got nothing. Very disappointing - the type of thing that will make me never look at your website again and scoff about it to my friends should they bring it up.
And a couple other nits:
1) Your registration page where I choose my plan type is not intuitive. Maybe it's because green is not a good color to call attention to stuff.
2) Your Register and Cancel buttons should be different colors. I clicked the wrong one and had to redo my registration.
3) Don't ask for number of employees and industry. They're not required and they will hurt your conversion rate slightly. And it indicates to me you might sell my data.
Employees and industry is there for us to learn who our customers/users are. This is a new type of service, and we need information on who are target audience are, because we frankly don't know that yet. These values are optional to fill in, though. Should we be more informative, maybe, and tell people that we will absolutely not under any circumstances give out information about them to a third party - would that reassure you?
Finally, your confirmation email might have been delayed as there was a DNS issue yesterday that caused outgoing email to be queued for a while. Apologies about that too and thank you for the feedback. It's very nice of you to give it even though you weren't happy with the service. Negative (but constructive) feedback is the most valuable of all for us.
You did a good job of making the reports and graphs pretty while still keeping a clean feel to the whole site. There seems to be a good amount of polish and attention to detail there too. I don't feel scared to click anything while the test is running for fear of breaking something or redirecting me to a FAQ page and canceling the test.
I look forward to seeing how this pans out.
- shelled off to the Pick a plan screen
- pick the free plan
- nothing happens, so click it a few more times
- scroll down to see if I missed something and find WAAY too many required fields
- click register
- screen scrolls up to the top, but nothing else happens
- click register again
- "this email is already registered"
- find login box, type in email an password
- get the homepage and a "Account not confirmed" message
- open my email client, find the mail, click the link
- get a "confirmed" page, and I STILL need to go type my email and password in again.
Guys, you're killing me. Such a good product, with such a terrible registration experience. It's completely unnecessary.
Ask for a username and password, and let me get on with trying your thing. If I like it I'll engage with you further. That's Startup Usability 101. Fix it and maybe you've got a shot.
What happened was that the free plan was preselected when you entered the registration page. This means that nothing happened when you clicked it (as it was already selected). When you then clicked "register" you probably got a "Registration successful - an email has been sent to you for account confirmation" message at the top of the screen, can you have missed that? Then when you click again on "Register" you get the error message that you're already registered.
Maybe we should think about transporting the user to another page when they choose a plan/a subscription. Have less things on one and the same page. Your experience suggests that that might be a good idea I think.
User interfaces are not easy, and it is so hard to see things with objective eyes as a developer.
- I had to scroll down on the plan selection page to even see there was a form to fill out. This is on a 1680x1050 screen btw.
- I expected to be taken somewhere when I clicked on the plan I wanted, instead it was highlighted and with the previous issue in mind, I was left confused about what to do next.
- The most confusing part for me was after I fill the form out, hit submit, and the page scrolls to the top. It took me a few moments to realize that a "Registration email has been sent" box had appeared. I would recommend going to another page after you submit the form.
That's all so far, I'm just starting to use the service so I may have more comments, but already I can see it being a very useful service for the small startup I work on.
I assume there is a setup process that is happening behind the scenes that is taking some time, but it left my finger itching to hit the 'refresh' button while I was waiting for this to complete. If it were possible to have some kind of ajax-y thing telling me what is going on here that would be great.
Otherwise I was very pleased with your service. I am definitely going to sign my company up for it, as I can see it being invaluable for tweaking the performance of our site.
As far as suggestions, it would be great if there was some way to work in extended metrics into the load testing. E.g. allow pre/post-test hooks between each test run that allowed me to associate arbitrary data such as load averages, memory usage, etc.
We have a list of URLS/sites that are exempted from these checks, but this list needs to be populated. So far we only have a few entries on it (like google-analytics). It seems your site loads things from googleapis.com so I added an exception for that address now. Try again and see if it works.
We probably need to work on the information to users whenever a test is denied also. If you know what the offending URL is, you can remove it from your load script and run the test without it.
[Edit] you mentioned googleapis.com already.
I have GA, Disqus comments, TipJoy and Reddit badges, and a few other little things.
Maybe you could have an option to restrict it to your own domain name.
We don't currently have any "crawler" mode though, where the system tries to follow links etc on the site automatically.
My paranoid mind can think of an scenario like this (INAL so I maybe be off): a web master asks for the load test, this is done and the site goes down for a while, incurring a monetary loss (loss in sales for example). The owner sues the testing company for a 'DoS' and the testing company has no way to prove that it was authorized (what text file, there's none!).
I personally wouldn't do a stress test on a server without written authorization of the owner. The way I would implement this on an online service is by asking the target web site and the email address, where this address has to be in the public whois information. Then I'd send a special message to that email address and by replying the owner gives authorization (this way I'd have something from the official email address). This is not a 'written authorization' but comes as close as it can and the whois email address is used for domain transactions etc so it has the same level of protection/legal procedure behind.
One of the sites I am working on now is moving to a new codebase. We wanted to load test for a few hours but found everything we looked at to be too expensive.
Can you handle very large sites? I assume your system is in the cloud and scales as needed?
We currently simulate max 5,000 concurrent users on a site and that is something our own servers can handle, so right now we're not using cloud services. We do have support for running the backend on a cloud though - we've been testing it and it works very well. If customers start asking about running larger tests, then we will probably implement a cloud backend mode that will feature a much, much larger traffic-generating capability (but also be more expensive).
Good luck!
Refs:
If you have the time, knowledge, interest and a server to spare, then ab or httperf (or I would recommend the Grinder - http://grinder.sourceforge.net/ ) is definitely worth a try however.
One feature request that would have me signup in an instant is if you could provide web service load testing as well.
I'd also like to know more about the session recorder and if I can feed parameterized data into the scripts.
The session recorder is pretty cool, we think. We have managed to make it zero-configuration (i.e. no user configuration necessary) while still being able to handle client-side javascript. There are other HTTP session recorders out there, but to be able to handle javascript they're usually implemented as a true HTTP proxy that you have to configure your browser to use, or they're a browser plugin that you have to download and install.
The recorder outputs a load script that you can edit afterwards, if you want. However, the load script is currently more of a "list" of things to load, with pauses in between. It has no logic, conditional statements etc. and does not support parameterized data. This is something we intend to change soon though. It just needs some thought so we don't open up new possibilities for abuse/overuse of resources when we let users execute their own load script code on our systems.
http://loadimpact.com/result/www.improvingtheweb.com-b75533d...
I think there is a lot we need to do to explain things better. That is the most difficult part of all.
i.e. Load increased with the following stress pattern, which can indicate an improvement is needed in areas A, B, and C.
That said, it would be great to incorporate some of their features such as different scenarios, load patterns, particular series of pages, random, etc. It would be great to not have to have a dedicated resource to run the stress tests, it's also tricky to know if the local bandwidth can properly emulate x number of machines.
-I believe that you'd have much more success if you charged on a test basis, your monthly billing might scare people away.
Also, is there a way to simulate more than 5000 users?
5,000 users is what one single instance of the non-threaded load generator can simulate today. We have some optimizations we can make that should improve this though, and we are also going to implement multi-source/multi-program load generation that means we can distribute the load generation for a single test over several processes and several load generator hosts. Using only our own infrastructure I think we can scale the system quite a lot, but obviously, if we want to run really large simulations with several hundred thousand, or millions, of simulated users, then we need to buy cloud server capacity.
If you ask me, however, I'd say that the feedback we are getting here is what's really invaluable. It will help us improve the service a lot.
Soasta does essentially the same thing as we do although they are a lot more expensive and aim for larger clients.
Traditionally, load testing has been both difficult and expensive, which is why mostly larger organizations have been doing it. Soasta I think has therefore aimed for this segment while we're betting on a larger, but so far unproven, market among small- and medium sized companies.
Does this mean somebody is DOSsing my site through your service?
I am going to fix so that we inform people what the offending URL/object is (rather than just saying "one of the systems involved in the test..."), so they can remove it from their load scripts (if you're a registered user you can edit your load scripts before starting a load test) and also contact us and ask us to put new systems into our exclusion list.
http://loadimpact.com/result/google.com-398c66a059f3ab3c096b...
1) Run several low user tests (200-300) to tweak server settings, app code, etc. 2) Then run one or two big tests (1000+) just once to see the results.
I MIGHT need this at most once a month. That just isn't worth $200 per month. I'd much prefer a subscription for the lower levels of consistent use, and then be able to purchase an additional once-off very large test for maybe $10-20 each.
http://www.linuxhaxor.net/2007/12/06/siege-an-httphttps-stre...
If you're interested in running your own load testing software I would recommend the Grinder - http://grinder.sourceforge.net - which is very nice and flexible. If you need to generate a lot of load, curl-loader might be of interest - http://curl-loader.sourceforge.net
I might as well explain how it works as people seem to mention these things a lot here: when we know what URLs we have to load in order to load a web page (i.e. when we have generated a load script for the test) we find out all unique host addresses involved, then resolve the IP addresses to all these hosts, then we check how many times the last 24 hours we have performed tests where these IPs have been involved, how many bytes we have transferred from and to each of these IPs the last 24 hours, and how many transactions we have subjected these IPs to the last 24 hours. If any of these values exceed certain predefined tresholds, we deny the test.