Ask HN: How do you test your web applications? | Hacker News Reader

Ask HN: How do you test your web applications?

56 pointsunignorant24 comments

I'm an undergraduate who happens to be taking a graduate seminar in Programming Languages. Recently, we read a paper on testing web applications, and the (perhaps cynical) notion among the grad. students was that "no one tests web apps," which I felt was wrong.

So to possibly vindicate myself, I'm curious as to how many here do test, and if so, how?

Edit: Thanks for the responses! To be more clear, we were discussing testing more along the lines of "finding bugs and vulnerabilities" and less along the lines of "usability."

View on HN

Testing is a vague term. There are lots of types of testing, which other posts go over quite well, but I'll give a rundown.

* Requirements testing - take the napkin you scribbled your idea on, and mentally run through it a few time. A feature that is 100% correct code but does the wrong thing, is still a "bug".

* Unit testing - low level, typically at the database and business logic level. Does this function return what I think it does?

* Integration / Frontend testing - automated scripts that run through and 'click' on elements, and verify content. Webrat and Selenium are a great tools for this.

* Usability testing - throw some real users at it, find bugs in your design and workflow.

* Penetration / security testing - attack your site from the outside. Alternately, audit the site from a code-level point of view. Find security holes, figure out ways that people will abuse your site.

* Load testing - once you get a site up, will it stay up after traffic hits. This is a rather deep topic in and of itself.

You get the idea. There are probably more that I didn't hit. But basically every step along the way to having the final app can and should be tested.

Testing is more than finding bugs, it's finding flaws in thinking. Whether it's the 'business analyst' aiming for the wrong target, the designer having a hard-to-find button on the mockup, or a coder using the wrong import() function override.

Since I do this professionally, I have a lot to say about testing web apps, but I'm going to keep my recommendation short and precise.

Go here:

  http://portswigger.net/suite/

Drop the $200 or so for a copy of Burp Suite Pro.

Burp is the industry standard tool for testing web apps; probably more than half of all web app penetration testers use it. It's like Firefox Tamper Data on steroids; it gets between you and your request, breaks out all the parameters for you, and gives you tools to manipulate and replay the requests.

You in particular want to understand the Burp Intruder, which is Burp's fuzzer. The fuzzer takes a request, tries to figure out where all the parameters are (you can help it with that), and then feeds malicious inputs through them and records the responses.

You want to fuzz every page in your application at every code drop. We've seen entry level QA people trained to do this, so delegate the task if you're not sure you can make yourself do it reliably. You want to collect every error response, and you want to grab your application logs and look for every exception or abnormal event. You want to know how the responses are different from the last run each time you ship a new version of the app.

Burp is so extremely effective that I'm amazed every web dev team doesn't already have a copy. I think it's because it's billed as a "security" tool. But it's obviously more than that.

Webdevs have a lot to say about unit testing and cloud testing and integration testing and test-first development. What I can tell you is that I've seen no correlation between test methodology and pentest outcome. The TDD teams fall just as fast as the seat-of-the-pants teams. I want to strongly recommend that you don't waste time strategizing on how you'll test or validate architecture or whatever. Burp is a no-BS tool. Get it, run it, and you will find bad things. Work out your effective bill rate and the amount of time it takes you to adopt or extend a test suite, and it's the cheapest quality enhancer that's probably available to you. Do whatever you want on the testing side, but do Burp first.

s/Burp Suite Pro/QuickCheck/.

Basically, there are two stages to a web application; understanding the request, and processing the request. The understanding part is a parser, so you are going to have to ensure the correctness of that with test cases -- you can never prove correctness, you can only show that it works on a lot of input.

The rest of your application, however, will be modeled on a finite number of possible states that this data structure represents. This means you can use better tools (things like Catch) to ensure correctness; you can literally prove that there is never a case you don't handle, for example. This gets you reliable code more quickly than random testing does.

Anyway, to answer the OP's questions, web apps should be tested like everything else. A web app is a pure function from Request -> Response, so test it like any other pure function.

Selenium is what I'm using, Selenium IDE (firefox plugin) makes it easy to set up tests. http://www.seleniumhq.org

UserTesting.com is great for usability testing, they have people follow some of your instructions and record their screen/voice as they go through it. Feedback from this has lead to a lot of critical improvements.

Another article you might be interested in is about how software is developed for the space shuttle (they test like crazy). Someone posted the link on HN last night and it's fascinating: http://www.fastcompany.com/magazine/06/writestuff.html

On a related note, one of the authors of Selenium IDE founded a startup called BrowserMob (http://browsermob.com/) that uses Selenium recordings for load testing and monitoring.

Great guys there, and a fantastic service. I've been a customer for over a year - it's one of the few companies I'm impressed with enough to evangelize.

Shinya Kasatini is the original creator of Selenium IDE, based on my version of Selenium (which was simply called "Selenium Core"). Shinya is also famous for creating the "Pocket Guitar" iPhone app.

Patrick Lightbody is founder of BrowserMob and is the co-author of Selenium RC. RC is the Java-based Selenium server for driving browser testing from any programming language.

I also do lots of stuff with Selenium at my startup, Sauce Labs. Sauce focuses on making your functional/acceptance testing cycle go faster by providing hosted "browser in the cloud" infrastructure where your tests run in parallel for a huge speed boost.

Went and checked out selenium based on your recommendation. Gotta say its REALLY nice. Thanks for the link!

Here's a guide on penetration testing you might find useful. I would assume some frameworks handle some/most issues "out of the box" but it's still a good guide to follow.

http://www.owasp.org/index.php/Category:OWASP_Testing_Projec...

Other than that, testing a web app is just like testing any other software. Look for edge cases, try different combination of things, etc. to say the least. But here are a few things to look out for off the top of my head:

* Don't be too restrictive on validation on fields like names, addresses/zip codes (if there are international customers), e-mails, etc. However, make sure people can't enter numbers in an SSN field or letters in a date field either.

* Make sure weird characters in different fields don't break anything. For example, entering O'Hara as the last name should be stored correctly in the DB and then rendered correctly to the user whenever you need to show it.

* Make sure your text box max lengths are not greater than the DB's column definitions. You don't want someone to enter a 50 letter name and have the field in the DB defined as varchar(20).

* Make sure the web app handles session time outs gracefully. You don't want a user filling out a registration form to get a phone call and then come back to complete the form, only to be redirected to the starting page when he clicks Submit.

There are also automated frameworks you can use to script automated tests, such as Watir/WatiN. (Please don't ever waste money on QTP.)

And as others have mentioned there's always load testing and stress testing you need to do at a certain point.

Thanks for posting this. I will definitely add this to my ever-expanding lit review for my thesis on SQL Injection Attacks/Web App Development Frameworks

I've been in a few web projects and all had a testing phase just as any other projects: creating test plans based on the requirements and specification, doing functional testing, integration testing, load testing, and so on.

For what is worth, most of these applications were internal apps for large companies (users are the employees, not customers) and might as well have been developed as desktop or client-server apps. I see no reason why the choice of architecture/deployment would affect the QA policies. It might affect the particular way of testing (tools, etc) but not whether to test or not.

I can't see quite see why is this even a question. Perhaps your colleagues are assuming that all web apps are like the typical apps they might use: B2C, free or "freemium", etc. Other things being equal, these are more likely to be released without careful testing.

Rails comes with automated testing baked in, and it's one of the best unit/automated testing packages I've ever seen.

Then...people went and made it better with things like RSpec, Shoulda, Factory Girl, and other enhancement packages.

I've given up trying to convince others to do automated testing, but for me and my projects, there's really nothing that beats what Rails and the Ruby community provides.

There are many types of testing, almost all of which are useful. From a client perspective, usability testing is probably the most important. Having a testing group come in, use your app, and then give you immediate feedback is invaluable.

There is also penetration testing to check you web apps security. Could you maybe be more specific? I think it would help you get some great answers from the community.

One of the many web apps I have worked on do something similar to the following:

before any feature rolls out each feature (from a list) is tested with different (but not every possible) set of inputs. This catches 90% of problems.

The other 10% that get past our testing are either sent in by the user through a feedback from, or by the system sending messages (errors warnings etc)

I've used Selenium to run unit test along with various flavors of Watir. Watir requires much less overhead and is programmable in Ruby. Selenium has libraries in all your favorite libraries, but requires running a Java proxy server. However, you can automate tasks by having Selenium record your actions on a page, which is super nice.

Selenium 2 does not require the Java proxy server. That was one of the top complaints with Selenium 1, so we got rid of it. :-) The Selenium 2 Ruby bindings are very good these days and have just as much overhead as Watir (that is to say, not much).

Jmeter for load testing. Selenium for regression testing. Penetration testing is also a pretty vital aspect if there web application has access to sensitive data. Web applications should be tested like any other applications.

Graduate students stating that "no one tests web apps" are showing why they are still called students.

I've just started using Django testing framework to do unit tests for my app. So far it helps catch a lot of errors that would have required more tedious manual testing, which is what I did previously. It sounds like Selenium would be a good way to test the javascript aspect.

IMO Biggest bang for your testing buck - cucumber with webrat

http://railscasts.com/episodes/155-beginning-with-cucumber

I use QUnit to test my JavaScript and whatever's on hand to test backend code (unittest in Python, Test::Unit in Ruby). I'd use Selenium for point-and-click-style testing but I haven't gotten to it.

click, click, click, mouseover, click. So tedious.

sounds like you need the Selenium Firefox plugin http://seleniumhq.org/

... I had no idea! This is awesome (Its like discovering Firebug for the first time)! Thanks!

Um, and if you like that, try the free enhanced version from Sauce Labs called Sauce IDE which records videos of your tests running in cloud space. Makes troubleshooting way easier. (I cofounded Sauce with @hugs)

Never knew that tool. Their Video need to be improved because the product seem nice but they really do not sell it well :P

Reply on news.ycombinator.com