I like the code for Skipfish a lot, but neither tool is particularly important in the profession; if you're working with a contract application security firm and they give you findings that came from sqlmap or Skipfish, you should be irritated.
The latter I agree with, the former I don't. Any automated security scanner has its shortcomings - combining the results of various scanner products generally has a positive effect upon the quality of the test. I've often been in a situation where one tool (commercial or otherwise) indicated or hinted at a vulnerability while all other tools (commercial or otherwise) used did not indicate anything.
But even practices that do use automated scanners won't be relying on Skipfish or sqlmap.
This obviously doesn't hold for network pentesters. For tests in which application flaws aren't the whole story, and you're just looking for the fastest way in you can find, sure; sqlmap is valuable there, much more so than Skipfish. We don't do those kinds of tests often.
I find myself writing a lot of custom fuzzers in python, and am looking to improve my toolkit.
Another faction of our team uses Mike Tracy's WWMD toolkit, which does some of what Burp does but from an irb prompt.
I'm allergic to both approaches and tend to just start from EventMachine and ev-http-request.
Good example of the kind of scenario that'll get me into "writing code to test web apps" mode: testing those goofy cryptographic tokens applications send via email to reset passwords.
Yes: the better approach is to have the token be a 256 bit cryptographically strong random number that corresponds to a row in the database with metadata sufficient to expire the token and invalidate it on password reset.
Have you looked into Haskell's QuickCheck? It's essentially a fuzzer for unit testing. You might find some interesting ideas there.
What it seems like you're saying is that you have found that scanners are almost always a distraction at best, misleading at worst, and that they offer zero useful information about the app or about the MO of the app developer.
You (matasano) are good enough that any information those scanners could reveal about the app or the developer, you'd probably turn up independently. Not everyone has those skills, and the results of a scan might include something that a lesser practitioner wouldn't have noticed any other way.
I'm not against automation. I don't think testers need to launch every fiddley little query they need to gauge behavior. What I'm against are tools that run some unspecified number of fiddley little queries, do some bogus analysis, and then spit out a "YES XSS" or "NO XSS" answer.
but I'm going through the exercise of building out an app sec practice, and for what it's worth, I've adopted the same approach (we're not using automated scanners).
I work in an AppSec practice. Scanners help me find the low hanging fruit. I can now spend more time looking for logic bugs.
Reasonable people can disagree on this point, but we're a pretty large, well-established practice and our belief isn't coming out of nowhere.
Your argument is literally the first thing anyone who wants to convince us to use scanners brings up. It is the point we've thought about and debated most. I just don't think it pans out in the real world:
(1) The bugs you find "because" your scanner took the low-hanging fruit will be bugs any good tester will find;
(2) meanwhile, the extra scrutiny you're not giving the app to find that low-hanging fruit is costing you insight that would reveal still more bugs...
(3) also, your scanner is missing bugs, probably in the neighborhood of 20-30%, and:
(4) you're not making up for that because it's very difficult to force yourself to focus on terrain that a scanner has covered and flagged bugs in.
I am not as philosophically opposed to scanners as I think Thomas is, I've just found that they provide nearly no useful value. For many years, I argued that although they provided no value, "they didn't hurt", so there was no harm in also running them at the end of a test to make sure there aren't any low-hanging fruit that a tester may have missed.
What started to turn me around in that belief was that I noticed an increasing number of tests being performed by my team where the scanner wasn't just not providing value, but actually causing issues.
Under the best case scenario, you are now having to take the time to validate your scanner findings (which are all things you would/should have found anyway but are relying on the scanner to do for you).
Under the scenarios I've witnessed play out, people assume that the scanners will actually find the low-hanging fruit, and they slack off on that part of the assessment (because, hey, the scanner will cover it, and now they can spend more time looking for logic bugs). Then the scanner doesn't find something trivial (which happens about one in every...oh, I don't know...actually it happens in nearly every test).
I'm happy you've found that scanners don't make your work product worse, but that's not what I've found at all.
EDIT: Furthermore, how embarrassing would it be if someone hired you and then ran one of these scanners themselves and did find something you missed? Given the magnitude of the potential downside, and the marginal cost of using them as described, it'd almost be more of a business insurance tactic than anything else.
Last point first:
Try real hard not to miss stuff. Seriously, that's it. Appsec is a competitive field. Forget scanners: many clients are going to hire a different firm for their next test, and some of those firms are really, really good. When you're testing an app, that is the thing that is animating your work: trying not to leave nuggets for the next team to find and shame you with. Think we're worried about missing things sqlmap finds? Try worrying about what iSec Partners is going to find.
To your first point: this is hard to articulate well, but let me take a stab at it: it's very hard to double back over terrain a scanner has already covered. Think of security testing like a treasure hunt. Have you ever had a fun treasure hunt? Can you get a bead on the feeling you had when you started out hunting for stuff? Imagine trying to put the same focus and motivation together if, before the treasure hunt, the organizers announced "we sent a bunch of people out before you to try to make sure there's no treasure".