I found a number of security holes. For example, the "files" resource was open to arbitrary directory traversal, so I could download any file on the system. (I reported it. It has since been fixed.)
One of the files I could download was the configuration file. It contains a hashed password, but the hash algorithm was trivial to break. Within a couple of hours I had a program which would generate a string that generates the same hash value. (This does not appear to have changed.) Schneier once wrote "There are two types of encryption: one that will prevent your sister from reading your diary and one that will prevent your government." The fitnesse.authentication.HashingCipher is of the first type, which is likely good enough for the purpose. However, something like <salt>+MD5 would be at least equally secure, shorter, and easier for others to understand.
Once I had a password to upload a file, there was another directory traversal error which would let me place a file anywhere on the file system. (This has since been fixed.)
There are still design errors. For example, to delete the file "foo.txt", do a GET request to http://localhost:8080/files?responder=deleteFile&filename=fo... . While FitNesse itself uses a POST, the handler can take a POST or a GET. This would easily allow someone to construct a malicious page which secretly wipes things from your server.
If you are Martin then don't follow this link: http://fitnesse.org/files?responder=deleteFile&filename=head... ;)
It's also still open to cross-site scripting. For example, http://localhost:8080/files?responder=%3Cscript%3Ealert%28%2... gives a "hi!" alert. Thus, if I can get a FitNesse admin to click on a well-crafted link then I can deface the site.
My conclusion was that TDD doesn't lead to any better security design than a non-TDD project. If your production code requires security then TDD isn't enough. My question to TDD people is: what other design techniques should one use in order to have a secure design?
So far I haven't gotten an answer.
I also noticed any number of places where the server, which claims to be an HTTP/1.1 server, doesn't actually follow the HTTP specification. For example, FitNesse does not include a "Date" header in the response, but section 14.18 of the spec says "Origin servers MUST include a Date header field in all responses, except in these cases", none of which include FitNesse. More esoteric, 5.1.2 of the spec says "all HTTP/1.1 servers MUST accept the absoluteURI form in requests, even though HTTP/1.1 clients will only generate them in requests to proxies", but FitNesse assumes the first character of the Request-URI is a "/", so accepts "GET $ HTTP/1.0" as a valid request line.
These are not critical details, and my own HTTP servers don't follow the details of the spec. For most people, the only important test is that the browsers can understand it.
My point here is only the observation that TDD doesn't give a better mechanism to ensure that the software design meets an external specification.
Strangely, it might be the case that programmers who have used some form of red-green-refactor TDD have realized its shortcomings, and are not eager to present that publicly, and also recognize that it cant be set in stone as the one true method to write correct code.
Maybe its time I stopped trying to research more about TDD, and just conclude that what works for writing correct programs is what has always been the workflow:
1. Spend time analyzing the problem.
2. Come up with an algorithm based on similar problems or something that makes sense to the programmer.
3. Ensure that there are tests to verify it.
4. Implement the algorithm(s).
5. Change code and tests iteratively based on more analysis.
"and are not eager to present that publicly"
I think http://lizkeogh.com/2012/06/24/beyond-test-driven-developmen... is a good expression of that. The author used TDD as a way to understand how to develop testable software, but then writes "Here’s a confession: I don’t always write my tests first. I often don’t automate scenarios up-front either. And I consider it right for me not to do so."
I found that page because I wanted to give an example of "Spike and Stabilize", which is an alternate to TDD more along the lines of what I do.
Your question has inspired me to write a follow-up to that 2009 essay. The main point will be that TDD is not that appropriate for non-functional testing. Beck's original definition is that everything can (and should) be structured around TDD. There are any number of quotes which use a weaker point; use TDD plus other techniques, but not TDD alone.
For just one of many examples, see http://disciplinedagiledelivery.wordpress.com/2012/10/23/str... .
That link is interesting because it highlights the importance of a QA group in agile, but calls it "parallel independent testing" or "test team" rather than the traditional name of "QA".
But thats exactly my point. The workflow for solving a problem and making sure its sufficiently correct cannot be pinned down to something as simple as red-green-refactor. Programmers have to be given the flexibility to approach problem-solving in different ways. As long as the program is correct and performs as stipulated by requirements (both functional and non-functional) it doesnt matter how it was arrived at.
Eagerly awaiting your follow-up writeup. And if you do hear of some open-source projects that have used TDD successfully, please mention them.
My goal in asking this question on HN is to have an open mind about TDD and learn to do it well. But I dont want to add that skill if it has not proven to be successful. IMO, there are lots of other skills that I could add to my toolbox that I think are more useful as a programmer.
The useful skill I think should be practiced more often is of breaking code. Take a code base and find out where it doesn't scale, where it core dumps, where it sits in an infinite loop, or where it lets you write read and write arbitrary files to the file system.
It's pretty fun, but it's a mindset contrary to that of most programmers, who are optimistic and goal oriented, rather than pessimistic and defensive like me.
In my essay I'll ask for pointers to public projects developed in full red-green-refactor style.
Cheers.
https://github.com/TryGhost/Ghost/
Choose any other random OSS, chances are it's using TDD/BDD in one form or another.
"TDD/BDD in one form or another". Theres the rub. That form mostly doesnt involve red-green-refactor and make the algorithm evolve from their tests. A lot of famous, widely-used open source software have tests, but doesnt mean they used red-green-refactor TDD. If you read my comment below the title, I am specifically looking for software that used the TDD method as prescribed by agile methodologists.