The 3 failures (
http://en.wikipedia.org/wiki/List_of_Falcon_1_launches):
1. Exploded 25 seconds after launch, rocket parts are still washing ashore
2. Successful transition to second stage, but didn't deploy its payload properly. 1st stage has not been recovered
3. 1st stage engine was 'more efficient than expected' and continued to burn after separation of 2nd, bumping into it. I got to see a video of this at a conference, very sad. But impressive that the 1st stage was able to generate thrust (pressure in its fuel tank was rediculously low). Mostly I was relieved because my satellite was almost on that launch, and now still has a chance of going up.
I wouldn't consider a launch delay/abort a failure in any sense, especially not due to design or related to safety. Every rocket is designed to work within certain parameters, and when those aren't satisfied, the launch is aborted. This can be due to weather (completely out of human control) or malfunction (not the designer's fault). In the case of a design error, operators usually don't know what faults to look for, and then there is a failure.
You bring up an important distinction, though. There is a difference between the safety of a design, and the safety record of its implementations. In the quote in question, Musk promises it will be the "safest rocket ever designed" which doesn't mean that he's promising the safest record, or that it will have the same 100% success rate as other rockets, but that it will be designed to be safe for the astronauts. Already it has a safer design than the shuttle, since it is using liquid rather than solid fuel. That means a launch can be aborted mid-flight, and the crew brought down safely somehow. The shuttle required a man on the ground to be ready to hit a 'kill' switch.
Edit: turns out that the top comment here wasn't actually quoting anything when he put "safest rocket ever designed" in quotes. Musk said "safest, most advanced crew vehicle ever flown," which I suppose could be interpreted as a promise about its as-flown safety record. I still don't see the harm in him saying that, since he's also said that he accepts a non-zero risk of casualties. It's not like the families of dead astronauts are going to sue him over this quote.