For starters, intent is factored into the law. There's several different classifications of murder depending on intent. Likewise someone carrying a kitchen knife home, still packaged, from the shops is unlikely to be reprimanded compared to someone carrying a more decorative knife around. They're both knives but one instance clearly carries a different intent to another.
The problem with youtube-dl is that having tests which work against copyrighted content and having the README describe usages against copyrighted content, it's much harder to argue that the intent of this is purely for copyleft content. Your point about this being an access tool (which actually makes no difference in terms of circumventing DRM anyway -- which was the claim for the take down) also doesn't fly because this tool creates a file on your local disk so it's hard to argue that the intent is for that file to be temporary.
I'm not saying I agree with the take down notice (I don't) but something a great many techies on here miss is that not every argument can be won with science.