In addition to your point, inbreeding isn't necessarily a problem if you engage in selective breeding. (Sterilizing and euthanizing animals with fatal recessives or health problems) Genetic testing has made this a much more effective solution. With in-vitro testing and implantation you can clear these problems before the animal is even born or implanted.
The problem is that a lack of diversity can lead to issues with adaptation to new environments and pressures. This is what has made a lot of these species endangered in the first place. They are adapted for very specific conditions which are suddenly changing due to habitat destruction. (Deforestation, urbanization, water works, farming)
A typical population has a lot of mutant lines that do okay in the current environment but may be more adapted to different environments. A population of clones will take time to develop these and may not survive long enough to adapt. Also if there is a problem with the current line and you don't catch it fast enough you are hosed.