Plutonium is, in itself, not all that toxic - it is significantly less toxic than many other heavy metals, and can be handled safely. I wouldn't suggest going out of your way to inhale plutonium dust - but that holds true for such large swathes of the periodic table such that it seems somewhat silly to base a power generation strategy on that alone.
It is of course radioactive, but that then falls into the problems I described above (the tradeoff between not-that-dangerous and not-that-long-lived). However the main reason people complaining about plutonium dust sound rather silly to me is this: plutonium is extremely dense - almost double the density of lead - and thus encouraging it to float around the atmosphere isn't all that easy. Unless you have someone machining doorknobs out of Pu-238 in your bedroom (in which case I would strongly urge you to find different housemates) then inhaling plutonium dust seems rather low on the list of things to worry about.