But of course, even for definitively "toxic" things, one must differentiate between exposure channels. I wouldn't care if I handled a piece of Greenrockite [1], but I wouldn't want to breathe the stuff in powdered form. Same with Cadmium glazes: orange pottery doesn't concern me, but I'd want to be careful if I were handling Cd-containing powdered glazes. You don't want your dry cleaner dumping used methylene chloride in the river, but it's commonly used in decaffeinating coffee.
The reason the author won't work with this particular compound isn't the fact that it contains Cadmium, but rather, that this particular compound has nasty tendencies, in addition to being toxic, that make it particularly dangerous.