Many pigments past and present a pretty toxic and always have been unsafe to handle. Of course it is also worse when old materials degrade and start shedding into the environment to be picked up or breathed in by people.
>presumably, there was a point when we stopped using these dyes
The discovery of toxicity and phasing out depends on the individual chemical, not really all at once.
In the past, it was often quite widely known how toxic these things were and they were used anyway.
Paris green was used as a rat poison, insecticide, and a pigment all at the same time, factory workers were often poisoned with it.
Today we are really paranoid about these kinds of things, sometimes maybe a little too much, but the comfort levels with known dangers in the past was surprisingly high.