Radiation damage is a game of statistics: A single charged particle, electron (whatever) striking the right molecule in your body may be enough for you to develop cancer and die in a particularly nasty way. The chances of this singular event happening and at the same time all safeguard of your body failing are minuscule, though. But the more events you have (the higher the radiation), the higher the chances that it will happen. This is what we basically know about radiation: it damages cells and for some events the natural cleanup doesn't work. Basically you could say that every bit radiation is harmful, even the natural background radiation [1]. All guidelines and limits are based on the question whether the increased risk outweighs the possible gain. Early X-Rays had the problem of exposing the doctors to high doses, modern machines are designed with safety in mind. Still, patients and doctors wear protection when possible.
[1] to make matters worse, there's different kinds of radiation: alpha rad which doesn't even penetrate the skin, but is very harmful if dust is inhaled to high energy gamma rad that is basically impossible to shield without lead or other heavy metals around. So "1 millisievert" doesn't even tell you what it is, 1ms of alpha is probably better to handle than 1ms of gamma rad..