Handyman here. You're largely correct.
The siding is absolutely no big deal because it doesn't release fibers unless it breaks. Because it's solid and brittle, when it _does_ break, hardly any goes into the air. Plus you'd be outside, likely protected by at least a gentle breeze.
Asbestos was used in floor tiles (generally on top of concrete), these are perfectly safe if kept waxed or with a clearcoat applied on top. But you'd want to wear a respirator if removing them.
There was asbestos insulation in attics and often between walls, or wrapped around pipes. That's the worst stuff and you should use a respirator just being around it. (I'd wear a respirator in any attic, to be honest.)