How come I haven't experienced these downsides?
At the very least support of nesting is a huge convenience, but in practice I've also found that it saves me from all sorts mistakes, whether it's because I'm being stupid, or refactoring things.
That said, overusing nested selectors can be a problem too. I generally try to limit myself to two levels of nesting (component -> element), and then a third for pseudo-classes (:hover, :selected, etc.) and pseudo-elements (:before, etc.).
I just feel that nesting (which Sass encourages) leads to complexity and fragility.