I could be wrong, and they both
can work if extreme caution is applied. My general take is that they make it way too easy to do the wrong thing though, at very little gain (assuming other parts of your system are setup properly).
---
For nesting, the problem with nesting is that is messed with specificity way too much. Since adding an extra class inside a nested block is trivial, people start adding more and more. But even just 2 classes is already too much in my opinion. Ideally each component would have classes on each of its inner elements, so that the component-level styles always have a specificity of 0010:
<div class="textfield">
<label class="textfield-label"></label>
<input class="textfield-input">
</div>
If you have a setup like that, you never need to nest because your CSS ends up looking like this:
.textfield {}
.textfield-label {}
.textfield-input {}
And that way all of your selectors are minimally invasive in terms of specificity. The problem with nesting is that you end up with selectors that have a specificity of 0020, which means anyone implements those components needs to be sure to achieve that level of specificity or greater to override them.
But honestly, when working with small components there just isn't that much typing to justify the possible downsides. Sure I have to type out the extra selectors in the beginning a few more times, but its worth it to do without the unintended consequences of the magic.
The one case where nesting really does come in handy is for page-specific or app-specific stylesheet files. Places where you are adding the "glue" CSS that should always be namespaced to avoid collisions.
---
For mixins, I feel like they only seem useful if the components you're using are broken into small enough pieces to begin with. For example, if you want a complex gradient system on your buttons, put that inside your button styles and then don't touch it from anywhere else. If you're constantly needing to use a mixin to re-define those styles, something else is wrong. Because realistically how many different color variations of those buttons do you need to make? It should be somewhat limited, and they should be able to exist in a single file without issue.
Where things really get screwed is if you aren't versioning your mixins (which most people aren't doing). At that point, you're just tightly coupling changes to that mixins across your entire codebase. If you want to change the way the buttons implement the gradient, with the mixin system you go change the mixin, but now you need to visit every other component that uses that mixin and double-check that you didn't break anything.
Instead, if you didn't use a mixin and the component just contained its gradient logic (slightly more verbose maybe, but way more maintainable) then you don't have that problem.
The other solution is to strictly version your mixins using something like Component and maintain your mixins in a separate repository on GitHub that gets versioned, then that also works because each other component can peg the version of the mixin it is using.
---
My big takeaway from CSS though is that none of the solutions for it are nice right now. And most of these problems should disappear (I think) with proper native extension.