The major problems I've seen with frameworks is that people just use the default look or try to add customisations on top of the default CSS, overriding styles. I mean, that is just setting yourself up for a world of pain.
But most CSS frameworks are really, really flexible in the build stage. You can tweak a few variables that can dramatically alter the appearance of a stylesheet and give you an awesome base to build on. Pretty much for free.
If you do all the CSS from scratch you're anyway going to write at the least a reset, some base typographic styles, responsive stuff, and some kind of grid system. Why not use something that is widely used, peer-reviewed, and very customisable?