At some point, the amount of CSS complexity becomes best handled by a preprocessor.

Stylus is great: http://learnboost.github.com/stylus/

I made a post about it, and personally prefer it to Sass and Less: http://nylira.com/stylus-the-revolutionary-successor-to-css/ The hardcore abbreviation mixins in my post seem to offend some coders. When you work in CSS and HTML hours every day though, every character saved adds up to a huge productivity boost.

My current project has 30 directories with 73 partial Sass files, which compile down to one 320kb file.

As to your question, I organize my partials based on controller and url structure.