1. Large web applications will eventually demand this of you to break the large solution into smaller chunks (components/modules).
See this methodology: http://coding.smashingmagazine.com/2012/04/16/a-new-front-en...
SASS will help you follow whatever standard/methodology your team may be following.
2. And having multiple files will ease development on a large team. Having multiple frontend developers checking in code, it helps to organize, and alleviate one developer stepping on another when checking in SASS files into source control. No conflicts, if developer A is working on the footer, and developer B is working on the header, which are separate SASS files.
I don't agree that it slows down refactoring, and would argue it speeds it up. In the end, the multiple SASS files will probably (depending on how you do it, and your situation), compile down into one file. SASS also has options that allow you to say what the output looks like too (nested, expanded, compressed).