Rather than filter in brands, I'd rather filter out brands (aka exclude). Maybe just an include/exclude toggle there? A select all UI element, then deselect might also be adequate.
I think your toggle idea is a good one though, and I'll look to implement that. I can see how some people might want that.