I belive it's because math is such a "clean" field that it doesn't take many people to 'clear' a topic and advance a subfield or branch into sub-subfields. So you don't have a large number of people all rooting around in the same area speaking the same language.
With complex sciences like biology, or worse, wet lab experimental science, it takes so much drudgery just to get data to work with, that you can have 100x people working in the same area, trying to cover the data/processing requirements to support theorizing and cataloging discovered entities.