The SWP is not massive. It's never been massive. Current paid-up membership is maybe four figures.
It's considered eccentric, comically cliched, ineffectual, and frankly irrelevant by virtually everyone on the British left.
It's the very definition of fringe.
Which is why banning the SWP sets a very bad example. Especially when there are far more dangerous organisations on the British far right which Facebook seems happy to host.