Even if you don't take them seriously, they can still do a lot of damage. Even ignoring the psychological elements of being constantly denigrated, these groups can usually drive people off of Twitter by sheer mass of tweets.

"Just block them", it's easy when there's one or two accounts being dicks. It's much more difficult when 4chan's recruited 50,000 accounts to be dicks.