Most drivers ignore them, but it only takes 1 driver going at the 20mph limit in a bottleneck to slow down a whole column of traffic. So both are true.
Let's, for the sake of argument, say I am one of these annoying people. I will drive at 20 mph with a tail of 6 angry drivers behind me until I reach a junction or a set of lights that invariably already have cars waiting to move forward.
All I am doing is delaying the drivers behind me from reaching this queue of cars a few seconds later than they would otherwise.
Of course, their rage is directed in my direction but that's another story.