I'm a _somewhat_ frequent flyer (5-8 trips a year). I've never experienced a plane being delayed by the time it take passengers to enter/exit the plane. I have, however, experienced delays because the baggage handlers are still loading the plane.
For that reason, I've never understood the obsession with loading the plane quickly.