It's inertia. Among other things, the much higher inertia of the broomstick has a damping effect on changes in its motion.