right but most of those men weren't samurai, were they? The Samurai were basically responsible for raising a private army in times of war like that, they were the first point of defense if someone threatened their people in peace times but they also brought fighting men along when their lord needed it.
Feudal social structures are just very counterintuitive to us. We think of is as being a job but it was a lot more built on personal relationships than that, and a Samurai's relationships with his inferiors depended largely on effectively scaring away unpopular criminals. That's why I say it's a role more like a police officer's than a soldier's, but it was really quite unlike either.