No, you have to look at the margin. The other 99 people are already there, and you have to choose whether or not to drive; you slow down everyone after you, but you don't slow yourself down.
In some situations the margin of newcomers is a fair way to look at it. But with drivers on the road, these people are all getting up in the morning and choosing to use this road at the same time. It is a continuous reaffirmation, day by day, that this is the path they want, no matter who they slow down, and they accept being slowed in return.