Shouldn't the same logic apply to a pedestrian? I mean his goals are getting to some arbitrary location and arriving at that location intact.

Why should only the driver carry the responsibility? Driving slower than average increases the driver's chances of getting in an accident.

Also, lower speeds aren't guaranteed safety. I know of an accident that occurred around 10-20km/h right next to a hospital, which resulted in instant death. Pedestrian was crossing on red, behind a car parked at the traffic light and flew right into a path of a vehicle, trying to pass the car parked at the semaphore, this bounced him just enough to fall and crack his head on the curb. Ironically, according to policemen, was that had the driver driven faster, he might have bounced pedestrian harder, and not kill him on the spot.