Roads are engineered for a particular speed and flow of traffic. If you exceed the design limits of the road, you may
feel very unsafe and be quite inclined to slow down again. This applies even to large, wide, noticeable emergency vehicles. Residential streets often participate in noticeable "traffic calming" measures, like roundabouts, speed humps, wasp waist narrowing, landscaping and trees, signage, etc.
Most of the time when I see emergency vehicles plowing down the street, they are pretty much going at the limit, but they look faster, because all the traffic signals favor them, and all the other motorists have pulled out of the way. But a fire truck has still gotta slow down or stop at a busy intersection or just a turn. Pedestrians are a thing. And scooters, and cyclists wearing headphones...