It doesn't get merely dusty unless you live somewhere very dry. I live in the PNW and it rains a good portion of the year. The car gets absolutely filthy during the rainy season.
If my shoes were that dirty, then heck yeah I would clean them. What a mess, it's easy to get your clothes dirty when the outside of your car is filthy. My kid's white dobok is especially sensitive to dirt on the car.