I don't think you can make rules based on potential speed, and your division is far too simplistic and arbitrary in terms of speed. While I agree my skateboarding is limited to 5-10 MPH (jogging-running speed), is that really that fast?
If I (and my skateboarding kind) should be in the street based on speed or potential speed, then so should rollerbladers, scooters, segways, motorized wheelchairs, anyone able-bodied person (as they have the potential to start running).
It's a complex issue and courtesy and being alert go a long way... as well as knowing the law. Wherever I'm supposed to be by law (NYC law leans toward the sidewalk) is where I need to be, and other people need to respect that.