As someone who lives in San Francisco and commutes using all three of bike, foot, and car (and back in college rode a skateboard around a large and busy campus) I don't think skateboards belong on the sidewalk - and definitely not motor powered ones.
Anything that has the potential to move much faster than a person running at full speed is extremely dangerous on the sidewalk for everyone involved: cars, bikes, and especially pedestrians. It breaks everyone's mental model for what could possibly be coming from where. Cars making a right turn do not expect bikes or skateboards to entering the sidewalk at 5-10 mph from their blind spot and so they almost never check. Pedestrians turning a corner don't expect the possibility of something oncoming that is moving too fast to stop before hitting then and so they almost never look. And so on..
As an individual YOU might be a considerate skateboarder who has learned when and where to be careful. That's great, but it doesn't change the fact that there are still idiots and assholes out there that are going to do idiotic and asshole things. The only system that makes sense is: fast-stuff in one place where everyone knows to look out for fast-stuff, and slow-stuff on the sidewalk.