I've found the most difficult thing about riding an e-bike is the other motorists have no idea how to react to you. You're not really a regular bicycle anymore due to your speed, but you're also not a motorcycle that deserves its own lane. I have at least one car turn in front of me almost every trip out just because they're misjudging my speed. I get honked and yelled at when on the road because folks get frustrated when I'm using the left-hand side of the right-turn lane as a bike path.
Sidewalks/bike paths tend to be a lot less safe in residential areas as well, since cars coming out of their driveways really don't expect an e-bike to come rolling through. I've learned to dramatically reduce speed in areas like this.
Aside from those things, I love it! I ride the e-bike whenever I'm going somewhere in range (I live in Florida so things tend to be spread out) and the weather permits. My bike gets about 80km which is more than enough for anyplace I want to go on a bicycle anyway.