In the city, you are correct. The far right in a 3 lane is often just for turning and merging, making the middle the travel lane. The point is not to travel in the passing lane.
The chief component in getting to one's destination is the traffic and the second one is number of lights you get stopped out. Since all the lights are timed to 40mph it doesn't make sense to go faster unless they re-engineer the road.
Generally any car that passes me will be stuck at the light ahead and if there is light traffic I will pass them because I have a rolling start and if there is heavy traffic it's all the same.
Staying to the right in this situation only makes sense if you're unable to accelerate at a reasonable speed or unable to go the speed limit at all.