I was just about to say that same thing - any time I've seen wormholes in sci-fi (I'm reading Peter F Hamilton right now!), you go through in an instant (for all observers).
It's like sci-fi uses wormholes as a practical alternative to travelling at sub-C, relativistic speeds.