Suppose you have a wormhole between points A and B. So from A to B - and from B to A - you can go by two different roads, one is pretty long (say, 1 light month), and another is very short (negligible distance).
Next, suppose B end swiftly rotates around point A, so fast that time in B goes slower than time in A. This means clock in B is going slower than in A, and over 20 years in A only 10 years will pass by clock in B. That can be confirmed by observation - telescopes in A will see slower clock in B, telescopes in B will see faster clock in A (it's B going around A, not vice versa).
That is, if we're talking about telescopes, looking through "regular" space. Since A and B are also connected by the wormhole, looking through wormhole will show us that clocks are synchronized. Over the wormhole, A and B are not moving, so clocks don't deviate from one another.
We can stop B flying around A now. Now suppose B clock has accumulated 10 years of difference from A clock - again, looking via "regular" space. We're leaving point A, flying to point B over the "regular" space and spending 5 years - way slower than speed of light, so our clock is practically synchronous to A clock. At the start of flight, A clock show 20 years, and B clock is 10 years, and by the end of flight A clock is 25 years and B clock is 15 years. After arriving at B, we jump into wormhole and get back to A, when A clock shows the same as B clock - that is, 15 years. According to A clock, we left at 20 years and came back at 15 years. Time travel.