I've experienced this behaviour. During the middle of the day at an intersection of a side street with a busy street it can take a long time to turn. At night when the busy street isn't so busy, it seems to often turn relatively quickly.

Are you waiting 20 seconds for the cross light to turn amber or 20 seconds for your walk light to come on? Because the latter is always going to take at least ~10 seconds -- the stale green time + the amber time + all-red time.