The tower still has to ping the rest of the network to find the service you were requesting to in the first place.
If total latency ~ 500 ms, phone-to-tower is maybe 50ms of that in 4G. Even if it's down to ~1ms, total latency is only down 10%.
Not a networking expert so maybe I'm missing something here. Perhaps better use of CDNs could make this much faster?