It's not the electric field; it's the magnetic field. There is still a potential, but the potential is a vector.

My ability to picture what is going on has never extended to the magnetic vector potential, so I have no intuition about how that plays in here...