Even at a distance it should be okay, sounds more like a peering/bandwidth issue between your ISP and his