Just select and split off all the text to a new layer, then apply the distortion to the plot layer and not the text.