That's for plugins with an explicit fixed latency to use PDC. If a plugin doesn't request latency, which most do not, they have the duration of the DAW audio buffer to complete their processing. If the combined processing of all active plugins on a single thread exceeds the audio buffer duration you get audio drop-outs, or portions of the buffer which are just zeroes because they were unable to be processed in time.

I wrote a DAW (https://ossia.io) and a few dozen plugins (actually 150+ if I count the various small tests and examples in https://github.com/celtera/avendish) and I can assure you that most plug-ins don't add latency.

> There is no way eliminate CPU cycles being spent on whatever the plugin does.

that's not how DAW works, they don't output audio immediately anyways, everything is buffered at the driver level or just above so that there's always 1 or 2 buffers of delay between the input and the output. There's no difference in terms of latency between putting one or 50 bitcrushers in series for instance because in the end the audio main loop looks like (in a very simplified way):

    void process_soundcard_buffers(float** in, float** out, int samples) {
        for(int i = 0; i < samples; i++)  // samples would usually be the buffer size you set in your audio config
          out[0][0] = f(in[0][0]);
    }

where f can be one distortion, or 50 distortions - what matters is that their processing time is < than the time you have to write the output and if so you'll always get the same fixed latency. (And if not so, you don't get delayed output, you get crackles because the soundcard will be reading its buffer whether you wrote something meaningful in it or not)