This is the main reason why by default the device disables the RF source, you have to go out of your way to enable it by editing an /etc .conf file - however it actually does make a slightly better random stream
With the assumption that it only receives, I would strongly prefer that it be left on by default. Adding a second source to the CRC mixing can only improve entropy, even if the source is controlled by an attacker. Since the noise from the zener has no simple proof of it's entropy output rate, and it can vary over time, and because zener noise TRNGs have been known to be power-supply noise sensitive (not that your's is), having that second source gives me much more confidence in the device.
Turning both RNGs on only gets you a about half a bit/byte of extra entropy - but it is a great belt and braces sort of thing, makes it much harder to attack, especially since the sampling clock inside the device for both sources is not visible eternally
Remember we don't use the output data directly, we mix it into the kernel's entropy pool (where it's whitened)