In conducting his affair he was breaking the lockdown rules that he had written, which harmed their credibility. The antivax crowd (I am not a member) quite reasonably extrapolated that if the architect of lockdown was not locking himself down then he wasn't too worried about the virus, ergo vaccines were being oversold.
When you are a public figure like that then the personal is political.