Sure, it’s not that bad from an absolute perspective, and probably no one was hurt by it, but the public generally expects individual members of the government to abide by the law no matter what.
The problem with the CA recall is that the only major opponents had a very different political position to him, but switching the state’s political control over a personal scandal seems absurd.
With that said, I think there's a hole in your reasoning -- If the people who enacted these laws had found you breaking them they would damn well have called you a criminal for doing so.
Johnson (and the UK treasurer) were both issued a fine for disobeying the COVID restrictions, while in Number 10 (and 11) the official residences.
They offended numerous times, including immediately before a press conference to deliver more onerous restrictions, and before the funeral of Prince Phillip, where the Queen (like many others) was mourning alone.
The Queen wasn't required to keep the COVID restrictions (she is literally "above the law"), but she dutifully obeyed them.
It's not the nature of the crime, it's the complete lack of remorse or even the perception that there was any harm.