I think that's a vocal minority. The majority doesn't care, and two equally small, vocal minority groups love and hate him, respectively
Source: https://www.pewresearch.org/fact-tank/2022/05/05/10-facts-ab...
Considering all the attention the acquisition received I was surprised to read that more people don't use Twitter. The idea that what gets posted on Twitter automatically becomes what the average Joe talks about is probably overstated.
And about 1 in 4 or so care.
The increasingly pubic tawdriness of his love life was a factor too; prior to then he'd done a good job cultivating an image of a family man.