I will concede that the current UKGov is less white male than it has been but UK politics has always been much more progressive in that regard.
> You can't judge the past by today's standards.
In some aspects, absolutely. In things like "did they invade, subjugate, enslave, and pillage?", you absolutely can. Just because they didn't have porcelain toilets and lightbulbs doesn't mean what the British Empire got up to was ok, for example.
> every single culture on Earth would've done exactly the same if they had the chance
Not exactly a justification.
> Saying that white people are punching down is racist, because it implies a hierarchy where white people are on top.
In terms of political, cultural, etc. power, they are, that's the point - you don't pick on people with less power than you (see also: British Empire.)
> you don't get to judge people for staying out of this shit flinging contest going on in today's politics.
Of course you do. "It has been said that for evil men to accomplish their purpose it is only necessary that good men should do nothing." (Reverend Charles Frederic Aked, 1916)