All good so far, for (say) HPC where you have people with guns guarding the machines you can turn on all the performance. The issue is that as CPUs have got more complicated, it's increasingly easy to attack the internal state of the processor. This is already possible in a single thread (i.e. the prototypical spectre and meltdown implementations), however imagine what you can do next door to a completely different "core". The processor maintains separation of structure but not of state, so you can use (say) timing side channels to extract information about what your neighbour is up to.

FWIW these issues are more of a problem for Cloud vendors than you or I as they usually require pretty specific knowledge of the hardware being attacked and even then are not easy to pull off.

In practice this allows some attacks to attach themselves to the same pipeline with the target process and nibble required information slowly, but surely.

IIRC FreeBSD disabled HT out of the box at the kernel level for this reason.

I'm tired and it's late here. I may worded some stuff wrong or plainly misremembered it. Please feel free to correct.