I've been disabling HT for security reasons on all my machines ever since the first vulnerabilities appeared, with no discernible negative impact on daily usage.
And like you said, the vulnerability consideration makes HT a hard sell for workloads that would benefit (hypervisors).
That i7 is a huge downgrade from what I was running before (long story) so I'm looking forward to Arrow Lake and I like everything I've read. In addition to removing HT, they're supporting Native JEDEC DDR5-6400 Memory so XMP won't be necessary. I've never liked XMP/Expo...
RAM fetch latency is what happens on CPU level.