I'm unsure of the security of simultaneous multithreading on modern x86 CPUs
utcc.utoronto.ca
utcc.utoronto.ca
I'm not qualified in computer science or programming but I could easily follow along with his explanation about challenging CPU architectural concepts. Gives me a great deal of confidence that AWS not only know what they're doing, but can communicate it well to management.
I remember disabling SMT in the mid-2000s because it was a performance degradation for some memory-sensitive code one of my users was running. I'd be really curious what percentage of workloads are not limited by I/O but still stay below the level where resource contention between threads lowers performance.
In 2002, our processors were so thin they could only execute maybe 2-instructions per clock tick. Today, we've got CPUs that hit 4 (x86 L1 cache), 6 (x86 uop cache), or 8 (Apple M1) instructions per tick.
With many problems still memory-latency bound (aka: any Java code going through blah->foo()->bar()... look at all those pointer indirections!!), it makes sense to "find other work to do" while the CPU stalls on memory latency.
The RAM supports higher bandwidth. The CPU supports more instructions-per-tick. Spending effort to find work to do just makes sense.
Apple's highly-competitive performance with processors which don't support SMT suggests that this is not such a clear win, especially given the non-trivial cost of the security mitigations.
* HT (Intel's SMT, aka Hyperthreading) wins on Blender, Parboil, Rodinia, HMMer Search, TTSIOD 3D Renderer, x264 Encoding, 7-Zip, Stockfish Chess AI, ASMFish Chess AI, Linux Kernel Compiles, PHP Compiles, C-Ray, POV-RAY, m-queens,
* HT loses on VP9 Encoding
* Virtually a tie on GraphicsMagick,
What has decreased quite a lot though is the time to transfer, which is of course stacked on top of CL. CPUs always have to fetch a full cache line (usually 64 bytes), and the time to get 64 bytes out of memory has more than halved each generation.
The vast majority of what people run is memory latency bound and in those cases using extra threads makes sense so that the explicit parallelism can compensate for memory latency.
...which perfectly describes a parallelized mat-vec-mult. Yes, that's not common in most applications, but I'd have a hard time naming a more basic operation in scientific (and related) computations.
The answer is that it is absolutely still useful since your example is niche and most software/systems can still benefit from being able to work around memory latency with more threads.
As a gross over generalization, disabling it can improve latency for workloads that are compute intensive. This is not most workloads, and almost everything will see some improvement and increase in throughput by having SMT turned on at lower utilizations.
As you push the workload beyond 60-70% utilization the wins from SMT fall away, but generally are not impacting you negatively. What happens is your processor is over performing at lower utilizations, and your performance curve bends to where it would have been without SMT at lower utilizations.
I have done benchmarks from time to time and it doesn't really matter much for my workloads.
If you are trying to prevent attacker code figuring out what code is running on another core: No. Patterns of L3 cache eviction and memory bus contention will probably leak sufficient information to have a good guess what another core is up to.
If you are trying to prevent attacker code figuring out data in another cores memory or registers: Probably, yes. As long as the core is doing something with no data dependent branches, or instructions with data dependent timings, it should be safe.
Very little code is written to have no data dependent branches, or instructions with data dependent timings. Some well written cryptography code has that, but thats about it.
Ooh I have some idea on how to use data dependent branches in your own code to leak critical information through side channels (Spectre) but how do you leak information from others' data dependent branches and instructions?
x = array[secretdata].
Another CPU, running an attackers code, can repeatedly try to load stuff from L3, and detect what was evicted and when to learn the secretdata.
Kernel considers recent Intel CPUs not vulnerable even with SMT, it's good enough for me
The default setting is "15000", but some distros change it (e.g. Arch sets it to "0" so that your system always jumps first--more performant, but less safe).
Conveniently, none of this is documented anywhere, all the other information you can find online is years out of date, and most sysadmins just accept the risk of cliff-jumping rather than trying to tune it manually.
I mean, the article is totally right of course, you shouldn't run untrusted code - THAT INCLUDES THE FREAKING HYPERVISOR.
SMT (a.k.a. hyperthreading) is kind of a security minefield, but so is out of order execution and crufty instruction sets like x86_64. But vulnerabilities of this sort tend to be hard to exploit in the wild. Not saying you shouldn't be concerned about them, but how concerned you are should depend on your threat model. If I were really paranoid and could pick I'd choose something like a low-power simple ARM or RISC-V core to do critical computation. Alternatively you could run your secure compute on any system as long as nobody is allowed to run any other code on that system or have physical access to it.
BTW I am deeply surprised that there has never been a cloud hypervisor doomsday vulnerability that has brought down AWS or some other huge provider. Back when virtualization in the cloud got big I would have bet serious money that this would have happened by now.
Edit: Here’s the apparent source of the quote: https://scholar.harvard.edu/files/mickens/files/thisworldofo...
Isn't that what Apple does with its Secure Enclave processor?
Meh, the vast majority of rational hackers will just sell a zero day capable of doing that either to the vendor themselves or to a nation state for several million dollars. I suppose they could bring down AWS for "fun", or try to short Amazon's stock and make a personal profit, but the consequences of getting caught are devastating. Selling the vuln is practically risk free and nets life changing money.
I'm not sure this is even true, so much as that it's just easier to do it some other way.
You could use speculative execution by measuring timing to extract secrets to get privilege escalation, but then you have to sit there with the CPU at 100% for ten minutes while doing timing statistics. Whereas you pull the latest 0-day off a list somewhere, and the target is going to install the patch tomorrow but hasn't yet, and that pops the system in less than a second. So people do that instead.
> BTW I am deeply surprised that there has never been a cloud hypervisor doomsday vulnerability that has brought down AWS or some other huge provider. Back when virtualization in the cloud got big I would have bet serious money that this would have happened by now.
The assumption there is that the goal of somebody who got in would be an externally visible denial of service, as opposed to e.g. data exfiltration which might never be detected.
Code wins arguments:
1. provide the exploit code that does it i.e. I go to a website and it reads a content that I have in a different browser tab.
2. provide the exploit code that while running on one VM on a host that does not have mitigations enabled reads a value of a file in a different VM.
This is why pretty much no one takes it seriously : it used to be that the fancy vulnerabilities came with a demo:
$ ./some-exploit-code
#
some-exploit-code won the argument. No one could argue that it did not matter as anyone who managed to get code executed on a target server got #. These days we just get lots of "OMG! We are all going to get pwned!" handwaving and zero demonstration that can be done in front of execs.
https://security.googleblog.com/2021/03/a-spectre-proof-of-c...
The VM ones don't really have anything to do with it being a VM. Being a separate VM just doesn't save you. The VM is still a thread running on the same core as the attacker.
The "If we play very hard with user browser we can sometimes get something out of user browser. We do not know what that something is, we need to know what kind of a system user has, we should profile profile it, etc." is not impressive as a demo. I cannot make it work under a Debian 10, Chromium and i4970K and i7-6500U. This not not Elon Musk throwing a brick and brick breaking a windshield. It is not having bricks to throw.
The demo is "Pull up website A into a tab 1. Go to website B in a tab 2. Website B says 'You have website A' opened". That's a demo.
> The VM ones don't really have anything to do with it being a VM. Being a separate VM just doesn't save you.
If that's the case Google should be able to develop an demo attack where content of a file in a VM A be readable in a VM B in a jiffy.
That doesn't mean you can't do it. It does mean you can't create a generic exploit that works against all software and all processors, instead of one targeting a specific application on a specific model of CPU. Then people say "it doesn't work for me" because they're using different code or hardware and the code has to be tailored for that, which is work, which nobody is going to do for free to appease hecklers.
> If that's the case Google should be able to develop an demo attack where content of a file in a VM A be readable in a VM B in a jiffy.
In a VM doing what? It can't just be idle. It has to be executing some code whose timing you can measure to extract its secrets, and the secrets in the address space of that process have to be useful in order to get the file, e.g. a password that can be used to sign into the VM. Then the exploit has to be tailored to that software and hardware.
The fact that nobody is willing to do this over a message board post is not proof that nobody can do it if there was a few thousand dollars in it for them. These exploits are not the low hanging fruit; that isn't the same thing as being impossible.
So they are absolutely positively irrelevant in the real world.
> In a VM doing what? It can't just be idle. It has to be executing some code whose timing you can measure to extract its secrets, and the secrets in the address space of that process have to be useful in order to get the file, e.g. a password that can be used to sign into the VM. Then the exploit has to be tailored to that software and hardware.
So again, they are absolutely irrelevant in the real world.
Security industry became an industry of chicken littles. 99.9999% of modern attacks are attacks that were executed successfully because someone was running the code that had sql injections, direct variable substitutions before doing sytem, lack of user input sanitation and validation, or pulling yet another multi-gigabyte unaudited pile of junk dependencies a-la left-pad.js, not because someone mounted a timing attack. Except that dealing with those issues is not sexy so instead we are getting everyone freaking out about some internet villains doing something ( that no one else can do ) to CPUs of a random Joe ( the villains knows everything about Joe's machine and processes running on it and can even control it ) to mount a sophisticated attack, never mind that he can just make Joe install a piece of code that will run with elevated privileges for 0.0001% of the effort.
This is why execs are considering security people to be snake oil salesmen.
I think at this point anyone that's capable of launching such an attack would much rather it be as transparent as possible as they'd have far, far more to gain from not disrupting services. Aside from ransomware attacks, I think the general trend has always been towards those type of attacks as they're way more likely to be profitable
There have been multiple cases of significant vulnerabilities where cloud providers had to perform patching and restart their services. No one, to my knowledge, has exploited those maliciously, but the vulns have had impact.
I think you inadvertently hit on why this is a good discussion to have: security is a concern across the entire stack and attackers have many points where they can successfully compromise your system. It's much harder to replace Xen than it is to disable SMT, so it's much more feasible for someone to consider the latter since it's a single configuration point with very little chance of side-effects other than cost whereas touching Xen would have massive ripple effects for ops & potentially compatibility.
And conversely it's much more beneficial for an attacker to consider the former (attacking the hypervisor), this exacerbates the situation by making red team more aggressive to one side whilst blue team is busy debating if they should disable SMT .
Spectre/etc reinforced that.
>> We're planning to get some high core count machines to be new compute machines in our environment of general multi-user Unix login servers
I think the "general multi-user Ubix login" thing assumes different users on the same CPU. Unless they're actually going to have more CPUs than users. It seems like sharing a CPU is actually less secure than sharing one via SMT because of all those SMT-specific exploits (in addition to whatever exploits might be possible due to CPU sharing).
It would make sense to me for a traditional multiuser shell server to have a scheduling policy prohibiting SMT sharing between users. As long as the scheduler work is not expensive, it would seem almost no-cost vs disabling SMT in case all users only run a single thread, but you can still get whatever benefits SMT provides within a user's workload.
I’ve found myself with more and more arm hardware these days, and I’d assumed (probably incorrectly) that it wasn’t as prone to these kinds of issues.
We do have a bunch of leaked intel firmware, or so I'm told...
I wish this weren’t the case, but whatever PureDarwin has right now is probably about the closest you can get to a Darwin environment that boots without nonfree code.
Where did you find such a thing?
Microarchitecture: Neoverse N2, NeoverseE1, Apple Firestorm (all the same ARMv8 architecture, but all different microarchitectures)
Of these: Neoverse E1 has SMT. Apple Firestorm does not.
So don't visit the web then.
[0] https://developer.mozilla.org/en-US/docs/Web/API/Performance...
These days I mostly have no idea what the hell any of my equipment is running. There is a whole industry that popped up to write tools to work out what the hell you are running and work out how screwed you are. Doesn't feel healthy to me.
Now if only I could figure out how to disable Windows Defender reliably.
How I feel about this all: https://xkcd.com/538/
I'm not sure 20-25 percent is even worth having SMT given that is' like going from 4 cores to 5.
Can you think of another 25% bonus that costs nothing in terms of chip area? SMT doesn't add any new registers (the register files and retirement buffers are the same). SMT doesn't add cache, or pipelines, or anything.
SMT just requires the chip's resources to be partitioned up in some cases, so its a complication to add to a chip but doesn't seem to take up much area at all.
Normally to get a 25% bonus, you'd have to double the L2 cache or something similar (and as the 3x L3 cache tests from Microsoft proves for Milan-X, even 300% L3 cache isn't anywhere close to 300% faster, despite taking up significantly more die area... albeit cheap die area in the form of 3d-stacked chiplets but still, a lot more silicon is being thrown at the problem there... compared to nearly-zero-silicon in the case of SMT)
AMD Zen 2 has 180 registers in its 64-bit register file (an additional set of registers, also nearly 200ish, is also for the 256-bit vector YMM registers). The 16 architectural registers (RAX, RBX, etc. etc.) are mapped to these 180 registers for out-of-order execution.
SMT is simply two threads splitting this 180-register file between each other.
---------
That's how out-of-order execution is accomplished. The "older registers" are remembered so that future code can be executed.
In practice, instructions like "xor rax, rax" are implemented as "malloc-new-register from the file, then set new register to zero".
As such, all you need is the "architectural file", saying "Thread#0's RAX instruction-pointer #0x80084412 is really RegisterFile[25]", and "Thread#0's RAX instruction pointer #0x80084418 is RegisterFile[26]", and finally "Thread#1's RAX instruction pointer #9001beef is RegisterFile[27]".
That allows all 3 contexts to execute out-of-order and concurrently. (In this case, the decoder has decided that Thread#0 has two things to execute in parallel for some reason, while Thread#1 only has one RAX allocated in its stream)
The retirement queue puts everything back into the original, programmer intended order before anyone notices whats up. Different threads on different sections of code will in practice use up a different number of registers.
A "linked-list" loop, like "while(node) node = node->next;" will be sequential and use almost none of the register file (impossible to extract parallelism, even with all of our Out-Of-Order tricks we have today), meaning Thread#2 can SMT and maybe use all 180-registers in the file for itself.
-----
Side note: this is why I think Apple's M1 design is kind of insane/terrible. Apple's register file is something like 380+ registers or somewhere there-abouts (on an ARM architecture only showing 31 registers to the assembly language), but does NOT in fact use SMT.
That is to say: Apple expects a singular thread to actually use those 380+ registers (!!!), and not really benefit from additional threads splitting up that absurdly huge register file. I'm sure Apple has done their research / simulations or whatever, and maybe it works out for the programs they've benchmarked / studied. But... I have to imagine that implementing SMT onto the Apple chips would be one of the easier ways to improve multithreaded performance.
Apple's main risk was deciding that such large register files were in fact useful to somebody.
The example frequently given is running an integer heavy and floating point heavy code at the same time - they don't use the same execution units, so they can better utilize available resources.
In the real world it's more likely that one thread is waiting on memory access or some other part of the system and the other thread can proceed during that wait.
"(For reasons beyond the scope of this entry, we can assume that SMT is worthwhile for us.)"
This is also coming from utoronto.ca - so... I guess the question I would have is actually more along the lines of whether there are actually adverse effects from exploiting SMT to snoop on other processes?
Is this going to be a whole lot of engineering students doing finite analysis homework? If so, what do you get out of an SMT exploit? There are probably a lot easier ways to cheat on your homework.
Are they doing analysis of medical data that hasn't yet been anonymized? Perhaps that's a bigger deal. Maybe.
1. There is no such thing as perfect security
2. Security is an economic decision. You have to know what you stand to lose before you decide if buying 2x the number of processors and disabling SMT is worth it.