Hacker News Reader

>are there many clouds that still run workloads from different users on the same physical core?

There are a vast number of VPS providers out there that aren’t AWS/GCP/Azure/etc where the answer is yes. Even the ones that sell ‘dedicated’ cores, which really just means unmetered cpu

What about burstable instances on AWS, and whatever is the equivalent in other clouds? Hard to imagine those having a dedicated core, would probably defeat the purpose.

https://docs.aws.amazon.com/whitepapers/latest/security-desi....

Not just burstable instances.

AWS Fargate, container as a service, allows specifying 0.25 or 0.5 CPU, and I would be surprised if those weren't shared.

Same probably? also applies to AWS Lambda.

My guess is there's likely less value in trying to target those kinds of environments... Just poking random data out of lambda or low end vps neighbors is a needle in a haystack the size of the moon in terms of finding anything useful.

It's more likely useful as part of a group of exploits to hit an individual, targeted system.

>Just poking random data out of lambda or low end vps neighbors is a needle in a haystack the size of the moon in terms of finding anything useful.

LLM's might change that. getting a firehouse of data and asking it to classify controls against NIST 53- rev5 produces interesting results.

Worse: by default it's not CPU but VCPU - a single core with multithreading = 2 VCPU

Per the paper, this looks like an attack against speculated instructions that modify the store forward buffer. The details aren't super clear, but that seems extremely unlikely to survive a context switch. In practice this is probably only an attack against hyperthread code running simultaneously on the same CPU, which I'd expect cloud hosts to have eliminated long ago.

Yeah, the way AWS has stock language like “AWS has designed and implemented its infrastructure with protections against this class of issues” supports the idea that you’re probably not getting anywhere with exploits of this class on a major cloud host any more.

The Spectre attack had to be patched in the kernel in a way that significantly slowed down execution on Intel CPU:s: https://www.notebookcheck.net/Spectre-v2-mitigation-wreaks-h...

What's interesting is that the FDIV bug from 1994 could also be worked around, but Intel recalled and wrote off those processors[1]. For their latest several problems, their response was more of a "sucks to be you". While they provided microcode updates and worked with OS vendors, there were performance impacts that materially affected the value of the chips.

1. https://www.intel.com/content/www/us/en/history/history-1994...

The software workaround for the FDIV bug required the actual userspace software to be modified and recompiled. There was a decade of pre-existing software out there that would be hard to fix, especially in the days before most people had the internet.

There was nothing the OS could do to work around the bug, short of disabling the entire FPU and falling back to expensive software emulation of all floating point math.

The workarounds for all these speculation bugs can be mostly applied at the operating system and/or microcode level, and is comparably cheap.

Yes, I think that's what I said? Every attack no matter how deep it seemed to be has been patchable in microcode, sometimes at a cost in performance. But so far nobody had to toss the physical silicon, at least not with Intel. The malleability of these chips is quite fascinating.

On some Skylake CPUs to get full mitigations you are taking a 30% performance penalty _and_ you need to disable SMT which is often another double digit penalty. It's not a literal "toss the physical silicon", but it's getting there.

In fact my thesis is that it's never a literal "toss the physical silicon" because the kernel is able to take control when switching between tasks so (with help from the microcode) it's able to wipe all potential speculation vectors (at an arbitrarily expensive cost) before switching to the next task. This also explains why SMT is unfixably broken on some processors, since by design the kernel does not intervene in the task switching on the virtual cores.

The meltdown and spectre mitigations weren't patched in microcode, they were patched by changing the internal calling methods of the kernel.

I think it was both. There were initial software only patches, but those were quickly superseded by microcode+kernel patches, where microcode added features the kernel enabled. IBRS or something like that.

It was both, and none of them completely addressed the problem. They did need new hardware for that.

> General caveats: are there many clouds that still run workloads from different users on the same physical core? I thought most had changed their schedulers years ago so you can't get cross-domain leaks between hyperthreads anymore.

Isn't this the whole point of AWS' t instances? It's my understanding that they are "shared" at the core level, or else there wouldn't be a reason for the CPU credit balance thing.

They are definitely time-sliced among tenants and very possibly two tenants may run at the same time on two hardware threads on the same core: but you could have a viable burstable instance with time-slicing alone.

Nevermind, AWS explicitly documents that all instance types, including burstable, never co-locate different tenants on the same physical core at the same time:

https://docs.aws.amazon.com/whitepapers/latest/security-desi...

It only seems to document that there's group scheduling for SMT cores. But that doesn't prevent issues due to switching between customers on the same physical core, no?

"It is possible, however, for two burstable performance EC2 instances to run sequentially (not simultaneously) on the same core. It is also possible for physical memory pages to be reused, remapped, and swapped in and out as virtual memory pages. However, even burstable instances never share the same core at the same time, and virtual memory pages are never shared across instances. "

I only started reading the paper - so I very well might be wrong here - but it doesn't look to me like you need victim/attacker to be scheduled simultaneously on two SMT threads, but that a single core sequentially executing victim / attacker code would be vulnerable. It's possible that the cross-customer "context switch" is larger than the the vulnerable window, but I'd not want to bet on it.

> But that doesn't prevent issues due to switching between customers on the same physical core, no?

Yes they are explicit that customers may be time-shared on a physical core ("burstable" instances don't really make sense without that). Most of these attacks aren't known to be possible in that scenario and in any case the mitigations are much easier since flushing sensitive state at group scheduling boundaries is much less costly than permanent dynamic changes to how concurrent SMT threads interact.

Yes, it's certainly easier to mitigate at a boundary that's already as costly as switching between VMs.

The paper documents that disabling SMT does not entirely mitigate the problem (In 9.1). They briefly mention trying instructions to avoid the microarchitectural leaks, but don't go into more detail than mentioning verw isn't sufficient.

They state that a switch to/from SGX, with SMT disabled, doesn't prevent the attacks. See 8.1. That's not the same as a cross-vm switch, but it's certainly interesting that the attempts at flushing microarchitectural state when exiting SGX don't provide protection.

Think of all the cloud resellers that are out there who really aren't segregating their tenants out or it's just a web shop with proxmox who recombined their own customers onto a core even though the cloud provider specifically segregated it

I think most if not all cloud VMs dedicate a core to you. Well, there are some that share like the T series on AWS and I think other clouds have similar, but my bet is they can put in an extra "flush" between users to prevent cross tenant leakage.

Of course cross process leakage for a single tenant is an issue, in cloud or on prem, and folks will have to decide how much they trust the processes on their machine to not become evil...

This concern I also share and it's probably worth converting into layman's terms so that all computer users understand what it is. Basically the job scheduler Behavior in the OS needs to surface to the user with understandable language they can read so they can make the trade-off decision.

> Claiming that it affects all users on the internet seems like a massive over-exaggeration, as he hasn't demonstrated any kind of browser based exploit and even if such a thing did exist

He's saying it likely affects "everyone on the Internet" because most servers are vulnerable.

Most servers being vulnerable to a local attack is generally pretty boring news.

If you throw in the same vulnerability that AMD has with the list from Intel I think it pretty much covers every server available for rent at Quadra.

> same physical core…between hyperthreads

These are not the same thing. Afaik, most “vCPU” are hyperthreads, not physical cores.

> I thought most had changed their schedulers years ago so you can't get cross-domain leaks between hyperthreads anymore

It would be great to have a source on this.

> These are not the same thing. Afaik, most “vCPU” are hyperthreads, not physical cores.

OP didn't say otherwise. They are saying that public clouds do not let work from different tenants run on the same physical core (on different hyperthreads) at the same time.

This doesn't prevent you from selling 1 hyperthread as 1 vCPU, it just means there are some scheduling restrictions and your smallest instance type will probably have 2 vCPUs if you have SMT-2 hardware (and that's exactly what you see on AWS outside of the burstable instance types).

Does Digital Ocean count as a major cloud player?

Yes. And Linode. And Quadra. And OVH.

A lot of people on YC are enterprisey-brained and only think there are 3 possible clouds, and then there is the rest of the planet who can't afford to park their cash at AWS and set it on fire.