Comment by lambda | Hacker News Reader

Comment by lambda | Hacker News Reader

Parent Full thread

> > Calls which have no limitations, of which there are only a very few, for example zx_clock_get() and zx_nanosleep() may be called by any thread. > > Having the clock be an ambient authority leaves the system open to easy timing attacks via implicit covert channels. I'm glad these kinds of timing attacks have gotten more attention with Spectre and Meltdown. Capability security folks have been pointing these out for decades.

Is there a way that these could mediated by a capability without having to incur syscall overhead? One of the reasons that these are bare functions is likely that they are in the vDSO, and are just simple function calls which can access some shared memory which contains the clock time. I suppose you could simply not give some processes access to that memory, and have the functions in the vDSO just return an error in that case.

I know that there was a time when the Linux kernel changed how their vDSO handling worked, so older glibcs would have to fall back to making an actual syscall for gettimeofday, and that seriously affected performance on some servers that updated the kernel without updating glibc. These functions are called quite often on servers for logging purposes, so adding overhead to make them go through a syscall can be a big performance hit.

> I'm hesitant to endorse any system calls with ambient authority, even if it's scoped by context like these. It's far too easy to introduce subtle vulnerabilities. For instance, these calls seem to permit a Confused Deputy attack as long as two processes are running in the same Job.

Yeah, this is a bit odd. In fact, it's not just these syscalls which have ambient authority, there's a whole list in https://fuchsia.googlesource.com/zircon/+/master/docs/syscal... and it includes VMOs, ports, sockets, and so on.

It does seem somewhat odd to have this capability system, but then ignore it for a number of actions which can only be limited at the job level.

Timing in particular is tricky. On Linux, you could use the vDSO for high-resolution timing but, from an attack perspective, it’s a red herring. Any serious attacker would use RDTSC, RDPMC, threads and shared memory, or some other hardware mechanism. On x86, RDTSC and RDPMC are controllable by the scheduler (there are bits to turn them off), but it doesn’t really fit in a capability model.

> you could use the vDSO for high-resolution timing but, from an attack perspective, it’s a red herring. Any serious attacker would use RDTSC,

Good point. I was going to say that in general the vDSO is just rdtsc + an offset applied. However they insert a barrier before it usually. That maybe or many not be helpful so I would probably still use rdtsc by itself.

> Is there a way that these could mediated by a capability without having to incur syscall overhead?

Is that even warranted? What applications can you imagine would make so many clock calls so as to incur noticeable overhead?

Your example of logging costs might work, but I'm very skeptical that user/kernel transition costs for a clock call would drown out the costs of writing the log entry to disk.

But to answer your question directly, the ability to access the clock in a shared memory segment can itself be reified as a handle that's granted to a process. The process would then issue a map operation and provide an address at which to map the clock (or you could just always map it at the same address too if that's preferable for some reason).

> What applications can you imagine would make so many clock calls so as to incur noticeable overhead?

Low latency processing or realtime-ish applications. If you have a budget of a few milliseconds only, lots of gettimeofday can start to add up. Now true that nowadays they are mostly vDSO so it doesn't matter as much. I remember before they were we saw a decent speedup when we upgraded to vDSO kernel version.

> But to answer your question directly, the ability to access the clock in a shared memory segment can itself be reified as a handle that's granted to a process.

rdtsc is not shared memory but more like a register read. Though it is a virtualized instruction so on proper VMs (not a container) it is possible to control what the guest sees as the value.

Indeed, but then access to a realtime clock factory should be reified as a handle. You would have to be given this handle and explicitly invoke it to install access to the clock if that's needed.

The point being, a handle should be involved at some point in order to make the access control explicit and not implicit and ambient.

I've got a customer that wants to get access to the PTP[1] registers of a NIC from user space. The customer suggests that context switches introduce too much latency and indeterminism (pre systems integration phase - lots of software from different teams ends up running on the final platform) to introduce a reference monitor (capability-based microkernel or otherwise) given known techniques on modern hardware. I'm afraid I can't share the specifics because I don't have them, but I trust the source.

I will grant that this case is an outlier, but this customer use-case is real enough to drive development dollars. It may not be the norm, but precision timing access appears to be very useful in some hard real-time contexts given current hw/sw realities.

[1] https://en.wikipedia.org/wiki/Precision_Time_Protocol

Wouldn't this involve modifying the driver then?

Yes, as well as some lower-level software. (sharing portions of a PCI BAR region isn't trivial in a hypervisor, when multiple guests are involved).

Mixing safety criticality levels is hard.

Each log entry isn't necessarily written to disk one at a time. They will generally be buffered until enough have happened to need to flush. And they can be written compressed, and frequently have a lot of redundancy, so it can take a while before you accumulate enough data to need to flush to disk.

And besides logging, there are things like nanosleep or spinlocks which may briefly spin while querying the time before yielding.

I know of at least one CDN which was measurably impacted by the gettimeofday issue.

And yes, you could have the access to the address itself be something you are given access to via a capability/handle, but then the system call itself (which is actually a vDSO call) wouldn't have to actually take a handle as a parameter.

> And besides logging, there are things like nanosleep or spinlocks which may briefly spin while querying the time before yielding.

Another poster said something similar, but using the clock for this seems strange to me. Yielding a time slice for a certain number of ticks doesn't need access to the current time. I have less objection to an ambient yield since that's really an operation on your own schedule capability.

If you're after some kind of exponential backoff for a spinlock, that again doesn't seem to need the current time so much as a growing counter of the number of ticks to sleep.

> And yes, you could have the access to the address itself be something you are given access to via a capability/handle, but then the system call itself (which is actually a vDSO call) wouldn't have to actually take a handle as a parameter.

Correct, you'd use the handle to install an ambient clock in your environment. This leaves open the possibility that you can easily virtualize the clock by proxying the clock handle, ie. instead of the kernel updating your shared memory segment, it's another process.

The point being that reifying everything as a handle makes arbitrary virtualization patterns possible, but having ambient authorities all the way down to root makes it much more difficult.

>Another poster said something similar, but using the clock for this seems strange to me.

Does it matter if it seems strange to you? The GP gave real world examples of where this is an impact. I'm not sure why you are arguing against the design.

> The GP gave real world examples of where this is an impact. I'm not sure why you are arguing against the design.

Because the design is questionable, and so the example is questionable. Isn't that obvious? Why else would I bring it up?

Yes, I agree that using the handle to install an ambient clock is probably better than just having one exist, for the reasons you cite.

clock_gettime is extremely heavily used by nearly everything. Any form of work queue that supports delayed work, for example, is sitting on clock_gettime. Any form of media uses it heavily, as does self-monitoring to look for performance regressions in the wild.

You might be shielded from this depending on what level you're working at, but there's a reason that vDSO exists basically solely for clock_gettime, too.

Yes it allows for timing attacks, but you can't really avoid that, either, not without utterly crippling your platform.

> Any form of work queue that supports delayed work, for example, is sitting on clock_gettime.

That seems strange to me. Why would you use the clock/gettime call for that instead of a work counter of some sort?

I meant time-delayed. Eg, run this in 10ms type of thing. How else would you do things like exponential backoff?

Right, but your original statement was about clock_gettime, hence my confusion.

Something like sleep() is an operation on your own schedule, not an operation on a global system clock. That's not nearly as problematic.

Consider if you wanted to virtualize a process, say to deterministically replay it to trigger a fault or something. sleep(100 ticks) doesn't require additional kernel support for virtualization, but an ambient clock requires a lot of extra kernel support.

If the clock were only accessible via a handle, then you could proxy invocations on the handle without any extra support in the kernel. See my other comment for more details: https://news.ycombinator.com/item?id=16817462

You could use sleep, sure, but then you require a seperate thread for every delayed message, which is a whole different ball of not-fun.

Otherwise what actually happens when you do a postDelayed(func, delay) is it immediately is translated into a postAt(func, clock_gettime() + delay). Then as messages are consumed the work queue consistently knows how long it should wait to wake back up, no matter how many delayed tasks are in flight.

I feel like I'm missing some context. I thought you were talking about work stealing in threaded systems, in which case sleeping seems reasonable.

Are you instead talking about some kind of event loop? If so, then clock_gettime seems like it's merely convenient, not essential. You could just as easily keep a ticket incremented on successful operations or successive loops (or some other metric), and exponential backoff is a wait operation on a ticket number.

Unless you're suggesting the delay must be based on real time for some reason?

> You could just as easily keep a ticket incremented [..], and exponential backoff is a wait operation on a ticket number.

Congrats, you've re-invented clock_gettime(CLOCK_MONOTONIC) :)

You're right that it's not essential to have, but if you can re-implement it then you've also just re-introduced the timing attack that the removal of clock_gettime was trying to prevent.

The trivial-ness with which a clock suitable for timing attacks is able to be created is literally why SharedArrayBuffer was panic-removed from all major browsers. Because that's all you need, shared memory & a worker. Congrats, you have a high-precision timer with zero OS support.

So as soon as you allow any form of threading or shared memory to occur you've given apps a high precision timer, so you might as well just give them an actual timer, too.

> You're right that it's not essential to have, but if you can re-implement it then you've also just re-introduced the timing attack that the removal of clock_gettime was trying to prevent.

I'm not convinced. The clock is global, shared among all processes, the ticket I suggested is local to a process, and so can't be used to signal between processes. Unless I'm again missing some context for what you mean.

You might want to go lookup the meltdown/spectre SharedArrayBuffer proof of concept attack. A timing attack just needs local timing to work, it doesn't care whatsoever about any time in any other process. That's just not how it works. All it needs is a stopwatch of any kind no matter how local.

> You might want to go lookup the meltdown/spectre SharedArrayBuffer proof of concept attack.

That still requires a piece shared state, like I said. That vulnerability results from running untrusted code in the same process as trusted code. The whole point of a process is to establish a protection boundary around potentially unsafe code, hence why I keep mentioning processes, and this is the whole point of microkernels and IPC. Within a process, all bets are off.

My earlier point was that a shared clock between processes amplifies this problem so that timing attacks cross even the process protection boundary. So when you started talking about job scheduling, I assumed the following:

1. we're in a microkernel context, where we partition trusted and untrusted code using processes.

2. the job scheduling system you mentioned either

a) is a process running its own code that it trusts and so in-process timing attacks don't matter, but timing attacks with another process might matter and so you don't want to grant a clock capability if it isn't needed, or

b) is a process scheduling system ala cron, where the job scheduler is trusted but the jobs being run are untrusted, and so they run in separate processes.

In case (a), the ticket system seems sufficient if you don't want/need to grant access to the clock consistent with least privilege, and for (b) the job scheduler may or may not have the clock installed or not, doesn't really matter, since job scheduling happens via IPC so there's no shared read, ie. delay(100 ticks, self) sends the relative delay which the cron-like scheduler adds to its own clock.

Hopefully that clarifies my context, and you can then describe what assumptions your scenario violates.

> My earlier point was that a shared clock between processes amplifies this problem so that timing attacks cross even the process protection boundary. So when you started talking about job scheduling, I assumed the following

It doesn't, though. Your point fundamentally misunderstands the nature of a timing attack for ex-filtration. It doesn't cross process boundaries. It doesn't use any process crossing state of any kind for timing. It simply times how long it takes to access cache lines, which is perfectly local & isolated. It does not involve time correletation or association across processes of any kind. It doesn't care about real time at all. All it needs is a stopwatch that can measure the difference between 0.5-2ns & 80-120ns.

The meltdown attack via SharedArrayBuffer did not use any untrusted code in the same process. It read kernel memory at will using 2 threads and an atomic int.

Ted Unangst had a few posts where time getting featured heavily, e.g. https://www.tedunangst.com/flak/post/firefox-vs-rthreads and the linked mailing list thread https://marc.info/?l=openbsd-ports&m=144649333120401&w=2

There's a few more related articles on his blog if you search, and I've read of it being implicated elsewhere, that's not to say the software calling that much was doing the right thing, just that it had impact.

NUMA systems might take 1 microsecond just to retrieve time internally from a shared hardware resource. The operation is not concurrent, so there's a chance for it to block for much longer.

This can happen because RDTSC is not synchronized between physical CPU sockets (NUMA regions).

Reply on news.ycombinator.com