Hacker News Reader

Comment by al2o3cr | Hacker News Reader

The Intel docs mention that PAUSE isn't the right choice for waits of "thousands of cycles or more" - even on the old architecture, 10ms is waaaaaay longer than recommended.

I also have to quibble with this conclusion:

    This is not a .NET issue. It affects all Spinlock implementations
    which use the pause instruction.

The instruction isn't the issue, the exponential backoff that .NET includes is. At the extreme (last iteration, on a 64-core CPU) it will spin for 1.2 million pause instructions...

View on HN

The whole point of spinning is to avoid the expense of a context switch by putting the thread to sleep, letting the OS schedule another. If you're going to spin for 10ms, that's longer than the timeslice most threads run for. That is going to be well above the overhead of any context switch. It makes no sense.

> That is going to be well above the overhead of any context switch.

Is it though?

The "pause" instruction gives all resources to the hyperthread sibling. So under high-utilization (ie: your core has a hyperthread-sibling ready and willing), a "pause" instruction is basically a super-low latency task-switch.

I wouldn't be surprised that a "spinlock with pause" would be better, even in 5ms to 10ms chunks of time (Windows has 15ms quantas, and may task switch as late as 30ms to 75ms depending on the situation).

Because its better to do the hardware-assisted "pause" task switch rather than actually cause the OS-scheduler to get involved. In fact, that's what Microsoft AND Intel have concluded!! To get better real-world metrics, both Microsoft AND Intel felt like the "pause" instruction should wait longer! Skylake-X increases the latency, while .NET created this exponential-backoff.

I bet it was good in isolation, but when combined, the overall effect was too much pausing.

Giving resources to the sibling hyperthread via pause is not the kind of context switch I'm talking about. I'm talking about the expensive context switch when the OS scheduler must swap out a running thread for another. That carries a steep obvious penalty switching between user space and the kernel and back, but actually can involve much heavier penalties where the cache and TLB are concerned - depending what thread runs next.

Pausing for as much as 10ms could be better under some circumstances, but that would seem to me to be heavier than a context-switch. Whether the system actually achieves better throughput or not in that case would be workload dependent. Worst case the hyperthread is also running the spin lock loop and no useful work at all is accomplished on the core.

> Giving resources to the sibling hyperthread via pause is not the kind of context switch I'm talking about.

Isn't this entire thread about "pause" ?? Anyway, i think we both understand the difference between "context switch" and "pause" here.

> Pausing for as much as 10ms could be better under some circumstances, but that would seem to me to be heavier than a context-switch. Whether the system actually achieves better throughput or not in that case would be workload dependent.

Seems reasonable. Ideally, threads / cores shouldn't be blocked that long, so 10ms+ blocked periods is definitely an anomaly.

But whether or not a context switch is actually worth it? Ehhh, its hard to say for sure.

> Is it though?

Yes, definitely, beyond any doubt.

The .NET approach only makes sense if you have exactly the same number of active threads as CPU "cores" and if your CPU always supports hyperthreading. If either of those is not true then what .NET is doing is a severe CPU bottleneck when the lock is actually contended.

And it is incredibly unlikely that that exact situation happens in the wild. You almost always have more than <cpu core count> threads/processes active in heavy multithreading workloads, and you definitely can't assume everyone has hyperthreading as that's a feature commonly cut from mid & low range CPUs.

For context here a thread context switch via something like futex is in the sub-10us range. So a 5ms spinlock robs you of ~4.9ms of usable CPU throughput that a different thread/process could have been doing instead. The timeout levels here are just shy of an eternity. 5ms is a really, really long amount of time.

By way of comparison pthread_mutex_lock in glibc will do one pause per attempt for at most 100 attempts. So under skylake it has a max pause duration of around 3-5us, depending on clock speed, which is pretty reasonable. It roughly aligns with the cost of a context switch, so once it starts spinning for longer than a context switch it switches to a context switch instead.

Right. The whole time I was thinking the hardware change probably exacerbated an existing software issue. Then I saw this:

"The issue has been fixed with .NET Core 2.1 and .NET Framework 4.8 Preview contains also the fixes for it."

And by the end of the article I was thinking their code might just run faster as a single thread on a ~4GHz processor.

Another thought was why do they have so much resource contention in the first place? That seems fishy.

Another thought: Perhaps they should write it in Rust with all of its "fearless concurrency" claims. While I'm skeptical of those claims, The exercise would probably highlight some of the real issues.

Any way I look at it, I keep thinking it's a software design problem.

Rust doesn't magically fix resources contention. It just provides safe concurrency primitives (and the surrounding language semantics) that make it difficult -to-impossible to misuse in such a way as to cause a deadlock or race condition.

Right, so if the language forces you to eliminate deadlocks and race conditions you may find less need for spinlocks which are the problem here.

Speculation on my part. I'm not actually advocating Rust but I am thinking about a project to give it a try.

if the language forces you to eliminate deadlocks and race conditions

It does not, it only prevents a specific problem known as a data race, which is an issue related to memory/ownership.

There is no deadlock protection. And race conditions are a much broader set of issues than just data races.

I like rust, just want to keep your expectations in check.

If I remember correctly, rust doesn't help with deadlocks - you still have to be careful of them. Though it does have primitives to help significantly with race conditions

The primitives help with taking the same lock around every access to a location. They don't help in finding locations that should use the same lock; you can easily write Rust code that uses too many separate locks and as such can still have race conditions.

> difficult -to-impossible to misuse in such a way as to cause a deadlock or race condition

Rust doesn't solve deadlocks or race conditions. Rust only makes it hard to accidentally share a memory location, but that's only a very small part of the problem, race conditions being a fundamental part of computer science, being baked into any language that supports expressing and joining asynchronous processes.

I would be generous in agreeing with a statement about Rust making it a little more difficult to end up with a deadlock or unwanted race condition, except that's not true when compared with many other high level languages.

Also nothing is impossible until you prove it via formal methods (e.g. TLA+).

> Rust only makes it hard to accidentally share a memory location,

Not just a memory location, any value. You can use the tools Rust gives you on any piece of state you care to manage - a database connection, a remote server, the office coffee machine - represent it as a value and Rust will let you check that it's owned where you want it to be owned, borrowed where you want it to be borrowed and so on.

> Also nothing is impossible until you prove it via formal methods (e.g. TLA+).

What's the distinction you're drawing between something like TLA+ and something like the Rust borrow checker?

> I would be generous in agreeing with a statement about Rust making it a little more difficult to end up with a deadlock or unwanted race condition, except that's not true when compared with many other high level languages.

It is easy to specifically describe what Rust prevents: unsynchronized memory access from multiple threads. What other high-level languages allow read-write shared-memory multithreading and guarantee that?

> Also nothing is impossible until you prove it via formal methods (e.g. TLA+).

The fact that unsynchronized memory access is impossible in safe Rust (modulo compiler bugs, CPU bugs, etc.) is something that nobody seriously doubts. It's obvious. I'm glad people are working on proving it, but the proof is an academic exercise.

> It is easy to specifically describe what Rust prevents: unsynchronized memory access from multiple threads. What other high-level languages allow read-write shared-memory multithreading and guarantee that?

Pony maybe? https://www.ponylang.org/discover/#what-is-pony

At least that's my impression of its reference capabilities.

No, it's something that Pony relatively explicitly doesn't allow: AIUI, iso (non-aliasable, read/write), val (aliasable, read-only, no write) and tag (aliasable, no read or write) are the only things that can be shared between actors/threads.

You're right about deadlocks and general race conditions. I was confused. Rust only (statically) prevents data races (arguably the only form of race condition that is always undesirable). Rust does nothing to prevent deadlocks.

The advantage of Rust over, e.g. Java, then, is that Rust can prevent data races at compile time?

Yup.

It's definitely .net. We've observed deadlocks in spin scenarios which came down to this implementation rather than our callers.

You're suggesting, then, that the spinlock implementation is not merely slower on some CPUs but actually _incorrect_? It seems more likely that your software had some combination of a deadlock and a race condition exposed by different lock timing -- inserting delays in execution is one way to expand race windows to reproduce these kinds of defects.

Yes I am suggesting the implementation is incorrect. We had a standalone test case for this which could reproduce the condition after a random wait of 1-45 minutes. I will dig it out and raise it against .Net Core if it's still valid there.

The problem was GC initiated i.e. the entry into GC caused the deadlock to start with which the process never recovered from as it was spinlocking all threads and no resources were available to complete GC.

Reply on news.ycombinator.com