Hacker News Reader

Comment by kccqzy | Hacker News Reader

The author abandoned the distributed lock service he/she is writing, and it might benefit to read the original Google paper on their distributed lock service Chubby to understand why it is so enduring at Google: https://static.googleusercontent.com/media/research.google.c...

> I abandoned the project after it very quickly become apparent that despite having written the service in this super fast, brand new language called golang, the service just wasn’t fast enough to handle the scale we threw at it.

This makes me think the author wishes to use the distributed lock service for some purpose that's not well served by distributed locks. It's not that distributed locks are bad, it's just that the author seems to have a particular use case already in mind that's poorly suited to a distributed lock service.

View on HN

> author wishes to use the distributed lock service for some purpose that's not well served by distributed locks

Exactly. It should be clear that a distributed lock service has a finite, and low, overall rate of progress. It obviously cannot be in the critical path of every transaction globally. But when using it for events which rarely happen, such as electing a new master of a partition of a database, or some other thing that happens once a week, then the low throughput is not an issue.

I ddon't know why they'd expect Go to be a panacea. It has everything to do with architecture.

What's an OSS equivalent to Chubby? Etcd?

etcd is fairly similar to Chubby indeed, or Zookeeper.

ZooKeeper or ClickHouse Keeper, Etcd, Consul.

Consul has an entire locking system that’s based on Chubby in the KV system. I’ve used it at pretty big scales and it seems fine.

If you want actual OSS you can use an older release, the KV system guts haven’t changed much in awhile iirc.

The mistake was using golang apparently.

If you need high-performance and fine control of synchronization, just use a low-level systems programming language.

You can get into that type of lock congestion trouble in any language. It's an algorithm problem, not a language problem.

I discovered last year that Wine has terrible internal lock problems inside its user-side storage allocator. That's in C. If you have enough threads calling "realloc", the allocator goes into futex congestion collapse and performance drops by two orders of magnitude. My graphics program went from 60 FPS to 0.5 FPS. They optimized too hard for the no-congestion case.

This is a Wine-only problem; Microsoft's own code doesn't have this problem.

I've had lock congestion problems in Rust. Sometimes you need a fair mutex, or something gets frozen out. Both fair and non-fair mutexes are available; see the "parking_lot" crate.

There's a place inside WGPU that has a lock congestion problem in one of three locks, and I'm going to have to add more profiling to someone else's code to find that. I can see the problem with Tracy, but need to add more profiling scopes to narrow it down.

But that is high-performance graphics stuff, where microseconds count. Sending spam (OK, bulk marketing emails) doesn't need to be that tightly coupled. Mailing list removal runs on a timescale of days, not milliseconds. What else in that space has to be tightly interlocked?

> I can see the problem with Tracy, but need to add more profiling scopes to narrow it down.

If you can run on Windows try Superluminal.

https://superluminal.eu/

A good language just gives you the necessary tooling to do whatever you want, it doesn't magically fix problems.

Only languages like C++ have a memory model that allows you to do lock-free programming for example (C and Rust copied the C++ model).

Also, what kind of serious person allocates memory from the system allocator in a real-time loop? Your problems seem self-inflicted. Regardless there are many allocators that optimize for concurrent allocations: tcmalloc, jemalloc, mimalloc...

Go has atomics.

Which don't have the same level of control as C++, it only provides a subset of sequentially consistent primitives, which are slow by design.

Considering the vast number of programs that wine works extremely well with I'm not so sure they spent too much optimizing the no-congestion case. You are just doing something extremely quirky in your program.

I've looked at the code in a debugger. Wine has futexes three deep in "malloc". The innermost one is a pure spinlock. The problem with "realloc" is that, when it can't grow an array in place, it has to copy the contents. The Wine implementation does that with the main lock on allocation still held. So, if you have Rust code with a lot of multithreaded vector "push" operations, and more threads than CPUs, you get futex congestion. It's possible to write applications that don't hit this bug, but it's Wine-only, not Windows, so not worth it.

What's "quirky" is trying to use all the CPUs with lower priority threads.

Why not send a patch?

It's hard to fix without a redesign of some crucial low-level code. The people who wrote it looked at the problem and decided it was too hard to fix.

I suspect that when it was chosen at Mailgun, Golang was still being billed as a systems programming language.

Exactly.

Go makes you think you control the details except you don't. Hackernews makes you think you don't control the details in C# except you do.

Yet another project that would have been able to solve its woes if it had picked a better option.

Does the language make a huge difference here? In a distributed system a signal to be sent over the network travels at the same speed whether it was transmitted by a C or Python program, right?

It would take you 1 minute to write a Go application that exploits per-CPU data structures and probably years to debug your attempt to do so in C.

actually its really _hard_ in go to make cpu bound control flow, state, and allocation. do goroutines have any notion of locality? i've been looking and haven't been able to find anything

Go comes out of the box with sync.Pool

You can pin the go routine to a current thread using[0] and then pin the thread to a core using[1] if that’s what’s you’re after

[0] - https://pkg.go.dev/runtime#LockOSThread

[1] - https://pkg.go.dev/golang.org/x/sys/unix#SchedSetaffinity

Nope. In system like this, if written correctly, main performance bottlenecks are network and disk latencies

Reply on news.ycombinator.com