Hacker News Reader

Comment by gostsamo | Hacker News Reader

How significant is this change? Are there utilities in the server world which do the same but with extra steps instead of having it as an os builtin?

View on HN

This is not some sort of "new" feature invented by OpenBSD.

It's literally "just" them making their kernel (finally) multi-core instead of locking everything every time a process does networking.

Pretty much every other OS out there has done that for years, if not decades.

(FreeBSD for almost 20 years, although arguably at first that release was quite unstable.)

I was recently looking at porting a cool OpenBSD feature to FreeBSD and was wondering how the locking could possibly work in terms of ordering. It turns out that all the OpenBSD locks boiled down to a global netlock rw lock for the entire IPv4/IPv6 stack.

This is what surprised me. It looks as a common sense approach, but it is on the front page of hn.

Usually when OpenBSD does something it's because there's a clean implementation.

OpenBSD is very seldom the first to actually do something; but they focus on making their code able to be understood by a single person.

Doing things is hard, doing them in a way that is clear, obvious and "simple" is even harder. So when OpenBSD finally manages to do something that others may have had for a while... it's a cause of celebration. Because if the implementation was a clean one it would have gone in much sooner.

Do you know of a write up that illustrates this extra simplicity by comparing OpenBSD’s implementation of something with a previous mainstream implementation?

I would say compare the OpenBSD documentation for ground up install to network serving to any other documentation source for the same process.

OpenBSD will definitely be straight forward and easy to follow all from a single "manual". I'm sure there are other OSs that compare favorable to OpenBSD but those other OSs would also be high quality.

OpenBSD recently built a simple httpd and I think a mailer (replaces sendmail) as well. There's probably a write up of one of those.

The big difference on those is a very limited scope.

It is insanely difficult to implement it. The idea sounds common sense, but the details are extremely difficult to get right and any mistake causes random crashes, or incorrect data (read security holes).

Is the challenge solved by basically "a lot of mutexes"?

That is one way to solve it. However it isn't always a good answer. Mutexs - by definition - mean that you stop everyone else from doing something when you are doing it, which means if two threads are trying to do something one will be stopped. It is also very easy to forget to lock/unlock a mutex at the right time, both of which will cause problems. Mutexs also send a message out to other CPUs which means even if no other thread wants the mutex you still slowed your system down. In the real world mutexs become painfully slow when more than a few threads are accessing the same shared data.

There are also atomics which don't lock, and so in some cases are faster that a mutex - but they can also be a lot slower if in some data contention patterns. While these are often faster, there are a lot of useful data structures that cannot be written with atomics.

The correct answer is design everything to avoid the needs for writing shared data as much as possible. Then use atomics and/or mutexs in those final places where data sharing is needed. You always need to benchmark to see what works best.

Don't mutexes provide, well, mutual exclusion, so that some operations would be forced to be single-threaded?

Right, but if done right you rarely are in a situation two threads want to share the thread, so in practice you are not. However doing this right is difficult.

Yeah, but most of the things that need exclusivity or consistency are relatively rare or very narrow.

You need consitency when you update the socket tables. (Relatively rare, but can be a bottleneck for use cases with frequent open/close and maybe accept)

You need exclusivity when send or recieve on tcp sockets. (Very narrow)

You also need exclusivity when putting packets one of on the NIC's sendqueues. That one's not really rare, better NICs have more queues for less contention though.

For really high volume packet processing, it's helpful to align socket processing so everything is pinned to the same core: nic interrupts, kernel rx, application read and processing, application send, kernel tx, etc. But last I checked OpenBSD doesn't do cpu pinning, so there's that.

If it's insanely difficult to implement in a monolithic kernel, where all the data structures are easily available and shared, then it must be plain impossible to implement in a microkernel!

Microkernels have pros and cons, but that isn't one.

I suppose we would need hard data; is there an example of a full-featured microkernel-based OS that implements it correctly?

Since microkernels are not popular (at least not open source ones, and the closed source often don't give enough data to know) that isn't even meaningful. We would need a full-features microkernel based OS that tried to do it correctly. Otherwise it could be easier to do, but nobody tried.

I deeply appreciate that they look like they should be easier, but several decades of people trying to make them work have mostly failed which makes me think there's some deeper problem that we're not seeing.

The problem is difficult no matter what. Many many more people are working on monokernels and so it is within the data we have to say that micro kernels are easier. There are not enough people to work on microkernels and there is a lot to do.

I like openbsd for what it is, but they do have a tendency to imply or outright say that the thing they just did is something they invented.

Like when they launched W^X (which isn't even an accurate term, because that implies that read-only non-execute pages don't exist), they basically claimed to have invented it. And they said that it was "impossible" to do on 32bit x86. Which was a surprise for those of us who had been running that on 32bit x86 for years, using grsec patches.

They defended themselves for this mis-statement by saying that they don't look at what other OSs, like Linux, do. Fair enough, but don't say you're the first to do it, then.

OpenBSD was late to the game of multiprocessing, deliberately ignoring multi-CPU machines. They only had to change their minds when suddenly consumer CPUs started being multi-core/multi-threaded. And since they have fewer developers (and not a focus on performance), they've understandably been lagging behind in this space.

> "OpenBSD was late to the game of multiprocessing, deliberately ignoring multi-CPU machines. They only had to change their minds when suddenly consumer CPUs started being multi-core/multi-threaded."

Which seems like an entirely reasonable choice if they felt there were bigger priorities. OpenBSD received SMP support in November 2004, right at the same time as the first mainstream dual-core CPU arrived on the market. I get what you mean by "late to the game", but I also feel that it's not really a fitting statement.

Yes, I said they focus on other things. It's not criticism. But it is true that they dismissed SMP, and did not see dual-core coming. They were late because they were reacting to dual core launching.

> I also feel that it's not really a fitting statement.

In what way? Linux got it almost a decade before that. Windows NT got it more than a decade before OpenBSD.

SunOS approaching 15 years before OpenBSD.

ChatGPT claims Mac OS X got it in 1999 with Mac OS X Server, but I could not immediately confirm that one.

Again, this is not criticism. But I don't see how it would be inaccurate to say they were late, when I think they were literally last.

> "In what way?"

They were last, but not late. SMP in the x86 world was miniscule in terms of multi-CPU setups. It didn't "happen" until the multi-core era, and OpenBSD was well in time for that.

I think you're nitpicking semantics, and don't know what your point is.

Anyway, this is not a productive semantics debate. Have a nice weekend.

If you were the last person to get to the train station but the train still hadn't left, then you really were not late for the train.

But this is only for raw IP, the moment the packet is determined to be say UDP, it gets queued like before this patch.

For OpenBSD it's significant and somewhat of a milestone because its network stack has been trailing a bit in terms of transfer speeds when compared to e.g. Linux. The kernel has had a lot of "big locks" that hamper executional performance in complex ways across many different venues, and the past few years the developers have been busy cutting these into smaller pieces.

Feels a little like rdma or mellanox/ib.

Hey, I’d personally love if something openbsd did got it some wider industrial use.

Like OpenSSH? ;)

I mean the actual kernel. But, you know, needs a killer feature people want to deploy at scale.

And like OpenSSH, due to the BSD licensing, it will be quickly be ported to other operating systems. Which is an advantage to us all, a great thing for humanity.

delusional every professional uses PuTTY /s

Of course they do. And what is that thing on the other side that Putty connects to for our professional friends in vast majority of cases?

...I was going to make a joke about Windows servers, but actually AFAIK Windows just uses OpenSSH for its SSH server so that kinda proves the point doesn't it?

Reply on news.ycombinator.com