Hacker News Reader

Comment by azakai | Hacker News Reader

Oh nice, the results are pretty good!

> On a set of kernel-intensive benchmarks (including NGINX and Redis) the fraction of kernel CPU time Biscuit spends on HLL features (primarily garbage collection and thread stack expansion checks) ranges up to 13%. The longest single GC-related pause suffered by NGINX was 115 microseconds; the longest observed sum of GC delays to a complete NGINX client request was 600 microseconds. In experiments comparing nearly identical system call, page fault, and context switch code paths written in Go and C, the Go version was 5% to 15% slower.

10% slowdown in return for memory safety could be a worthwhile tradeoff in some cases. And GC pauses were almost not an issue (less than 1ms in the worst measured).

View on HN

This is pretty remarkable and surprising that it's only 5-15% slower. The Go compiler isn't very aggressive about optimizing even ignoring GC overhead concerns. I'm also curious to understand better how they bootstrapped the dynamic memory and scheduling facilities since Go obviously relies on an underlying operating system for these things.

I believe the Midori project at Microsoft found similar in trying to write an OS in C#.

Several posts from Joe Duffy, who led that project, are at [1].

[1]: http://joeduffyblog.com/2015/11/03/blogging-about-midori/

iirc they modified Go runtime

Obviously, usually a language runtime expects an underlying OS.

When one isn't available, the runtime plays the role of an OS, and much be adapted as such.

It's because typically for most applications the OS isn't very much overhead.

If the kernel was only using 1% of the CPU time before, and you rewrite it in QBASIC for a 5x slowdown, your application only sees a ~4% slowdown...

But didn't you just explain why our kernels should be in QBASIC? Or any other safe language? I feel like this is the next version of Christmas not being true. That unsafe languages were a dumb scam.

Most of it comes from a few bad decisions in C in particular that we ended up with stuck for a very long time (nulls, badly designed array type) OSes were written to run on very different hardware from what we use now and over time the hardware was adapted to running specifically C code well. High-level languages like we expect these days hardly existed at all before the mid 90s or so, and they have been gaining in dominance ever since.

You will notice that Go shares some authors with C.

They did the same with C back in the day, versus what was being done in systems programming since the 60's.

The language C builds on, B, is a spin off of BCPL, a language designed to Bootstrap CPL.

Thanks to UNIX's success we ended up with a language whose original purpose was to bootstrap compilers, not to be a full stack programming language.

Here, systems programming in 1961, 10 years before C was born, still being sold by Unisys.

https://en.wikipedia.org/wiki/Burroughs_large_systems

Or at US military,

https://en.wikipedia.org/wiki/JOVIAL

Or during the 70's,

https://en.wikipedia.org/wiki/PL/8

https://www.computerhistory.org/revolution/input-output/14/3...

https://en.wikipedia.org/wiki/Interlisp

https://en.wikipedia.org/wiki/BLISS

https://en.wikipedia.org/wiki/Modula-2

https://en.wikipedia.org/wiki/Modula-2%2B

Remember that project Bell Labs stepped away from, which gave C's authors plenty of free time?

https://en.wikipedia.org/wiki/PL/I

https://multicians.org/myths.html

"Thirty Years Later: Lessons from the Multics Security Evaluation"

https://hack.org/mc/texts/classic-multics.pdf

Future programming historians may indeed look back on Ken Thompson as the Thomas Midgley of the software world, having plagued us more than once with costly and dangerous mistakes: null pointers, C, and the abhorrent POSIX API.

> In 1923, Midgley took a long vacation in Miami, Florida, to cure himself of lead poisoning. He "[found] that my lungs have been affected and that it is necessary to drop all work and get a large supply of fresh air".[9]

Null terminated strings.

In, "Trusting Trust" Ken tacitly admitted that C itself was the inside job. Once you align yourself with Cs core tenets, performance above all else, the game itself is up and your mind has been infected against looking at things holistically.

I’m not sure what you’re trying to tell me but I’m certainly aware of these facts already.

Comments in these forums are directed at two populations, the direct participants and everyone else viewing from the bleachers. I don't think the parent is implying that you don't know these things, but added them for context.

This is true. I got a lot of downvotes for my GP.

That was indeed the case.

> High-level languages like we expect these days hardly existed at all before the mid 90s or so

I beg to differ.

Lisp has been influential since the 60s, especially in certain niches (AI, symbolic algebra, etc.) and is still going strong.

Smalltalk was used throughout the 70s (although first "released" in 1980), and is far more "dynamic" than the scripting languages widely used today (e.g. Python); the classic example is 'ifTrue' being a method call which we can override (via a built-in live editor, no less!).

Prolog has been around for a while, and saw a lot of attention in the 1980s. It's much more high-level than most languages, since its interpreter runs a search algorithm to calculate results, rather than dumbly stepping through a single execution path.

Most of the cool features popping up in "modern" languages, like pattern-matching, currying, (Hindley-Milner) type inference, etc. came from ML in the 70s.

Scheme is also from the 70s, and has features like tail-call elimination which many languages/platforms are still lacking (spawing a whole sub-genre of blog posts about how to implement it in language X via trampolines). Scheme also brought attention to continuations, giving us things like coroutines. The async/await and yield features added to many languages recently are close cousins of Scheme's call/cc; as are try/catch/throw, for that matter! Although call/cc is undelimited, delimited continuations have been around since the 80s.

That's just off the top of my head, wearing my Programming Language Theory hat. If I put on my Hacker hat I could mention Sh, Snobol, Awk, Icon, ABC (Python's predecessor), etc.

I’m aware of all these facts but Lisp wasn’t practical except on Lisp machines for quite a long time, and most of these languages only started running on microcomputers in the mid to late 80s. Caml Light, (Xavier Leroy) the standardization of ML, object oriented scripting languages, etc.

My initial statement was too strong, what I meant to say was that the higher level languages were not practical for commodity application development and distribution until more recently. They’ve been along for a very long time but Smalltalk is an infamously isolated system and it was very slow even when running on an Alto (I’ve used it!). I admittedly know much less about the practical history of Scheme and Prolog.

I think you are both right.

To jolux's point, Scala, F#, Ocaml/Reason, Clojure, Haskell, Erlang while not as huge as Java and C#, they have billions in value attributable to them.

The quoted text says that overhead on the kernel code was 15%, not the application.

Yes, the throughput results are pretty great, as is this work in general. It's worth noting the memory usage, though -- "Go’s garbage collector needs a factor of 2 to 3 of heap headroom to run efficiently (see section 8.6)".

Amortize it over the life the machine. Go is no gold standard, but what does it mean in terms of total energy cost? Manufacturing cost? A twice larger cache and half the frequency ram bus? I dunno what the answer is, but I think being able to run a safe language for the OS gets us to an entirely different regime.

Then use Rust.

Then improve compile times and borrow checker usability.

If you don't mind, what are your grievances with Rust's borrow checker, and do you have any ideas on how it could be improved, or any alternative implementations of the idea of "ownership" and "borrowing" that do some things better?

Here is one example,

https://github.com/rust-lang/rust/issues/63818

Alternative ideas, here are four, combining the productivity of automatic memory management with ownership,

https://swift.org/blog/swift-5-exclusivity/

https://github.com/apple/swift/blob/master/docs/OwnershipMan...

https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in...

"Linear Haskell", https://arxiv.org/abs/1710.09756

In the context of Rust, maybe showing lifetimes graphically in an IDE would help productivity versus languages that offer automatic memory management alongside ownership.

> 10% slowdown in return for memory safety could be a worthwhile tradeoff in some cases. And GC pauses were almost not an issue (less than 1ms in the worst measured).

Implementing Nim's bounded time GC could cut that overhead down and defer work for idle periods. That should be first priority when implementing an OS in managed code.

We've been discussing the "10% slowdown" for two decades now, since at least the time of Java's first release.

Microsoft even tried to rewrite Windows in C# on the premise.

Meanwhile in 2020 Java programs still suck ass to use. (Sorry for the crude language, but it's true.) Microsoft memoryholed the whole C# thing and went back to promoting C++.

Yes, because GC is a woefully inefficient memory management strategy.

Rust is not just the future of systems programming -- it's the future of all programming. You may not program directly in Rust the language, but borrow-checked static memory management is coming to an application language near you.

What is inefficient are programmers that don't learn to use the features for value types and deterministic memory management that many GC languages offer them.

It is faster to write an app with GC than to fight the borrow checker. For OS kernel development Rust may be worth the fight, but for userland a language with GC is better in most cases (much easier to learn, use and maintain for relatively small speed and memory penalty).

> Meanwhile in 2020 Java programs still suck ass to use

Isn't that due to memory usage, rather than speed?

I agree with the poster below. GC's problem isn't speed, per se; the problem is that GC only works effectively if you assume infinite memory and cache. Once you start approaching memory limits you start feeling pain.

Another problem is that most developers using GC languages seem to keep failing at learning that there are other means to manage memory in GC languages, but writing new is so easy...

What other means are you referring to here? Arenas are all that comes to mind.

Many GC enabled languages offer:

- value types

- some form of RAII

- untraced reference (aka raw pointers)

- stack and global memory segment allocation

Examples of such languages:

Mesa/Cedar, Active Oberon, Mesa/Cedar, Component Pascal, D, Modula-3, Nim, Swift, C# (depends on the version, C# 9 is already quite sweet)

All those are just ways to avoid actually using the GC. Which proves the point: GC is ineffective and works best when not used at all.

The future is not garbage collected.

pjmlp's point is that GC can be the default, and manual memory management can be used where necessary.

It's essentially the Pareto principle. Most of your garbage is generated by a minority of your program. So only small bits of your program need to be optimised to not use the GC.

Yes that is what anti-GC crowd tends to sell, except that it the same set of capabilities that languages like C also offer, with the difference that use after free and leaks abound.

The future is pretty much GC with affine types, even Rust has to deal with crude RC<RefCell<>>, without the corresponding tracing GC performance, when doing GUI applications.

Yet another one that doesn't understand GCs.

And you are mistaken with "Going Native fever from MS", it was a flop that produced UWP, now being fixed with Project Reunion.

Most of the System C# features from Midori ended up in C# 7.x, 8.0 and 9.0 brings even more goodies like naked function pointers and C ABI to call into .NET code.

Yeah java is the gold standard of GC language design. It sucks so all GC languages must be horrible.

Excuse me, Java is a horrible design. The gold standard is Common Lisp, and this is wonderful.

Also Go's GC still is terribly naive. They recently fixed the latency problem, but it still needs to be redesigned to match a good Common Lisp GC.

I suspect that higherordermap's comment was sarcasm...

At a cost of requiring two to three times as much RAM as a non-GC solution to get that performance, as noted elsewhere in this thread. Indeed, it's been known for years[1] that to have comparable performance to an equivalent program without GC, a GC-enabled program needs several times as much memory available.

Yeah, no thank you. The OS community should stick with Rust.

[1] Matthew Hertz, Emery D. Berger. Quantifying the performance of garbage collection. ACM SIGPLAN Notices, volume 40, issue 10, October 2005. https://people.cs.umass.edu/~emery/pubs/gcvsmalloc.pdf

The OS community is almost only C and C++. If by community you mean the people working on operating systems. Or you're only considering a small subset to make your point?

The OS community seems to use other tools.

> Swift is intended as a replacement for C-based languages (C, C++, and Objective-C).

-- https://swift.org/about/

Reference Counting, chapter 5 from The Garbage Collection Handbook.

Yeah once you have a old paper that compares a not-particularly-efficient JVM of that time with manual memory management and concludes GC needs 3x memory, it is even easier to spread GC misinformation.

> The longest single GC-related pause suffered by NGINX was 115 microseconds;

Wow!

It is funny that we have these capabilities within arms (fingers) reach but it takes someone daring enough to put symbols in a certain order to show that a different world is possible.

It is impossible to fight cargo cult with facts, only a generation change helps.

Joe Duffy mentioned in one of his talks (Rust Keynote) that even with Midori running in front of them (it even powered part of Bing for a while), there were people on the Windows team quite sceptical that it was possible at all.

Anti-GC cargo cult goes a long way.

Seeing these results touted as being pretty good makes me wonder why we haven't fully adopted microkernels yet. Their singular disadvantage, which is the need for two IPC calls vs one for a monolithic kernel, is completely irrelevant in comparison to these numbers. The average IPC cost of an L4 IPC call is about 500 cycles, which is about half a microsecond on a 1ghz processor. Cache locality is a big deal, and L4, which can fit entirely within a core's l2 cache, is king when it comes to cache locality.

And you don't have to worry about memory safety either. Memory safety has been formally proven as part of the SeL4 spec.

I've also wondered why we haven't seen more industrial effort around using sel4. Seems like it'd be perfect for mission critical firmware in cars and such too. For replacing linux, I guess there is still a lot of effort that would be required, but it seems worth it from my admittedly surface viewpoint.

> Seeing these results touted as being pretty good makes me wonder why we haven't fully adopted microkernels yet. Their singular disadvantage, which is the need for two IPC calls vs one for a monolithic kernel, is completely irrelevant in comparison to these numbers.

And that's even the worst case of synchronous/blocking calls. With different API design - e.g. asynchronous interactions as done with io_uring or IOCP - the latency becomes less important as long as the overall efficiency and thereby throughput is high.

Depends on who "we" is. Apple's OS is largely a microkernel design & that ships on billions of devices. Build services for that OS at Apple from a safety/security/performance/battery life perspective is a dream compared to other OSes. Granted outside of that it's a bit rarer.

>Apple's OS is largely a microkernel design

They are hybrid design, or some would even refer them to monolithic. Definitely not microkernel, although they are moving in that direction with DriverKit and other kit initiative.

True they are hybrid but most drivers lived in user space except for ultra critical performance stuff if I recall correctly (mainly graphics stuff if I recall correctly but I never took a complete inventory). IPC was a core part of how both drivers, system daemons, and apps communicated with each other (with thin shim APIs to make it friendlier). It was pretty great at how consistent a lot of it was.

Hypervisors are the microkernels that are socially acceptable.

That is 500 cycles without mitigation for CPU bugs, and without considering the non-local performance effects.

Non-local performance effects arise from TLB and other cache invalidation, which is required when changing from one task to another. You can't avoid that without putting everything in the same address space, which would make the system a fake microkernel. Fake microkernels have the less-readable code of microkernel design but without any benefits other than buzzword compliance.

CPU bugs like Meltdown and Spectre mean that every IPC now needs to blow away all sorts of caches and prediction. For example, you wipe out branch history. The cycles spent during the IPC call are only a tiny portion of the cost. The loss of cache content (TLB, branch history, data cache, code cache, etc.) greatly slows down the CPU.

Oh, I wouldn't dismiss "fake microkernels" so completely. One of them used to power a rather popular (proprietary) NAS Filer. I don't have a reference -- I believe this little gem never found its way into a publication (though the file system that ran on top of it did: https://www.cs.princeton.edu/courses/archive/fall04/cos318/d...).

That is the thing, when you put all mitigations in place, the supposed performance gain is already out of the window.

And there is the whole point that many of the performance issues with microkernels have long been solved.

Well, for one, those bugs aren't an issue on non-shared devices, and anything else outside of x86. That is not insignificant...that is still the majority of devices.

Two, the benchmarks for IPC usually include the cost of cache invalidation. This one [0] for SeL4 clearly shows the difference on Skylake for the kernels compiled with Meltdown mitigations. Still in the sub-microsecond range.

[0] https://sel4.systems/About/Performance/home.pml

Plenty of non-x86 got hit by those bugs. ARM got both. PowerPC was hit as well.

Meltdown: Intel x86, IBM POWER, ARM Cortex-A75

Spectre: Intel x86, AMD x86, ARM (Cortex-R7, Cortex-R8, Cortex-A8, Cortex-A9, Cortex-A15, Cortex-A17, Cortex-A57, Cortex-A72, Cortex-A73, various Apple-designed cores) and some IBM hardware.

Lots of devices are shared, thanks to things like javascript.

How about you choose rust. You get 0% slowdown while also getting a more safe language in general.

Also never finish implementing, just blogging about it and mentioning rust everywhere.

Plaudits to all involved.

Reply on news.ycombinator.com