Static Calls in Linux 5.10 | Hacker News Reader

Static Calls in Linux 5.10

blog.yossarian.net

98 pointsingve20 comments

Author here: I'm now wishing that I had made the title slightly flashier, since the implications of this change are significant: many bits of the Linux kernel that originally had to go through retpolines (as part of the original Spectre v2 mitigation) can now go through "static calls" instead, yielding a decent performance boost.

Great post!

Percentage-wise, how many hot call points have moved to static call sites versus remain based on retpoline? Are there any in-vivo performance measurements?

Thank you!

Unfortunately, I don't have numbers for this. The patchset itself contains some changes to `perf_events`, as well as a few other subsystems. I also saw some summaries documenting performance improvements in the mid-single-digit range (3-5%), but I'm not aware of anything in-vivo. The kernel mailing list for this patch range[1] might be a good reference.

[1]: https://lore.kernel.org/lkml/cover.1547073843.git.jpoimboe@r...

There aren't many static calls yet, but for example I am planning to use them for KVM (which has lots of indirect calls in order to support both Intel and AMD virtualization extensions). Besides some improvements in sheer cycle count, it would also allow some code cleanups, removing code that has to be duplicated to avoid sticking a retpoline in the middle of a hot path.

Learning curve for kernel development gets steeper and steeper, and there seems nothing we can do about it.

It's a well designed machine already, so we might not care about it slipping into the realm of mysterious black boxes, but someday it's going to become one. A black box that exceeds human understanding

IMO, individual parts of the Linux kernel are still relatively digestable.

I had never looked at this particular part before, and I could figure out maybe 95% of the behavior with a few hours of close reading (which, IMO, is a reasonable amount of time for a piece of code that's outside of both userspace and peeks under C's semantics to do trampolines, etc).

My experience so far (mostly writing kernel modules) is that the complexity comes from discovery, not the actual code itself: I have a much harder time finding the right APIs to use than I do actually writing things once I'm on the right track.

I haven't done any kernel programming yet, but the curiosity has been there. What has been your main difficulty in finding the right APIs? Lack of documentation? Poor or unclear documentation? Something else?

> What has been your main difficulty in finding the right APIs? Lack of documentation? Poor or unclear documentation? Something else?

Definitely poor (or more often, old) documentation. In particular, this has popped up for me when trying to write modules that add entries to procfs or sysfs: these interfaces are documented and widely used, but much of the documentation is years old and the APIs (or their best practices) have changed subtly. My best reference has just been the subsystem source itself, as well as other kernel modules that I know work.

Another thing that pops up: the Linux kernel is so big and has so many architecture-specific subtrees that it can be difficult to find the "effective" definition for some function or macro. Woboq and Elixir help with this a bit, but it's very easy to go down a rabbit hole.

Kernel documentation isn't great, and it tends to be distributed as one big long-form text file. This has it's benefits because it encourages explaining the rationale behind a given design, but it also means that you can end up with rotten old information looking just as recent as the new stuff. Man pages often don't actually cover a complete listing of everything an API has, too.

eBPF is a good microcosm of this, where using it isn't all that complicated but actually working out what to do can take ages because most resources use a middle-man to talk to the kernel.

> and there seems nothing we can do about it.

The complexity's moving into the hardware, so I wouldn't assume the kernel'd be having to handle this sort of thing in 50 yrs.

That's a very interesting take. I wonder how extensive future SoC will be, and how close to "thinking silicon" will we get.

I've spent far too much time reading this while I'm supposed to be working - and I still don't understand it. Hopefully someone can help.

    __static_call(name)

is a macro that expands to (name inlined):

    ({         \
        __ADDRESSABLE(__SCK__x86_pmu_add);    \
        &__SCT__x86_pmu_add_tramp;     \
    })

It is used in this context

    #define static_call(name) __static_call(name)

Which is used in this context

    static_call(name)(args...);

But... this (expanding the last version) isn't valid syntax?

    ({         \
        __ADDRESSABLE(__SCK__x86_pmu_add);    \
        &__SCT__x86_pmu_add_tramp;     \
    })(args...)

A block isn't callable in C?

---

It might help if I understood what the final product (assembly) is supposed to look like, but I don't. For instance, at the end of the article the final assembly produced is discussed, but it's not clear to me what code is producing that assembly/what the surrounding assembly is supposed to look like. Taking the last example

    call __SCK__x86_pmu_add
    ; in call + "trampoline"
    ret ; "trampoline" returns, completing the call

Is this code generated inside a trampoline function (so that last ret is actually a ret)? Is this code called inside the caller, and someone played with the return address? Something else?

I believe in GCC C a block can return a value. In this case it returns a pointer to __SCT__x86_pmu_add_tramp. And that pointer is then treated as a function.

Author here: It's a GCC extension[1]!

> Is this code generated inside a trampoline function (so that last ret is actually a ret)? Is this code called inside the caller, and someone played with the return address? Something else?

The first: the "trampoline" in this case is just a ret. Calling the trampoline causes the compiler to emit a call instruction (which does the normal stack ops associated with one), so our "trampoline" just cleans that call up, completing the behavior of `static_call` in the case where we don't actually have a function registered.

[1]: https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html

Good to know about that gcc extension, that's cool :)

Ok, I think I understand what's going on now. To write it down in a bit of a different way for anyone else who is confused

Start with some user written code like this

    ... usercode(...) {
        ...
        static_call(x86_pmu_add)(args..);
        ...
    }

That code is expanded to (desugaring the gcc extension as well)

   ... usercode(...) {
       ...
       __ADDRESSABLE(__SCK__x86_pmu_add); // Just making sure the compiler doesn't think this is unused
       (&__SCT__x86_pmu_add_tramp)(args...);
       ...
    }

__SCT__x86_pmu_add_tramp is where we've "jitted" the function address to jump to. If we've written a function pointer, that assembly is just a "jmp" instruction to the function we wrote (note that one function jumping to another is just a tail call), and in the other case it's just a "ret" instruction because we don't want to do anything.

Thanks woodruffw! This is some really Interesting (and intricate) code

You're welcome, and thanks for breaking it out like that!

The kernel developers, of course, deserve the majority of the praise -- I just summarized my understanding as I read through it :-)

The overhead is apparently quite real and easily measurable. There was a proposal (not sure if accepted) to make the KVM virtualization module monolithic to avoid indirect jumps through a table of function pointers called kvm_x86_ops: https://people.redhat.com/~aarcange/slides/2019-KVM-monolith... (page 13 ff.)

The problem with that proposal was that it didn't allow compiling a non-modular kernel with both AMD and Intel processor support. But we'll use static calls instead. :)

Since the main thread about Kernel Release is old now I'm asking here: Is the PREMPT_RT patch fully merged into the mainline yet, I heard it was supposed to happen sometime in 2020, may be 5.10 has it merged already?

Small building pieces of it are being included in each release, including this one, not sure when they will finish.

Reply on news.ycombinator.com