Building the Linux kernel with clang | Hacker News Reader

Building the Linux kernel with clang

lwn.net

132 pointsingve61 comments

This is a great step forward. Compiling with clang will allow kernel developers to run code checks and analysis unavailable to gcc.

This will definitely have a positive effect on code quality.

Not even mentioning it should be a little bit easier to configure IDEs like CLion for working with Linux source tree.

Is there a list of checks and analysis that you can't do from a gcc pipeline?

Maybe he's talking about the sanitizers? eg: https://clang.llvm.org/docs/AddressSanitizer.html

Those sanitizers are present in gcc too[0].

[0] https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.h...

ah! thanks

Present != Equal, feature wise. Not by a long shot. At the same conf, one of my friends who develops the sanitizers said it's easier to develop for LLVM, so that's where Dev work is done.

> Present != Equal, feature wise.

It is developed by google[0] afaik. The link says it is present in LLVM and gcc. I don't know what makes the difference (may be License?). On the contrary I have felt gcc better for my work. I have been following a clang bug[1] for long and have seen no fix so far.

[0] https://github.com/google/sanitizers/wiki/AddressSanitizer

[1] https://bugs.llvm.org/show_bug.cgi?id=23910

Clang has better support for plugins and calling it as a library; that sort of thing was resisted by GCC. Google has a bunch of large-scale tooling based on the Clang frontend which AFAIK wouldn't be possible with GCC.

sure sure, but today (and even two years ago) you can add -fsanitize={address,undefined,etc} on both clang and gcc. So this does not answer the question.

All sanitizers are also available in GCC, so no.

Are you sure? Last time I checked the ones llvm had and gcc did not were:

Dynamic alloca overflow detection

Symbol size changing for global variables

Adaptive global redzone sizes

Invalid pointer pairs detection

Support for no_sanitize attribute

Support for dead stripping of globals on Linux

I might be out of date on some or even all, so please correct me where I'm wrong. Thank you in advance.

Wrong. Not all kernel versions.

Sanitizers are run-time tools; I'd be surprised if any of them worked out of the box with the kernel.

Address sanitizer was "ported" to the kernel a while ago; not sure if UBSAN is or will be in the future.

There are kernel versions. KASAN, KUBSAN, KMSAN.

the static analyzer, along with all the other checks available from clang-tidy: https://clang.llvm.org/extra/clang-tidy/checks/list.html

I'm surprised there is a static analyzer available for clang that isn't somewhat equivalently offered in gcc toolchains. I guess I'm assuming folks would be using something like Coverity for major static analysis.

Specifically, this sounds like it is just preferring the other toolchain. Are there literally things that don't have equivalents?

I know this isn't the opinion of many kernel developers, but I think a lot of the problems encountered are better solved by removing proprietary GNU C extensions from the kernel code. I would love to start sending patches but I expect they'd just be NACKed.

Sounds like this is a prime example where evidence would help. Just thinking "a lot of the problems..." doesn't do much to show why that is the case. Especially without even knowing which extensions you are specifically referring to.

That is, my guess is these extensions are the equivalent of Chesterton's fence. They were introduced for a reason. If you have evidence that that reason is no longer apropros, present the evidence. But don't just complain about the fence.

I mean, the problems faced by compiling the kernel with Clang (or any other standards-implementing C compiler). The problems are pretty easy to find, just look at the relevant commits in Clang and Linux. The main problem Clang faced with the kernel was lots of use of GNU C extensions everywhere.

I think it's self-evident that the use of proprietary, nonstandard language extensions is a bad idea. It leads to multi-year efforts getting other compilers to support "features" that aren't in the spec (and as a result, are typically underspeced by the compilers that implement them and require reverse engineering to fully understand and implement correctly). Linux isn't written in C, it's written in some other language that's deceptively similar to C. It should be enough to write a compiler that accurately implements the C standard to compile the kernel with.

But, I could just as easily say it is self-evident that the use of proprietary, nonstandard languages extensions was required to get the linux kernel written.

Don't get me wrong, I'm highly amenable to this argument, but it is not an evidence based one.

Now, there could be some easy evidence that would help. For example, evidence that kernels which are built using clang build just as easily using gcc. Or icc. Or ___. That would help. Right off, I do not know if that is the case.

I mean, I said it was self-evident but then provided evidence as well. Here is the evidence:

- Multi-year effort by Clang, a well regarded standards-compliant C compiler, to support the kernel

- Proprietary extensions do not conform to the same strict guidelines for specifications as the C standard and may require reverse engineering to implement correctly

- C is defined by the C specification and Linux is not written in C by this metric

- Therefore, C compilers cannot compile the kernel

Correct me if I'm wrong but I cannot extract similar facts from your comment.

But my counter evidence was that these extensions were required to actually build the kernel. Which is also somewhat self-evident.

So my question to you is if you have evidence that those extensions are not required to actually build the kernel? I get that they are the "sticking" point to moving to clang. But are they also required for the kernel to actually do its job?

I see what you mean. Yes, there are counter examples of kernels that can be compiled without extensions. With my rough knowldge of the Linux kernel in particular, I imagine there are a few places where the use of extensions makes for much better code than without, and for those places the change is of debatable utility. However, extensions are the rule rather than the exception for the kernel. The bulk of the kernel could be written without them.

Unix managed to be written without those extensions.

Which one? And did it successfully compile with multiple compilers?

> Which one?

UNIX, the genesis of C and all that derived from AT&T original code.

> And did it successfully compile with multiple compilers?

Any that was K&R C compatibile.

GCC was a toy compiler not used for any serious UNIX development until Sun decided to start selling the SDK as part of a SunOS developers edition, quickly followed by other UNIX vendors.

So by definition, until GCC became an usefull project, there were no UNIX kernels written with GCC C extensions.

So, I would have understood if you said the original AT&T UNIX. I've always heard the term used to cover the entire family. And, I honestly had no clue whether or not the likes of SunOS/HP-UX/etc were compiled with multiple compilers. My understanding was each typically came with their own compiler.

FreeBSD can be built with gcc, although it's built with clang by default: https://wiki.freebsd.org/BuildingFreeBSDWithClang

The patches to make it work for icc is simple enough that this is almost a non-issue: https://lists.freebsd.org/pipermail/freebsd-arch/2003-Novemb... (albeit old)

An inability to compile on nothing but GCC is a feature for a lot of zealots out there.

I wonder what Torvalds' opinion on Clang is. He had harsh words to say about a recent version of gcc, as I remember.

IIRC, his complaints are of the nasal demon variety.

He hates compiler writers being tricky.

Context:

In the C community, undefined behavior may be humorously referred to as "nasal demons", after a comp.std.c post that explained undefined behavior as allowing the compiler to do anything it chooses, even "to make demons fly out of your nose".

https://en.wikipedia.org/wiki/Undefined_behavior

I found that clang compiled code was requiring more memory on the stack than the same code compiling with gcc. Was it some obscure switch that did that? no idea. However with the Linux kernel you have some tight limitations with regard to stack size. Could the author run his clang compiled kernel? The article does not tell.

how do you even have VLAs where the array is the non-terminal field? do you embed the size of the array in the type definition or in the structure?

It's weird. As far as I can tell, GCC's documentation[1] clearly states:

Flexible array members may only appear as the last member of a struct that is otherwise non-empty.

What is the article referring to, then?

1: https://gcc.gnu.org/onlinedocs/gcc-7.2.0/gcc/Zero-Length.htm...

flexible array member ≠ variable-length array

See: https://news.ycombinator.com/item?id=15303211

It has to be in the definition since it is a VLA, not a flexible array member. Accessing the VLAs or members after/between them will generate an offset based on whatever values were used to define the length of the VLAs, so the definition needs to be in scope. Seems like the use case is for struct types defined within a function and used only within that same function execution, or perhaps also accessed through another execution of that same function that results in the exact same definition. In the latter case, you would need to cast the memory to void* for storage between executions, and then back to the regenerated type, since you can only reference types within the scope that they are defined.

Weird, but I can see the potential utility when you're optimizing for locality in heap allocations, since the allocator will give you a single contiguous memory block with all of the VLAs, while making several allocations for multiple VLAs could very well return disjoint memory ranges. And then there's the aesthetic utility of behaving like objects in higher-level languages, where having multiple variable-length arrays within an object is standard.

See https://stackoverflow.com/questions/14629504/variable-length... and linked pages.

This would be a really useful feature in C if it was made more general, even if the scope of the definition remained limited. But with GCC 7

  struct { int arr[n]; } s = { 0 };

gives

  error: variable-sized object may not be initialized

And

  (struct { int arr[n]; }){ 0 }

gives

  error: compound literal has variable size

GCC still doesn't even permit a simpler compound literal like (int[n]){ 0 }. I understand that compound literals and VLAs are distinct, and that the above limitations are, strictly speaking, consistent with the standard constructs. But I would think that once you have variable-length struct members that it wouldn't be too difficult to unify the semantics of VLAs and compound literals. If GCC can't be bothered to improve language consistency, then that's pretty strong evidence that VLA struct members are a dead-end and I can't fault clang for avoiding it.

But if GCC did improve consistency, it might significantly help push C toward safer array semantics. The last hurdle would be definition scope and working out how declarations worked. Presumably the ABI would require fat pointers that embed VLA annotations. Getting there would be a long, winding road, but this is how it would happen. It's complex, but the cost+benefit ratio would, I think, be better for both compiler writer and programmer compared to the complexity of C++.

You're thinking of flexible or zero-length array members (`int[]` as the terminal field, or mostly equivalently, where supported, `int[0]`). VLAs in a struct only really make sense for a local type i.e. a struct defined and used within a function:

    void foo (int n) {
        struct S { int x[n]; };
    }

https://gcc.gnu.org/onlinedocs/gcc/Variable-Length.html https://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html

thanks. that makes sense.

    #include <stdio.h>
    void foo (int n) {
      struct S { int x[n]; int y; };

      printf("offset: %ld\n", __builtin_offsetof(struct S, y));
    }

    int main() {
      foo(2);
      foo(16);
    }

    offset: 8
    offset: 64

It's about whether you can put stack allocated VLAs into temporary structs:

   keno@anubis:~$ cat test2.c
   void bar(int *);
   int foo(int n) {
       struct {
           int non_terminal_val[n];
           int some_other_member;
       } val;
       bar(val.non_terminal_val);
       return val.some_other_member;
    }
    keno@anubis:~$ gcc -c test2.c
    keno@anubis:~$ clang test2.c
    test2.c:4:13: error: fields must have a constant size: 
    'variable length array in
          structure' extension will never be supported
            int non_terminal_val[n];
                ^
    1 error generated.

"will never be supported" I wonder how many times you have to be asked before you put that in the error message.

The Linux kernel use the variable length array in the structure which is a GCC extension and not part of the C standard.

Clang don't want to support that GCC extension because it's too ugly.

Given the number of operating system kernels in the world, it might be helpful to change the title to say "the Linux kernel" instead of "the kernel". I understand TFA is on lwn.net, which has "Linux" in it's title, so it's clear there, but here it becomes a little vague.

Perhaps, that was in reference to https://www.kernel.org/.

This. FreeBSD's been building on CLANG for amd64 and i386 since before 10.0 (now on 11).

See, for ex., https://wiki.freebsd.org/BuildingFreeBSDWithClang https://unix.stackexchange.com/questions/49906/why-is-freebs...

OpenBSD is also now building the kernel/userland and most of the ports tree with Clang for i386/amd64 and arm64. 6.2 will ship with it as the default system compiler.

> Kroah-Hartman said that he wished there was a competitor to the Linux kernel.

Linux people refuse to look outside their vacuum chamber.

That was not what he said. Article is wrong. He started he wants more competition amongst compiler vendors for the benefit of the kernel.

> Sorry, but you might have missed my follow-on comment which was, "I wish we had a viable competitor to the Linux kernel as well". Jake correctly quoted me here.

https://lwn.net/Articles/734405/

I, for one, assumed they meant linux.

You may have background knowledge.

It's about a kernel for which compiling with Clang is (1) interesting and (2) novel. If you know Linux, you'll recognize that's Linux.

OK! We've added “Linux”.

Thanks!

Oh, come on... Don't tell me you even for a second thought it might have been another kernel.

Actually, L in LWN no longer stands for Linux. From their FAQ <https://lwn.net/op/FAQ.lwn >:

What does LWN stand for, anyway?

LWN, initially, was "Linux Weekly News." That name has been deemphasized over time as we have moved beyond just the weekly coverage, and as we have looked at the free software community as a whole. We have yet to come up with a better meaning for LWN, however.

What about Little Wicked Newsletter?

Clang builds the kernel, EFF leaves W3C... What's next ? Amazon rewrites libc ?

1000 years of darkness!!!

Every C compiler has a libc.

I didn't know...

https://libcxx.llvm.org/

But according to what I see on that page, my interrogation remains : once the kernel is built without gcc, then GNU's power will fade away.

GPL and HTML have, in their own ways, enforced that some important code remains open (and sharable and modifiable). By weakening them, the openness will be weakened too.

Of course it won't be 1000 years of darkness, but well, it makes me a bit sad :-)

Why do you think Apple, Google, Sony, Nintendo and many other embedded OEMs have moved to clang?

It wasn't only due to the better plugin support.

Reply on news.ycombinator.com