LLVM 9.0 | Hacker News Reader

LLVM 9.0

lists.llvm.org

307 pointsAndyKelley107 comments

Note that this release (9.x) marks the first to be no longer available under the original permissive NCSA (BSD-like) LLVM license, going forward the LLVM project has moved to an "Apache 2.0 with LLVM Exceptions" for everything. And not just the compiler itself, but also sub-projects like the supporting libc++/libc++abi standard libraries. This change only happened due to corporate pressure, and was done without considering the larger impact towards the open source community.

This will undoubtedly effect many projects and companies alike going forward, but notably the OpenBSD project can no longer merge in new changes, which will ship with 8.0.1 later this year in 6.6. The Apache 2.0 license is not considered permissive enough to be included in the base system of OpenBSD, it muddies the line between Copyright law and US Contract law; adding additional restrictions related to patents that may terminate rights.

https://lists.llvm.org/pipermail/llvm-dev/2017-April/112300....

https://www.openbsd.org/policy.html

It is very unfortunate that the LLVM project has ignored the concerns expressed by developers of an open source operating system project that have been eagerly adopting a clang based toolchain, across many platforms (arm64/armv7/i386/amd64/octeon/..) by default, and has been working on new innovative security mitigations (RETGUARD, ROPGadgetFixup framework, register allocator rearranging) along the way.

I will take this opportunity to ask, if you're a current LLVM developer/contributor, please consider making your changes also available under the terms of the original permissive license. For example, explicitly stating publically as such when mailing patchsets or pull requests. This would save having to privately ask permission to backport changes on case-by-case basis.

https://github.com/llvm/llvm-project/blob/release/8.x/clang/...

> This change only happened due to corporate pressure, and was done without considering the impact towards the open source community.

FWIW, it's worth noting that only OpenBSD has this stance, and it was brought up a few times on the mailing list. The general summary is that "every lawyer we've had look into this issue has said that OpenBSD's fears about the patent licensing clauses are FUD, and OpenBSD has provided no legal opinions to back its claims up."

Thanks, can you give some links to that discussion?

It was, as you might guess, a series of very long discussions over several years:

Initial RFC: https://lists.llvm.org/pipermail/llvm-dev/2015-October/09153...

RFC 2: http://lists.llvm.org/pipermail/llvm-dev/2016-September/1047...

RFC 3: http://lists.llvm.org/pipermail/llvm-dev/2017-April/112142.h...

Update: http://lists.llvm.org/pipermail/llvm-dev/2018-October/126991...

OpenBSD objection subthread: http://lists.llvm.org/pipermail/llvm-dev/2017-April/112300.h...

Doesn't the "LLVM Exceptions" part of the license take care of the patent issue you mention?

"[...] if a court of competent jurisdiction determines that the patent provision (Section 3), [...] conflicts with the conditions of the GPLv2, you may retroactively and prospectively choose to deem waived or otherwise exclude such Section(s) of the License [...]"

Source: https://github.com/llvm/llvm-project/blob/master/llvm/LICENS...

No, it doesn't. The aforementioned concerns have nothing to do with the GPL license.

https://github.com/llvm/llvm-project/blob/master/llvm/LICENS...

> .. "If you combine or link compiled forms of this Software with software that is licensed under the GPLv2 .. "

It wouldn't cover anything from OpenBSD, if as excerpted by you, it only covers exceptions related to GPLv2 as I don't believe OpenBSD considers GPLv2 to be acceptable either.

> I don't believe OpenBSD considers GPLv2 to be acceptable either.

The GPLv3 is not acceptable in base at all, GPLv2 is unacceptable for new code, but we still have a few examples in base, very few.

Ah that'd probably be why I was thinking it was any code, small difference that's easy to forget about.

My reading of the LLVM Exceptions statement is that if a judge understands the the Apache 2.0 patent provision section (and others) to not be the same as GPLv2, then the section is waived. The preamble of the GPLv2 summarizes its stance on patents in lay terms that seem to be compatible with what OpenBSD wants:

"We wish to avoid the danger that redistributors of a free program will individually obtain patent licenses, in effect making the program proprietary. To prevent this, we have made it clear that any patent must be licensed for everyone's free use or not licensed at all."

Source: https://opensource.org/licenses/gpl-2.0.php

(I'm not a lawyer)

The exemption you quoted very specifically says it applies "if you combine or link compiled forms of this Software with software that is licensed under the GPLv2 ("Combined Software") and if a court of competent jurisdiction determines that the patent provision […] conflicts with the conditions of the GPLv2" (emphasis mine). I don't know how you could read that as applying to situations where nothing is licensed under the GPLv2.

In other words, it sounds like this would address OpenBSD's concerns if OpenBSD were GPL-licensed — but it isn't.

As someone completely out of the loop:

Are openBSD switching compiler? Are they doing a fork from the 8 version?

How was code relicensed in the first place?

For them to switch (which is a lot of work), there would have to be an alternative to switch to.

So it looks like they'll keep using 8 in the base system, i.e. maintain a de-facto fork (and probably provide 9 through ports).

This would particularly make sense since it's effectively what they did with GCC for years.

What does that mean for compilers that use LLVM underneath? (I'm not a lawyer).

> "Support for asm goto, enabling for example the mainline Linux kernel for x86_64 to build with Clang"

This is big. Support for asm goto was merged into the mainline earlier this year, but now it's released [1]. Aside from the obvious implications of this - being able to build the kernel with LLVM - working with eBPF/XDP just got way easier [2].

[1] https://github.com/llvm/llvm-project/commit/784929d0454c4df6...

[2] https://www.spinics.net/lists/netdev/msg544057.html

Can anyone give a good concise explanation as to why GOTO was not in LLVM from the beginning? It seems like something quite obvious to need. I can only guess that it had something to do with their compliation strategy not being able to deal with the ... nonlinearity* ... introduced by GOTO.

*Probably a better word for that

GOTO was supported in LLVM from the beginning.

The new feature is ASM GOTO. An ASM block is opaque to the compiler, and the compiler treats it similarly to a function call. A called function may contain control flow internal to itself, but can't cause control flow to happen in the caller. ASM GOTO is a new syntax for informing the compiler that this inline ASM may contain a GOTO to a given label. GCC added it around 4.5-ish, and LLVM is adding it in LLVM 9.

*FYI, the name for the thing that GOTO permits which combinations of if and loops doesn't is called "unstructured control flow".

Pros of asm goto: it's useful for introducing low overhead tracing and debugging abstractions. Tracepoints in the Linux kernel are one example [1]. It's also used for implementing certain Meltdown/Spectre mitigations [2].

Cons of asm goto: as parent points out, asm blocks are largely opaque to the compiler, which stands in the way of optimization. Control flow analysis can't take into account internal jumps, and for GCC at least, every asm goto block gets flagged as volatile. That disables whole classes of optimization.

Unless you're relying on hardware-specific instructions that the compiler doesn't know about or expose as an intrinsic, there's always an alternative to asm goto. That said, those alternatives might prove more expensive than the loss of optimizations precluded by the use of asm goto [2].

The pros/cons of including asm goto in clang were discussed in [3].

[1] https://lists.llvm.org/pipermail/llvm-dev/2017-April/111748....

[2] https://lwn.net/Articles/748074/

[3] http://lists.llvm.org/pipermail/llvm-dev/2018-October/127239...

This is not about "goto", it's about "asm goto", which is a gcc extension which allows inline assembly (itself a gcc extension) to jump to a label outside the inline assembly.

Inline assembly is part of the C11 standard as Asm("assembly goes here")

re: nonlinearity, I believe the word is irreducibility (https://en.wikipedia.org/wiki/Control-flow_graph#Reducibilit...)

Does this mean the Linux x86_64 kernel could only be built with gcc before this LLVM release?

Are there any downfalls of switching from gcc -> LLVM for kernel compilation? Why might somebody want to stick with gcc, why might somebody want to switch to LLVM?

> Does this mean the Linux x86_64 kernel could only be built with gcc before this LLVM release?

It was already possible to compile the kernel with clang/llvm. But a few releases ago, the Linux kernel dropped support for being built with older compilers which did not have support for the "asm goto" gcc extension.

The Linux kernel has always targeted gcc exclusively, and makes heavy use of gcc extensions since its very first release. The only reason clang can compile the Linux kernel is that the clang developers implemented support for the many gcc extensions the Linux kernel uses.

Interesting so this means that if someone really want llvm to support an extension (say a C attribute) that GCC supports, then you "just" have to make a contribution to the Linux kernel that use the gcc attribute and that is a minimum useful. Then LLVM will prioritize the support of said attribute.

As such, hardcore developers might become new Linux contributors just as an Indirect (but effective) way to force LLVM/clang to add support for an attribute.

Yes, this is in fact a viable strategy. Source: I contributed a few GCC extension in Clang forced by Linux kernel.

Rational indeed

Who in the world would do that rather than just supporting both compilers?

You don't seems to understand that extensions/attributes can have a usefulness that far outweight cross compiler compatibility?

Type of reasons: Performance critical; Security; Expressivity;

Each of them can be a necessary reason. E.g "polymorphism" in C through transparent unions

No but it's complicated. We shipped ARM64 clang built linux kernels with clang-4.0 in Google Pixel 2.

x86_64 was buildable up until ~4.20 Linux kernel release. (so we could build 4.4, 4.9, 4.14, and 4.19 LTS x86_64 kernels pre-clang-9). The x86_64 maintainers required the use of CONFIG_JUMP_LABEL/asm goto in the 4.20 kernel release IIRC.

Certain configs and drivers needed compiler fixes.

Pretty soon we'll be moving Android and ChromeOS' arm32 kernels over to Clang as well. CONFIG_FUNCTION_TRACER makes the compiler emit __gnu_mcount_nc calls with a non-standard calling convention just on arm32. You should have seen the Apple reviewer's reaction to the LLVM patch implementing support in LLVM for that!

There's also traces of code in the kernel that suggest it was once build-able with ICC at one point. The code looks broken to me and I suspect it doesn't work. That said, if anyone knows anyone compiling the Linux kernel with ICC, I'd love to get in touch.

I want to say Intel was advertising compiling the kernel with ICC around 2009 or so. (ICC and I parted ways around 2010).

Mainstream yes. Android has moved to clang for quite a while now, the Linux kernel is the only GPL piece left.

Fuschia seems to be non-GPL. Are Google trying to remove GPL entirely from Android?

Every major corporation is trying to get rid of GPL. On the embedded market Linux is competing against alternatives like mbed, RTOS, Zephyr, NuttX.

In a couple of more years, we will be back to days of public domain and freeware, a subset of features free and the enticing ones all closed source.

Or running in some cloud (aka mainframe), with the only open part being the browser code, or not even that depending how WebAssembly + WebGL/WebGPU happen to evolve (depending on how Google and Apple feel like driving it).

Then the anti-GPL crowd might remember the days that they got GCC, GNU and the Linux kernel, or maybe not.

Although for many of us that deal daily with commercial software that is kind of irrelevant anyway.

The biggest issue of fuchsia isn't its license, it's rather the design choice to allow proprietary driver blobs. Linux, for the exception of a few blobs here and there, rejects such an idea. In other words, the question is less about license and more about attitude / tolerance for proprietary extensions/tooling. The main power that an open source maintainer has is the design of the codebase and what to accept inside, not the power to litigate a copyleft license. The more it cooperates with proprietary software, the more likely proprietary software can establish itself and restrict user freedoms.

Google has already achieved that with Project Treble by the way.

But they had to pay for the project. For others it might not have been economic to do it. Also, in fact there were Android-custom abstraction layers before project Treble, and the Treble abstraction layer is not around the device drivers but the entire kernel including the drivers.

I am removing GCC/binutils from Android.

I wrote: https://android.googlesource.com/platform/prebuilts/clang/ho...

It has nothing to do with GPL. It has everything to do with Google having 50+ LLVM developers and 0 GCC/binutils developers.

Surely that's just a layer of indirection over whatever the real reason is? What prevents Google from having GCC/binutils developers on staff (whether by hiring or training)?

Supporting one toolchain is already a ton of work; splitting resources between two toolchains is too costly, even for Google. If you don't think so, you underestimate the cost.

Maybe some of them could be reassigned to the NDK tooling then.

I and other NDK users have several ideas of what is missing versus Java/Kotlin.

A lot of my teammates focus on LLVM for Android, including the NDK, which redistributes AOSP LLVM.

> what is missing versus Java/Kotlin.

I'd be happy to pass on feedback, though as worded I'm not sure if you're referring to language features or what exactly?

As mentioned already a couple of times with online discussions with Dan and AMA on Reddit, from me and others:

An NDK version of Android JetPack/AndroidX, for starters proper C++ API, instead of forcing everyone to code 90's style, each re-creating their own C++ wrappers for the Android C++ APIs exposed as C function calls.

Which by the way, given the whole security theme on Android, means more insecure code than if we had those C++ APIs to start with.

Have one shot libraries like Oboe actually be part of NDK and not in another github repo with uncertain future versus the Android platform.

For basic stuff like reading files (using the platform codecs e.g. BitmapFactory, now SAF as well), asking permissions, calling Java from C++, it would be nice if all NDK users wouldn't be force to keep writing those from scratch as rite of passage.

Actually provide build infrastructure to create NDK libraries that can be shared across Android projects, instead of forcing us to somehow cram library and header files into AAR packages while having build scripts extract them for correct placement where cmake and ndk-build expect to find them. cdep never went anywhere, while conan and vcpkg aren't the right answer when many NDK libraries need Java code to come along due to JNI wrappers for like 90% of Android APIs.

Android Studio networking debugger support, currently only supported for networking calls done in Java/Kotlin code.

Easier CMake support for multiple libraries on our projects, which is found lacking versus how easy it is on ndk-build.

As mentioned, these are just a couple of issues that we keep mentioning, always getting some kind of "thanks for the feedback, we will look into it". Just like the NDK status talk at Google IO 2019.

From the outside it doesn't look like NDK is that staffed with engineers, more like a 20% project where a little bit is done every year, like how long it took to fix the header files, adding minimal C++ support to Android Studio, or settling into CMake as the future build system.

EDIT: Almost forgot this one, fix the regressions regarding missing GCC features in clang (Android NDK tracker used to have a couple of them listed).

> C++ libraries etc

We do have limited resources, so we have to prioritize what gets done first. We couldn't have provided good C++ APIs before we were even able to provide good C++. We now do provide a pretty good C++ language experience, but this is still not at the top of the todo list. No one else I've spoken to thinks this is more important than the other work we're doing.

> Actually provide build infrastructure to create NDK libraries that can be shared across Android projects, instead of forcing us to somehow cram library and header files into AAR packages while having build scripts extract them for correct placement where cmake and ndk-build expect to find them. cdep never went anywhere, while conan and vcpkg aren't the right answer when many NDK libraries need Java code to come along due to JNI wrappers for like 90% of Android APIs.

This is what I've been working on for most of the year.

> like how long it took to fix the header files

This only took a single release (the rest of the time was to give users a chance to transition), and we did plenty of other things in the same release, so I'm not sure what you're getting at.

> Easier CMake support for multiple libraries on our projects, which is found lacking versus how easy it is on ndk-build.

Not sure what you mean here, but it sounds like CMake is what you don't like? Just use ndk-build then.

Thanks for jumping in.

So it is really a matter of not having the same resources as Java/Kotlin teams.

Thanks for the update on possible NDK library management.

Regarding the headers, what I am getting at, is why it was such a big issue to fix them, instead of being proper from the start, like other OSes, including embedded ones.

Anyway, thanks for improving the experience versus what it used to be in the early NDK releases.

> Regarding the headers, what I am getting at, is why it was such a big issue to fix them, instead of being proper from the start, like other OSes, including embedded ones.

Our time machine isn't working yet ;)

That's before my time, so all I can really say is that we have the benefit of hindsight here.

Thanks for the feedback, we will look into it. :P

I'm happy to pass these along to Dan and your comment is easy for me to link to, but I don't know why you're suggesting Google move LLVM compiler engineers to work on NDK, which is more of a userspace C++ framework for third party applications to interface with the Android platform. Kind of feels orthogonal to this thread which is about the llvm 9.0 release.

> fix the regressions regarding missing GCC features in clang (Android NDK tracker used to have a couple of them listed).

Now that sounds more like work for Android LLVM engineers, but it sounds like the list is removed? Could you elaborate and point me to some? (Assume anything you claim in a response I'll just ask for a link)

* EDIT * (it seems I can't reply further)

Thanks for the links. I'll pass on both links as feedback and follow up on the state of those two LLVM reported regressions.

I am suggesting that, because as I mentioned it looks understaffed from the outside, either that or there isn't that much willingness to spend resources on improving it beyond "good enough".

Given that you mentioned, that there were actually so many LLVM engineers around, I thought they could give them an hand. :)

Regarding the clang regressations:

https://github.com/android/ndk/issues/742

https://github.com/android/ndk/issues/721

There are way less than I remembered though, sorry for the confusion.

They're following in Apple's stead, I guess.

>Support for asm goto, enabling for example the mainline Linux kernel for x86_64 to build with Clang

This is great for Linux, no longer being dependent on a single compiler for one of the most popular ISAs can only be a good thing for the overall health of the project. To be clear, I have nothing against gcc.

2 cents prediction :

Android will leave linux (fuschia), linux will leave gcc because companies developing the kernel will push for clang, clang will have media coverage, gcc will therefore loose a huge user base and the power of the FSF will go south.

(as a free software zealot ( :-) ), I'm worried)

> LLVM will now remove stores to constant memory (since this is a contradiction) under the assumption the code in question must be dead. This has proven to be problematic for some C/C++ code bases which expect to be able to cast away ‘const’. This is (and has always been) undefined behavior, but up until now had not been actively utilized for optimization purposes in this exact way. For more information, please see: bug 42763 and post commit discussion.

This will break all sorts of code. Often we have things that are const for callers of an API but not necessarily for the implementation of an API. Other cases are where an API should have used const but didn't and then got partly constified, and there are places where you still need to cast away const-ness.

I think it will break a lot less code than you think.

The C standard has a subtlety with regards to const in that it allows you to (for example) cast away the const-ness of a pointer and use it to write to it, as long as the original memory was not const (i.e., it was gratuitously cast to const). That's used by a lot of code, and LLVM still respects that.

This optimization (AFAIK) only removes stores that are disallowed by the standard, which is when the memory itself you're trying to store to was actually declared const. This would most likely cause a crash anyway, the memory would in most cases be inside a read-only segment. The bug report mentioned is from code inside a kernel, not from a normal executable that would be loaded with proper memory protection (like making read-only sections actually read-only), which is probably why the code worked without the optimization.

See the example in[1]:

- The function g casts away the const to write to the pointed memory, the generated assembly does what you'd expect (it contains a store).

- The function f tries to write to memory the compiler knows is actually const, clang optimizes the store away. Trying to actually write there would crash if the program was loaded in a normal OS, since the memory is inside a read-only segment.

[1] https://godbolt.org/z/as3Mh5

You can have const variables on the stack. Those will not have hardware write protection but it's still undefined to write to them.

I don't see this change breaking the examples you have given. You can always modify non-const objects. What you can't do is modify const objects through a pointer to non-const.

e.g.

    ((char*)"fooo")[0] = 1

is broken code but

    int x = 0;
    int const* y = &x;
    *((int*)y) = 1;

is not.

That's not what I got from the quoted paragraph. I'll have to chase the links to be sure.

In LLVM terms, it's referring to these kinds of stores:

   @vval = constant double 1.0, align 8
   define void @foo(double *val) {
     store double %val, double @cval, align 8
   |

At the LLVM level, the "const" qualifier to pointers and other non-global values get reflected nowhere in the type system. But if you declare a const global variable, that variable gets placed in a read-only data section.

Got it. Thanks!

So I've been bitten by this twice already, including just today.

1. This was the last bug preventing us from booting a MIPS Linux kernel. 2. An out of tree MediaTek display driver in Android was causing a kernel panic.

To be clear, this is explicitly undefined behavior in C. But this optimization to me feels like semantics from C bleeding into LLVM IR a little.

It's hardly C semantics that const memory should not ever be written to, otherwise there would be no need for the distinction at the LLVM ir level.

Optimizing away even more calls of memset, memcpy or memmove is the completely wrong way. gcc went down this bug and called it "optimization", now llvm did the same. Can we at least keep at least one sane working compiler? At least emit a prominent warning when "optimizing" away lib calls. This helps only artificial benchmarks and will hurt everybody.

If an assignment to const memory is illegal, treat it as error, and don't silently remove it.

Fear we'll have to blacklist clang-9 also, not just gcc-9.

How about you fix your code to not rely on undefined behavior specific to a compiler version? Poor coding standards is not an excuse.

> This helps only artificial benchmarks and will hurt everybody.

Any source of this claim? As I understand, on SPEC 2006 lib call optimizations make 5% improvement with GCC, the least artificial benchmark among SPEC 2006.

I believe rurban might have some point - for example, consider this CCC talk:

https://www.youtube.com/watch?v=0WzjAKABSDk

The point is that memset is often critical for security and compilers don't always do a good job of optimizing it out. If they strike a false positive, information leak vulnerabilities are very likely. It's also pretty difficult to detect those failures without recompiling entire program with a modified compiler, as shown in the talk.

I guess the question is: will it silently optimize them away or will it be a compile error to use such expressions?

It can't be a compile error. It's not an error for a program to contain such code, it's only an error if that code executes.

That's not strictly true. Undefined behavior includes unsuccessful compilation as well [0]. But for this the compiler has to prove that the offending code is executed every time the program is executed. I think compilers don't bother with this though.

[0] https://eel.is/c++draft/intro.defs#defns.undefined-sentence-...

Repo of the Linux kernel built with clang:

https://github.com/ClangBuiltLinux/linux

"This branch is even with torvalds:master." Does that mean there are no patches necessary for any GCC builtins? Or does the actual clang-buildable part live on another branch?

With LLVM 9, it is now possible to build the Linux kernel with no patches (for x86_64).

https://www.phoronix.com/scan.php?page=article&item=clang-li...

Note: we also build+boot under CI:

* arm (v5, v6, v7)

* arm64 (v8)

* ppc32

* ppc64

* x86_64

* mips

https://travis-ci.com/ClangBuiltLinux/continuous-integration...

No out of tree patches; I just use that repo for github's issue tracker. Previously the LLVMLinux team had an out of tree patchset that grew too large to be upstreamed, IMO. We picked up their work, got it upstreamed, and have been carrying the torch ever since.

Yes, there are still some stability issues though. Phoronix benchmarked the Linux kernel built with Clang/LLVM against GCC recently: https://www.phoronix.com/scan.php?page=article&item=clang-li...

Android and CrOS have not had stability issues related to the compiler since switching to Clang.

And Phoronix didn't send me any reports; I reached out to Michael and he said he'll send them to me next time he tries.

I was referring to the problems in said benchmark test on Phoronix, where Michael stated:

'The main culprits were Blender and GeekBench 5 causing the Clang-built kernel to cause the system to reset during testing while this behavior wasn't seen when the kernel was built under GCC, so likely some problematic code generation on the Clang side.'

Sometimes its a compiler code gen related bug, sometimes its bad source code. Without a dmesg+vmlinux, it's just anecdata.

And our site: https://clangbuiltlinux.github.io/

Come check out our once-every-two-weeks public meeting!

Meanwhile, Mesa developers merged ACO compiler for AMDGPU, as the future replacement for llvm.

Fun fact; AMDGPU driver can't run when built with LLVM right now because they:

1. build their driver with a stack alignment of 16B (standard for x86_64 userspace).

2. use floating point in the kernel, which the kernel has limited support for.

3. The rest of the x86_64 kernel uses 8B stack alignment.

Code from 3 calls into 1. And then they have sse2 instructions in their driver (double precision arithmetic). Guess what happens?

This has been patched and will be fixed in 5.4 if I recall correctly, right?

Heh, no, sorry, my fault. I wrote the patch you're referring to; I got it to link. When AMD went to test it on hardware (which I don't have), they hit alignment related general protection faults that I traced back to a movaps instruction. I'm still working with CrOS and AMD folks to figure out what to do, but there's a few questionable things that are going on in their driver that have a few kernel devs confused. For now, people are just happy to see AMD participate more in upstream kernel dev, but definitely some parts of their driver will need to be rewritten I fear at this point.

Fun fact - they do work fine... under FreeBSD :-)

Why does it work with gcc?

GCC just so happens to not select any instructions with alignment requirements at `-msse -mno-sse2`, while clang emits calls to soft-fp functions that aren't implemented in the kernel. At -msse2, clang emits sse2 instructions with alignment requirements. You'd think their 16B-aligned-stack driver would be ok, but its caller (the rest of the kernel) has a different ABI (8B-aligned-stack) which is kind of insane to mix TUs with two different ABIs in the same executable and make calls between them.

Then again, all x86_64 processors support sse2, so I'm not surprised that Clang at -mno-sse2 doesn't work better. There's numerous bugs filed against Clang where it crashes when -mno-sse2 is set, but double precision floating point is used.

(I've spent almost all of this week debugging this, and working with CrOS and AMD engineers on this. It's not clear to me at this point whether the 16B-stack-alignment ABI change was well intentioned or a stroke of luck).

> The optimizer will now convert calls to memcmp into a calls to bcmp in some circumstances.

What is the improvement from that change? I thought they were essentially the same.

memcmp has to find the first offset where the buffers differ, and check which of them is larger.

bcmp only has to figure out whether the buffers differ. It doesn’t have to figure out where they differ.

For example, for s=1024, bcmp can do (at most) 128 64-bit compares (using vector registers, it could even use larger steps)

memcmp could do the same, with an additional “figure out where in the last 8-byte parts compared the difference lies”.

That’s an amount of work that’s independent of the buffer sizes, so it wouldn’t add much, relatively, _if_ the buffers compared are large and the difference (if any) most of the time is near the end of the buffers. I doubt those ifs often hold, though.

This would be true, except bcmp has been removed from posix some years ago. There's no particular requirement that a function called bcmp does what you expect.

http://releases.llvm.org/9.0.0/docs/ReleaseNotes.html#notewo...:

”The optimizer will now convert calls to memcmp into a calls to bcmp in some circumstances. Users who are building freestanding code (not depending on the platform’s libc) without specifying -ffreestanding may need to either pass -fno-builtin-bcmp, or provide a bcmp* function.”*

So, he’s, you may have to provide it yourself. I expect you can fairly easily copy-paste it from various BSD-licensed libraries, though (example: https://github.com/freebsd/freebsd/blob/master/sys/libkern/b..., but be warned about that “I don't believe this is a problem since AFAIK, objects are not protected at smaller than longword boundaries”. That likely is, but may not be true on your platform)

So the problem here is I may already have a function in my program called bcmp, which takes no arguments and prints "better consider more possibilities" to standard output. This is all fine and legal. My program also uses memcmp, because of course it does. But now calls to memcmp are going to be replaced to with calls to bcmp, which does not have the same effect at all.

That’s nothing new, though. If you have a memcmp that isn’t memcmp (and don’t explicitly call a lib-provided memcmp anywhere) and build without libc, your memcmp will incorrectly be used, too.

MSVC is even more notorious for this kind of issue, silently adding a number of function calls it assumes the standard library will provide even when your code doesn’t use the standard library at all.

The compiler will surely use a (most likely intrinsic) function with appropriate semantics in that case.

The release notes go on to say that you may need to provide your own implementation of bcmp, which does not sound very intrinsic.

Huh, interesting. I was assuming that the compiler would statically link in a bcmp in that case, but it looks like it will literally replace it with a call to bcmp in certain cases (as defined by the remainder of that point in the release notes). Apparently this was brought up and brushed aside as not an issue: https://bugs.llvm.org/show_bug.cgi?id=41035

You may if you’re building freestanding code (code that doesn’t use the standard library), but do not specify that you are. I think that’s reasonable.

I do wonder why this is a LLVM feature, though, and not a clang one. The system should only make this change if it came from a C(++) compiler and the source included the system header (<string.h> or <cstring>) to get memcmp.

How can LLVM know that the symbol memcmp it sees came from that header, and not from user C code that may do something different, or even from, say, a Cobol compiler?

IIRC, Google engineers used fleetwide profiling to see that many cycles were being spent in memcmp() == 0 calls. They weren't checking the return value was anything other than zero. If you don't care where two buffers mismatched, you can speed thing up by converting that idiom into a call to bcmp during optimization.

memcmp returns the lexicographic ordering of the parameters (<, = or >) while bcmp only returns equality (anything nonzero is difference so memcmp is a suitable bcmp), so the LLVM folks believe worst case scenario bcmp will alias to memcmp, and possibly it'll have better performances since it has to do less: https://reviews.llvm.org/D56593

bcmp just tells you "equal" or "not equal", while memcmp tells you "lesser", "equal", or "greater".

I don't really know, but I suppose it might be possible to make bcmp faster since you have less strict requirements on the return value.

memcmp promises a nonzero return value tells you something about the lexicographic ordering. bcmp does not: so if you only care if it's exactly equal or not, it might be faster.

Hey Andy, will this mean I can try (experimental) zig on risc-v in the near future? :))

llvm9 branch will be merged into master today (as soon as the CI infrastructure is updated with LLVM 9) and it has basic risc-v support in the std lib, as well as cross compilation targeting risc-v using glibc and musl.

edit: [me updating the CI infrastructure](https://twitter.com/andy_kelley/status/1174769700343074818)

> Support for asm goto, enabling for example the mainline Linux kernel for x86_64 to build with Clang

This is an awesome step!

That was fast, has the release cycle changed?

LLVM changed their versioning scheme a few releases back (maybe around 3.9 -> 4.0 IIRC?), so major version increases will be more common now. I think it used to be roughly tied to bitcode compatibility?

Someone who works on LLVM can give a more specific answer, but the gist of it is we'll see more major version increments now.

http://blog.llvm.org/2016/12/llvms-new-versioning-scheme.htm... explains the exact change.

Ahh thanks!

No, it's been 6 months

Good to see LLVM IR becoming the ultimate bridge between Software and Hardware

Reply on news.ycombinator.com