If any Ubuntu maintainers are listening: PLEASE STOP DOING THIS | Hacker News Reader

If any Ubuntu maintainers are listening: PLEASE STOP DOING THIS

github.com

228 pointszdw60 comments

(Kernel person here. I have nothing to do with Ubuntu or gcc, but I can maybe help fix the problem.) It would be great if this came with an explanation of what broke. CET should not break setjmp.

I'd imagine the problem is that iPXE is preboot firmware, so the emitted setjmp/longjmp builtins assume a shadow stack that's set up, etc.

Ubuntu should not enable the flag when -ffreestanding is set.

I'm not sure what causes gcc to emit the builtin setjmp/longjmp, but I tried compiling this:

    #include <setjmp.h>
    
    void __attribute__((noinline)) test(jmp_buf buf)
    {
     __builtin_longjmp(buf, 1);
    }
    
    int main()
    {
     jmp_buf buf;
     if (__builtin_setjmp(buf) == 0)
      test(buf);
     return 0;
    }

and ran it on a non-CET machine. It works fine. It does contain a CET fixup, but that CET fixup is smart enough to work regardless of whether CET is enabled.

However, gcc does generate non-working code if the setjmp and the longjmp are in different files and only one is compiled with -fcf-protection=full. Ouch. I'll file a bug.

edit: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98997

The issue isn't a non-CET machine. The CET becomes NOPs on non-CET machines.

The issue is expecting it to work in a freestanding environment.

It ought to work in a freestanding environment. If it doesn’t, then it’s IMO a gcc bug. I found a GCC bug which may or may not be the ipxe issue. Are you seeing another failure in a freestanding environment? If so, I don’t suppose you either have a reproducer or can point at a problem in the generated code?

I don't know any specifics. It was my understanding, though, that you needed to be careful mixing CET-aware code and non-CET aware code. Is this wrong?

Can you call into the BIOS, EFI drivers, and have interrupt handlers and not worry about any of this at all? It really should be OK freestanding without any additional CET-specific initialization code support and mixed with non-CET code?

If so, I stand corrected.

(But then, what's all this machinery for marking code that doesn't use endbranch in glibc, ld.so, the kernel, etc, and the legacy code page bitmap for?)

I would say that the issue isn't really mixing CET-aware and non-CET-aware code. The issue is using non-CET-aware code with CET enabled. If you have code that plays games with mismatched CALL and RET or with rewriting the stack (which includes retpolines and possibly 32-bit PIC code), then SHSTK will make it crash. And if you have code that takes pointers to non-IBT functions, then you will crash unless you set up the legacy bitmap. But all of this is moot unless something turns CET on.

The actual semantics of when the combination of gcc, ld, and glibc will try to turn CET on are not documented, as far as I can tell.

And, to be fair, as my bug report shows, there can also be issues if something has a different ABI when compiled with and without CET enabled. This shouldn't happen, but apparently it did.

> The actual semantics of when the combination of gcc, ld, and glibc will try to turn CET on are not documented, as far as I can tell.

Yes. And a complicating thing is here there's no loader or normal initialization code, because PXE is freestanding and invoked directly by the BIOS/EFI. CET is -probably- off...

And yes, are there any dependencies between the init code changes and GCC enabling CET and GCC builtins?

Any CET init code ought to be confined to glibc and/or the kernel or whatever code is untimely loading your program. I hope glibc isn’t involved in iPXE :). CET had better be off for UEFI programs when they’re loaded unless that program has a special header, based on spec that I doubt exists, telling the loader to enable CET.

If anyone like me is wondering what -fcf-protection=full means: https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.h...

Enable code instrumentation of control-flow transfers to increase program security by checking that target addresses of control-flow transfer instructions (such as indirect function call, function return, indirect jump) are valid. This prevents diverting the flow of control to an unexpected target. This is intended to protect against such threats as Return-oriented Programming (ROP), and similarly call/jmp-oriented programming (COP/JOP).

The introduction of https://software.intel.com/sites/default/files/managed/4d/2a... has as much detail as I could ask for. Seems the GCC flag only enables the Intel implementation for now

Do you know what this flag considers to be a "valid" control flow transfer? The documentation makes it sound that this feature can safely be enabled all the time. But the linked article suggests otherwise...

It's safe almost always, unless A) you're doing something really undefined-behavior-ish and strange... or, here, for iPXE, I'm guessing the problem is B) you're running code that it emits assuming it's running on the Linux kernel in a fully initialized environment in a preboot/embedded environment.. and still expect setjmp/longjmp to work.

That is, it doesn't make sense with -ffreestanding, but Ubuntu enables it in that case.

The GCC docs seem basically useless. I have no idea if GCC plans to keep it like this forever but, right now, this is about x86's CET features. CET is an Intel spec that adds two very loosely related features: Indirect Branch Tracking (IBT) and Shadow Stacks (SHSTK). Neither is supported yet by upstream Linux.

IBT is a rather weak indirect-branch-and-call protection scheme. IMO it's weak enough that it has dubious value, and software-only schemes can be a lot stronger. The main benefits of IBT are that it can guarantee that a program won't, even when attacked, jump to the middle of an instruction and that, since IBT is a hardware mitigation, it can provide a degree of protection against Spectre-like attacks. Specifically, the CPU won't speculatively branch to an address that isn't an indirect branch target.

SHSTK is a very strong return address protection. With SHSTK on, it's really quite difficult to convince the CPU to let RET branch to anywhere other than were the original CALL came from. This even integrates with the page tables -- the memory it uses cannot be written by normal stores. The latter part is a large part of why this isn't upstream yet: the x86 pagetable format is already a technically-indebted mess, and SHSTK makes it worse in ways that are quite nasty. It'll get sorted out eventually.

You can find SHSTK in Zen 3 (yes, AMD brought Intel's feature to market before Intel -- go AMD!) and, apparently, in one Tiger Lake part. The latter also supports IBT.

You can play with the GCC support by compiling a trivial program like this:

    void test(void (*fn)())
    {
     fn();
    }

(no Godbolt link because I didn't figure out how to extract the right output from Godbolt.) If you compile with -S -fcf-protection=branch, the indirect jump is unchanged but the function itself gets an endbr64 (or endbr32) instruction at the beginning. That's a new special NOP that means "this is a valid indirect branch target". The CPU will only allow indirect branches that land on an endbr64 instruction, so the actual indirect jump in this function is unchanged. I think GCC is clever enough to omit the endbr64 if it can prove that the address of the function is not taken.

The other interesting thing that GCC does is to emit this:

            .section        .note.gnu.property,"a"
            .align 8
            .long    1f - 0f
            .long    4f - 1f
            .long    5
    0:
            .string  "GNU"
    1:
            .align 8
            .long    0xc0000002
            .long    3f - 2f
    2:
            .long    0x3
    3:
            .align 8

The 0x3 changes depending on the -fcf-protection option. This is a magic incantation for the benefit of the linker and kernel declaring whether this object supports IBT and/or SHSTK. This way you don't crash if you link a non-endbr-instrumented file into a IBT-using program.

Note that SHSTK support doesn't affect the code at all. The CALL and RET magic is handled by the CPU, and library functions like setjmp use special new instructions in order to keep working.

There is also CPU support for CET and IBT in the kernel. SHSTK in the kernel is a huge mess. Specifically, the x86 architecture is starting to fall apart -- there are various ill-conceived features like SYSCALL, NMI, the machine check architecture, and some fancy virtualization features that, collectively, conspire to make it extremely complicated to correctly handle all types of interrupts and interrupt-like events. SHSTK-in-kernel makes it worse, and I don't think anyone wants to try to deal with this yet. I and others have been badgering Intel and AMD for quite some time to do something about the mess that x86 kernel entries have become, and so far there are no results.

Thank you for taking the time to write this very in-depth comment, I can say I understand some of these words now!

Ubuntu's gcc also randomly makes me need to add -Wl,--no-as-needed to my flags. Probably because I'm doing something slightly wrong, but whatever it is I'm doing works fine on Fedora/EL.

How can anyone possibly think this is a good idea? This is such a bad decision that it almost disappoints me more than Ubuntu's relentless push of the badly broken and poorly designed Snap ecosystem.

Why should it not? For the most part this flag seems to be working. And it improves security. Windows can't do something like that for most of the software targeting it because it's all in binary form. For the software available through the default Ubuntu repos, source is available, so Ubuntu can offer this.

"If any Ubuntu maintainers are listening"....

You'd be lucky, in my experience. They Know Better

Do you have a case in point?

I've seen plenty of things were they couldn't meet the demand, but were they were simply taking the high ground?

I cannot give you details, like dates or version numbers, but I can give you a anecdote about when I lost the trust.

When they introduced Unity (? Their version of Gnome Shell, I think that is what it was called). I found unity after I did a `full-upgrade`. Suddenly my user interface was something that I could not understand, had no idea of how to use, and did not work very well. There was no path back to the user interface I knew well. I had to stop everything I was doing with the computer, way beyond what I had allocated for the upgrade.

I grizzled on our local Linux users email group (I vented I think) which was interesting because two people who worked for Canonical (Ubuntu's maker) and took it personally that I would call their employer a whole lot of bad things, for giving me "free software". I was furious.

Turns out I found later that Unity was not even ready for release and had been rushed out (I think I remember the reasons, but not sure enough to relate them).

All they needed to do was do a parallel release then I could have learnt it, evaluated it, they could have debugged it.

I went back to Ubuntu when they adopted the Gnome Shell - it is a good interface, I am convinced now.

But wind on several years and software I needed to have, but not use often, had to be installed in snaps was the final straw. Not just the waste (I want my laptop to be a integrated system, not a mess of different walled off packages) but the fact that I would no longer be in control of the updates. After the debacle of having a very buggy user interface foisted on me I am not ready to give them such control.

I understand what snaps are good for, but I am headed in a different direction.

It's a funny anecdote: at the time half-baked Unity was rolled out, Gnome Shell was introduced as well, and it was worse (no really: even if you liked the UX design, it was so buggy to be really unusable; Unity was simply a plugin for Compiz trying to turn off all features that were buggy). And that's coming from someone with "emeritus" @gnome.org email address.

So basically you were going to get an entirely unfamiliar interface anyway. You could get unpolished Red Hat contraption of Gnome Shell, or unpolished Canonical contraption of Unity.

If anything, Unity had more resemblance to the interface of old (eg you had static virtual desktops, which you only kinda have even today with Gnome), and if you ever used a spatial grid of workspaces (to leverage your human spatial memory: I used 3x3 grid with Emacs in the centre one and task-based workspaces around it), you were out of luck with Gnome 3.

I think Gnome 3 has become significantly more robust even before Unity's demise, but UX design-wise, Unity is much better even today after being unmaintained for years (though buggy as hell).

Interesting. At the time I looked at Gnome Shell. I do not recall the details but I retreated to Mate (? was that the name of the Linux Mint desktop?) that had the old style.

I thought I decided I wanted the comfort of the familiar. Over time it won me over. Which was my point.

Canonical at the time were so arrogant that they did not try to win me over, they dumped me in it

Exactly, you retreated to Mate that shipped with Gnome 2 because Gnome 3 sucked too :)

However, Gnome 2 was all but abandoned at the time, and it did not make sense for Canonical to commit to supporting it for the next 5 years with their LTS release without the help of the upstream.

All the big distributions were "forced" to switch to an unfamiliar interface. Fedora did too with Gnome Shell. Ubuntu at least had derivatives like Mate or Kubuntu that you could easily move to with an apt-get install.

But they could have had a transition.

They did not.

Users == Lusers thinking.

You seem to overestimate the resources Canonical had on Ubuntu, and especially on Ubuntu Desktop.

But you seem to have been already decided that they did it because they are mean, so I don't think it makes sense to continue the discussion.

I get that breaking packages is a bad user experience, but the Linux community at large should start paying more notice to security-first software engineering. Learn a little bit from Microsoft's experience - they made 100% backward compatibility their mantra for years, and look where it got them - a reputation as an unsecure platform with lots of exploits. Linux has many exploits too, its just a less popular platform for research by security researchers so less security bugs are found then Windows. But when those are found - its catastrophic like the last SUDO bug. They way to forward is to understand that even great software developers have bugs with security implementation, as writing reasoning about the correctness of software is very hard, and start implementing wide reaching mitigations by default. Control flow mitigations made implementing exploits on Windows much much harder, raising the bar of "bug to real-world exploit" significantly. Ubuntu setting CF protections to default will encourage many maintainers to finally fix their code.

It also gained them a reputation as literally the only operating system that cares one whit for backwards compatibility, which is one of the reasons it has such high penetration, especially in enterprise. Linux explicitly has no kernel ABI guarantees, and Apple gleefully breaks shit between minor releases.

For what its worth, they do seem to patch the documentation as well:

https://salsa.debian.org/toolchain-team/gcc/-/commit/2f9aefe...

The documentation about this flag is on this Ubuntu wiki page:

https://wiki.ubuntu.com/ToolChain/CompilerFlags

Ugh. I rebooted my machine last week and VirtualBox and CUDA broke because Ubuntu upgraded the kernel. When can we have a system that updates itself without breaking stuff?

>When can we have a system that updates itself without breaking stuff?

We did, it was called CentOS.

Anecdotal but we left centos for debian because it kept failing after kernel upgrades (machine would not come back / kernel hang: we had it so many times that we migrated completely off it): never had that issue with debian (or ubuntu) and I use both fulltime on servers and desktops (laptops). We had other issues with broken deps with yum as well.

I think GPUs might be a different story but we have not needed those beyond the trivial usage, and those were not discrete.

Were you using funny third party repositories?

Nope, then it would be logical. Especially on servers it was all standard stuff.

You probably installed those things with some third-party installer, instead of through the package manager. You can't expect Ubuntu to know about that. The only way it could not break those things is by never upgrading the kernel.

I don't think that should be an excuse for things not working, especially when the third party installer is NVIDIA's official installer.

Macs and Windows don't break when you install NVIDIA official drivers. Ubuntu breaks because nvidia cuda-10.0 installs nvidia-450 which fights with Ubuntu which wants to install nvidia-460 with the latest kernel, which also breaks virtualbox's official repo IN the Ubuntu repo. So either the kernel breaks or cuda and virtualbox breaks.

In any case, I as a user shouldn't have to worry about stupid things like this.

Why not put the "THIS" in the title?

What’s the best alternative to Ubuntu? Debian?

Depends what you're looking for. If you're used to Ubuntu you'll find Debian to be similar: just a less-polished, less-branded UI and maybe slightly older (perhaps more stable) packages.

You might also like Fedora as a pretty mainstream, up to date desktop distro, but under the hood you'll find more differences like rpm instead of apt.

edit: I'd add that Linux distros are complicated: the issue in this article is hardly a reason to leave Ubuntu if it's otherwise working well for you.

Don't use Debian stable for desktop. Debian testing or unstable is a much better option.

I have found the one big feature of Ubuntu that I miss in Debian is PPAs. Other than that it's largely the same. Configuring the system requires more file editing and command line work than I remember with Ubuntu, but the documentation helps quite a bit.

EDIT: By default, non-free drivers are placed in a separate repository, and receive less support. In my case the biggest pain has been Nvidia drivers. I needed a cheap card to drive an HDMI display and all they had at the store was a GT710. That meant installing the proprietary drivers, since the free alternative had poor performance last time I checked. Every few updates it breaks, and I need to reinstall it and sign the kernel module each time. That is something to watch out for if you switch over.

Yes, no PPAs but enough third party repos available.

On the matter of PPAs, these days I find that most of the things I used to use third party repos for I can now get as flatpaks from flathub. Really helped cut down on how many third party repos I'm using.

I'm not sure why this got downvoted. It is a very high-signal comment.

Debian is usually a great and pretty stable ecosystem choice. Also APT makes it easy to manage third party repositories (package pinning).

One big advantage I see with other distributions is that the release support cycles are longer (5y+).

But it will greatly depend on your usage, we are pretty happy with Debian running on somewhat 300+ machines.

OpenSUSE is the one I landed on, and it's been great.

I keep hearing high praise about Manjaro too, that it's solid, fast, and newbie-friendly; but I haven't tried it out myself yet.

Manjaro is too bleeding edge for most people. I don't think it's a good idea unless you are quite experienced with linux and like fixing things. It's great IF you need bleeding edge though, and seems okay enough for a daily driver if all you do is browsing, zoom, and email.

Oh yes! OpenSUSE Tumbleweed has been really excellent. It’s very polished and quite an easy thing to set up.

FreeBSD.

There is a mess of things wrong with the linux ecosystem and a lot of it comes down to packaging systems and package management. Once you get over a little learning curve, working with FreeBSD is a breath of fresh air.

It’s a little hard to believe that any BSD is a drop in replacement for most use cases.

For sure there are some narrow cases where it is, but Linux just supports sooo much more

It might strain your imagination, but go look at everything available in ports, and also the linux compatibility layer.

I wouldn't call it a universal drop-in replacement, you can indeed do a whole lot with FreeBSD that you can do with a linux box.

Depends on your needs. Is it desktop? Is it server? Is it container base? If there any specific software/hardware you rely on? Etc. There's no "best alternative".

I used Debian for years, it is stable and has very few surprises. Ubuntu is a clone of Debian and can be a bit buggy. That been said Ubuntu is great for trying new things. Probably my favourite distro of late is Fedora, works well, the s/w packaging system is stable and quite user friendly. My advice is try them all. Oh and FreeBSD.

openSUSE

For desktop use, check out Mint.

I imagine why would Ubuntu developers do it like that: to 'fix' the 'broken' build systems which do not respect CFLAGS.

How many build systems are that broken, and unpatchably so?

None.

But for the lazy package manager, this route is easier.

They should do their job regardless, but I’d place all bets on CMakeLists.txt. With two decades of experience, only this one I can read and get fucking nothing of it.

Reply on news.ycombinator.com