The Insecurity of Debian
unix.foo
unix.foo
SELinux is complex, badly documented, policy code is obscure macro incantation, and basic debugging tools often aren't installed out of the box on server distros (such as audit2allow). But for the day to day administration of systems, policies are included for distribution packages and most issues can be fixed by enabling a boolean here and there, and relabeling files.
The principles, basic admin & debugging part can be learned in a couple of hours, and when you have custom service software, you can throw it in /opt and have it run unconfined (ie: not subject to SELinux rules).
The underlying concepts of SELinux aren't so hard but trying to manage it in any sort of coherent way is a nightmare - up to and including the provisions in it for a network based policy server component which just never appeared.
And it sucks! In theory it does so many things we really really want, and should do more. Like I as a user have a great interest in ensuring my home directory files follow sensible markings based on their content - my SSH keys, AWS keys, or banking files all exist in different logical zones of control.
And this is a concept SELinux can handle...but the tools are just so bad at surfacing it.
In principle, yes. In reality I've gone looking for a resource that I could do this with and come up short.
(I am starting to get really annoyed at things where I can find a million "paste this bit to do that thing" and there are so darned many of those on the internet that any hope of finding a good resource that gets to the underlying structure such that I can figure out how to do these things myself is virtually nil. It seems like this is getting worse as the search engines continue their trend of taking your query as a vague suggestion of the sort of thing you're looking for.)
It won't make you an expert but it takes the voodoo out of the whole process, if that makes sense. And it is reasonably short if you skip the stuff that you're (probably) never going to configure like passing labelled traffic between hosts with IPSEC.
Then again, Disabling SELinux is necessary. For example, cPanel requires disabling SELinux on CentOS, AlmaLinux OS, CloudLinux, and Rocky Linux. AppArmor is fine on Ubuntu (https://docs.cpanel.net/installation-guide/system-requiremen...).
My servers all run with selinux, it's really trivial. Just the ssh client and tailscale recipes are missing by default. Selinux gives you precise choices if something is rejected.
Then in practice someone ends up writing a couple policy statements and filing a couple forms then disabling it anyway, nearly every time.
If that’s the case it doesn’t need to actually work in practice, just hypothetically.
It's not the only project like it, it's the one that is most well known because it has the NSA attached and because it got incorporated into the main kernel.
It works in practice, absolutely, but most people are too intimidated or lazy to put in the effort to learn it.
I couldn't find it in the Debian spec (probably because it uses AppArmor), but the RHEL benchmark has these.
Currently, server level 1 only requires permissive mode:
https://www.tenable.com/audits/items/CIS_Red_Hat_Enterprise_...
CIS Red Hat Enterprise Linux 9 v2.0.0 L1 Server — 1.3.1.4 Ensure the SELinux mode is not disabled
https://www.tenable.com/audits/items/CIS_Red_Hat_Enterprise_...
CIS Red Hat Enterprise Linux 9 v2.0.0 L2 Server — 1.3.1.5 Ensure the SELinux mode is enforcingIs this poor design or simply, not designed _for_you_?
I agree, its a royal pain to manage, and it might be overkill for a small shop trying to lock down their web server. Thankfully there are other solutions, and operating systems that may better fit your use cases.
https://en.wikipedia.org/wiki/Common_Criteria
I find it enlightening to read what kinds of justifications the proponents of SElinux use. It's never about the quality of the software; it's about how there's more band-aid tooling to make it easier to work with, or about how it's not as bad as it was, or that it gives you all these knobs and levers to have more control. It's not what you focus on when you're serious about quality software engineering.
Imagine if we were talking about something like Gnome or the Windows 11 interface: yeah, the interface is a real pain to navigate, but we added even more menus and buttons and the rightclick menu is twice as long now, so you can do even more stuff with it, and we even added Clippy back in to help you when you get stuck!
SELinux is designed to fulfill to primary goals. First, to secure the messy and complicated Linux architecture. And second, to be flexible enough to accommodate (highly) complex security architectures, as well as potentially unique and/or unforeseen needs. With that in mind, it's difficult to imagine any equivalent being practically more simple and/or elegant than SELinux.
The primary problem with SELinux is the broad lack of experience amongst users and sysadmins, opaque documentation, and primitive tooling. And in many ways, it is a negative feedback loop. If SELinux was used everywhere, improvements to its documentation and tooling would naturally follow.
You're describing the linux architecture as messy and complicated, but that describes the SELinux architecture as well; if complexity & mess are bugs that should be squashed in pursuit of security, SELinux is ill-suited to the task.
> And second, to be flexible enough to accommodate (highly) complex security architectures, as well as potentially unique and/or unforeseen needs.
>> It's technically capable of any kind of restriction a bureaucrat might envision
Sounds like we're on the same page there. Or at least looking at the same phenomenon.
Your last paragraph is definitely outside the pattern of justifications I listed, but it's not much better: you're just blaming the users. Sysadmins use all kinds of complex software to accomplish any number of delicate tasks - if the tool is well-built, they don't tend to complain that it isn't. SELinux is not. Don't blame the user when the tool's at fault.
Hypervisors predate Unix, in fact they practically predate general purpose operating systems as a whole. The reason hypervisors came first is because they are substantially more simple than an OS. Tens of billions of dollars has been spent on virtualization technologies because of its reliability.
Sure, a virtual machine running on a type-2 hypervisor like KVM looks like a complete mess. But the sad part is that such an architecture is easier to secure than an operating system, whether OpenBSD or otherwise. Raadt may disagree, but AWS, Azure, GCP, etc depend on hypervisors/virtualization, not OpenBSD.
You're describing the linux architecture as messy and complicated, but that describes the SELinux architecture as well...
Well, yes. SELinux has to cope with the deficiencies of Linux. I'm not pretending otherwise.
...if complexity & mess are bugs that should be squashed in pursuit of security, SELinux is ill-suited to the task.
The problem is that the market settled on Linux, despite its "complexity & mess". SELinux isn't nice but it's the most concrete solution available today. Indeed, after twenty years of criticism, no one has been able to design a competent replacement or alternative.
>> It's technically capable of any kind of restriction a bureaucrat might envision Sounds like we're on the same page there. Or at least looking at the same phenomenon.
There is a material difference between our viewpoints. The government has systems running in a dizzying amount of configurations and environments. Moreover, government agencies and government contractors operate in a far more dangerous security environment than the vast majority private companies. Perhaps your workplace doesn't need SELinux, but companies like Lockheed Martin or Aerojet Rocketdyne definitely do.
...you're just blaming the users.
Maybe it seems like splitting hairs, but I'm not blaming anyone. Rather, I was lamenting over the bad situation of things. After all, I've been there as a new sysadmin. Trying to grok SELinux for the first time is not a pleasant experience.
Don't blame the user when the tool's at fault.
Look, if security isn't important for someone and/or their organization, then fine, don't bother. However, we have seen time and time again that compromises in supposedly "unimportant" systems ends up causing quite a bit of harm.
Ultimately SELinux exists because of the shortcomings of Linux itself. Nothing has replaced SELinux because genuinely securing a Linux system is extremely difficult and it fundamentally cannot be made simple.
Yes, money has been spent, and cloud infrastructure built, on hypervisors because of their reliability, and because they are selling virtual machines. But reliability, while paramount, is not security, and the goal is to sell VMs. OpenBSD, with its focus on security over performance, is the wrong tool for the job.
> coping with the deficiencies of linux
Good designs can take a messy domain and provide a clean interface on top of it. SELinux does not.
SELinux is partly a big matrix of tags, with definable security associations between any two such tags. That's great when a bureaucrat in a defense contractor security department writes up some new policy definition - you never know what they might come up with - and I would not make the mistake of assuming security is well-informed on the internals of Linux when they write those rules. SELinux is elegantly designed to be as granular as needed to accommodate that. But that's what it's designed for: checkbox security within massive agencies.
> it's all we have
Yes, I agree, it's the only thing that can accommodate that kind of security; since few people outside regulated industries are interested in catering to it, there's not a lot of push to make something else to fill that niche.
> the user or the tool
Arguing that security is important is not relevant to justifying SELinux. I will agree though, it's definitely very hard to twist a Linux system into a shape that fits bureaucratic security policies.
This is fundamentally not true. Don't buy into the aggressive marketing. OpenBSD has a less secure design than pretty much any modern Linux. Their reputation for security is based on disabling things by default when it wasn't common 20 years ago, that's pretty much it.
First of all "OpenBSD stands on the logic of its own merits" what in the actual heck?
OpenBSD has had two remote holes in it's default install and importantly, *no mechanisms in place or restrict what can then be done*. That's not in line with a secure system.
You're vastly overstating and assuming the merits OpenBSD has, and then even worse, assuming logic exists based on that to support your position. I would say it doesn't, and I challenge you to show your work and demonstrate otherwise.
SELinux is a mature product that has seen widespread use in enterprise deployment and has real world examples of stopping attacks that OpenBSD couldn't hope to on it's best day.
If you want to go by merit and logic and not assumption and marketing, then SELinux will come out on top every time. It's actual, provable, tested security, not dreams and half-measures.
If you are looking for a justification to excuse bad security practices, you won't find it in the origin story of SELinux.
AppArmor is simply less granular. For example, it doesn't provide true RBAC or MLS security models. It also uses paths instead of inodes, so a hard link can be used to override some policies.
So it just depends on what the exploit or attack is trying to do. If an attacker gets root and is trying to overwrite a file, they may be able to. Maybe they can't, but they could probably still execute any code they can write and compile themselves. And perhaps they can write to other files and do damage.
SELinux and similar systems allow a lot more granularity. Programs and users can only talk to explicit what they are allowed to talk to, and maybe you want to limit the access to say, append instead of full write access.
It just allows a lot more granularity and restriction, that's the difference.
The link rules can get pretty granular and seem explicitly designed to prevent that scenario.
https://man.archlinux.org/man/extra/apparmor/apparmor.d.5.en...
> they could probably still execute any code they can write and compile themselves
Assuming the AppArmor profile allows writing to and executing the same files. Which isn't particularly common.
> maybe you want to limit the access to say, append instead of full write access
This is possible with AppArmor.
It's still an inherent weakness. No getting around that really.
> Assuming the AppArmor profile allows writing to and executing the same files. Which isn't particularly common.
I don't really want to try and come up with examples just so you can show there might be some hacky way of accomplishing something similar to what SELinux can offer - it would be missing my point.
Point is there's a lot more you can do under AppArmor than SELinux. AppArmor isn't as granular and you can't lock down a system to the same extent, period. Is it good enough, sure. Is it better than nothing? Absolutely. Is it comparable to an optimized SELinux config? Not remotely.
> This is possible with AppArmor.
See above.
Yes.
> Literally every example you gave of AA not being "granular" enough was flat misinformation.
No, there was no misinformation, and this stance you're committed to defending is one of the most bizarre stances I've ever come across.
There can be no question that SELinux is significantly more granular than AppArmor any more than there is that the earth is not flat. Looking at the introductory documentation for both systems should be more than enough to make that abundantly clear to anyone.
> There are dedicated rules to prevent writing and executing the same file, prevent using hardlinks to gain privileges, and prevent overwriting a file that should be append only. No hacks here. Just facts.
So just before I put more effort into replying to you, I want to be 100% clear on your stance. If I am paraphrasing or misconstruing, please correct.
It seems like you are claiming that AppArmor using hardlinks is not any sort of vulnerability or weakness and cannot be, and has never been bypassed? Is this a fair reading of your position?
Can you offer some examples of things you can restrict with SELinux that you wouldn't be able to with AppArmor?
Only valid answer in the thread has been port bindings - AppArmor's network rules don't allow restricting port number, but SELinux can do that.
You tried to claim that SELinux could prevent processes from overwriting files instead of just appending to them while AppArmor could not do the same, but that statement of yours was easily disprovable -- the man page of apparmor.d shows that append-only rules are supported. If you don't want me to call your statement misinformation, then maybe invent another word because that is the only word I have to describe what you said.
I listed some of the ways AppArmor falls short, which you dismissed.
If I know an object is 3x3x3 feet, and we have a box that is 5x5x5, and another object that is 7x7x7, I don't need to thoroughly test every aspect of these items or even see them to know one of the won't fit in the box.
> Only valid answer in the thread has been port bindings
Not true. AppArmor lacks several of the models SELinux does, and thus, as has been said, is less granular, and thus, and as has also been said, it covers less area than SELinux does. AppArmor doesn't even consider user accounts as far as policy decisions go, and you can't bind policies to user objects. You realize already what a limitation that is, right?
It's sufficient to look at the designs of both systems to see this, see where one falls short, and not need practical examples to understand.
If you want practical, real world examples of SELinux blocking something AppArmor couldn't, as I said to someone else, a comparison of Debian and redHat security advisories should show this, as I would think it is extremely likely that Debian would significantly less often be able to say the issue isn't a threat if AppArmor is enabled vs RHAT saying the same for SELinux.
But, you want a setup. OK. Does AppArmor allow you to basically take root out of the equation entirely, by assigning only the capabilities a user needs to run specific programs (e.g. like binding a port under 1024) to a non root account? Does it then allow severely limiting the root account so it can't really doing anything and 'getting root' is pointless because you can eliminate the entire concept of an all powerful account? No, it doesn't, and there is plenty more it doesn't allow because it's a simpler and more limited system by design.
> You tried to claim that SELinux could prevent processes from overwriting files instead of just appending to them while AppArmor could not do the same, but that statement of yours was easily disprovable
You're right, this was my mistake. AppArmor either didn't have that functionally the last time I really played with it, or I forgot it had it. That's a bad example, sure, but the overall point is still perfectly valid.
> If you don't want me to call your statement misinformation, then maybe invent another word because that is the only word I have to describe what you said.
As far as AppArmor being able to enforce append only functionality, sure. As far as anything else, not so.
It's been a while and we hacked around it, don't remember how. Except that it was not the #1 solution, disable SELinux altogether.
Regarding how to do that, it's left as an exercise to the PhD holding student.
This is so funny because whenever I suggest Fedora Silverblue to a moderately experienced Linux user who wants a simple distro, the first thing I do is recommend turning SELinux on permissive mode, and I get a bunch of comments hand wringing about how you shouldn't do that.
It's almost like a silent filter working in the background of your OS that doesn't even tell you when it blocks something is a pretty user hostile feature and no one wants to learn how to speak SELinux so they can effectively use it.
Sometimes it seems like Linux people don't want others using it. Even when they belong to evangelist platforms, they like to create huge barriers for entry and then blame new users for not "getting it."
-- A developer whose app needs to run as root (for a well-documented reason, and with a tight systemd sandbox hiding most of the filesystem from it)
I used to work at a payment provider and we had to deal with lots of monitoring and security stuff. Some of it was (obviously) busywork and needless checkbox filing, but other parts were genuinely useful. Setting up systems was tedious and difficult, but ultimately worthwhile and necessary.
HP at one point said to "chmod -R 777 /sbin" in their ArcSight install documentation, and that's a 'security' product.
Few will instead read the RHEL provided documentation. Then they could maybe figure out whether there's simply a tunable (getsebool -a) which would enable the desired behavior, or if properly labeling files (semanage fcontext / restorecon) would do it, or even take the steps to add to an existing policy to allow for a specific scenario which somehow was not implemented. Even adding your own policies "from scratch" is certainly doable and provides a great safety net especially for networked applications.
Anyway... we all know disabling security or not implementing it in the first place can really save you a lot of time. At least in the short run.
If you want to dive deeper: "SELinux System Administration" by Sven Vermeulen.
The way I put it to my clients, and staff, is simply that security comes at the cost of convenience.
I'm hoping it sticks. Just check audit logs when you get an error, it is not that hard, right?
Sometimes, when the developers make a mistake, which is unavoidable in a large project, it is nice to be able lock down applications as the administrator.I just don't think SELinux is the right tool, because the chance of you making a mistake in the configuration is pretty high. The functionality is there, but it needs to be easier to write policies and maybe that comes at the cost of some flexibility.
On the other hand, the administrator knows their system better than the developer. There could be certain network connections or file paths that you want to block on one system but not on another.
Most of the places who used CentOS for a 'free' Linux ended up shutting it off, though.
One crucial change for the better was leaving third-party software in a permissive state. From that point onwards, disabling SELinux is cargo-cult sysadmin'ing.
SELinux is not hard if you understand its basic principles. But no one bothers, because SELinux is the bogeyman.
Yes, writing policies means getting knee-deep in macros, and it's hard because many services try to access anything and everything. But almost no one needs to write a policy.
At most you need to tell SELinux that some non-default directory should have some label. That's not hard.
In my experience, it's not just directory labels ("semanage fcontext -a -e ..." and friends). You also need once in a while to set some booleans ("semanage boolean ..."). Yes, it's not hard once you know about it.
But that's exactly what I would like to do! I've never seen a real guide for how to set up a policy for a custom daemon I wrote myself. Or when a specific software doesn't come with a policy.
Many years ago I decided to face the Bogeyman and went from knowing very little about SELinux to writing a policy from scratch in about a month. The policy is simple enough (but realistic) that it might help in the absence of a guide:
https://github.com/carlosefr/kyoto/tree/master/selinux
I used it as an example in a couple of talks, whose slides might also provide additional context:
https://github.com/carlosefr/public-talks/blob/master/presen...
Yes, people/sysadmins should take their time to properly configure SELinux when things don't work, instead of just disabling it completely for good. I tried for a whole year in a place where we used CentOS, and then finally I gave up, too many hours wasted in finding the right conf for this new program or configurations etc.
On OpenSUSE/MicroOS who is employing SELinux boot takes about 5 minutes on every kernel change, because of home-relabel. I hear you, that they probably do it wrong, but that's what you get with SELinux. Not enough to push me to disable SELinux, but maybe to avoid SELinux distributions in the future.
https://www.ctrl.blog/entry/selinux-unmanageable.html
I'm not going to waste my time fighting SELinux to stop non-existent threats (I'm just using a desktop and I'm not a high profile target). Too many false positives and I'll just turn it off. And in my experience there are always too many false positives.
It's painfully obvious to me that the people who create SELinux and its documentation live in some alternate universe where they don't do anything the way I do, so I just turn it off.
It won't carry you all the way to applying policies via Ansible or RPM packages, but definitely took me from running random audit2allow commands to taking a more holistic view of my SELinux policies.
It also looks like a long read but if you fast-forward through chapters that aren't relevant to you (looking at you IPSEC) it isn't such a slog.
Watching a 15 minute selinux tutorial video will give you a moat ahead of 90% of the community but it won't matter because management kind of agrees that anything that slows you down has to go, and security policy is ultimately just a type of insurance rather than a revenue generating activity. Disabling selinux reduces cost today so we might as well go for it.
I do think it is worth having on any public webserver since it's only a matter of time before your app gets popped and you want that sucker in jail, but I gave up on internal servers a long time ago.
What sort of disaster area do you work in where it's the developers job to tell security ops how firewalls work?
It usually goes like:
“What are the outgoing ports?” “1024-65535, I mean, the app is using X language’s standard library to make an HTTPS request.”
“What are the IPs we have to whitelist?” “You can either allow app.example.com or take AWS’s IP range JSON file and allow all of those, we don’t control what IP gets assigned to AWS’s API Gateway service”
Then some cloud provider’s SA/SE gets looped in to say the same stuff to the security team.
Some exec then gets escalated and approves this as a risk.
Tale as old as time.
- Network ports the service listens on - What security permissions does the application need - What commands have to be run so the application starts - What platform does the service use? Java, Node, C#, C/C++, Go, something else? - What GIT repository or repositories contains the service's code? - How does the build work? - What needs to deployed to the machine? - Any configuration changes the application needs
There are also a lot of join decisions where the operations engineer and the software engineers have to work together. Here are some examples: - What cloud will the team use? (AWS, Azure, GCE, etc.)? - What cloud technologies will the team use? - What database will the team use? - How will logs and alerts work? - How will the on-call rotation work?
My main point is you cannot just tell an operations person to deploy something and expect good results. They will have a lot of reasonable questions and software engineers should be able to answer them.
Is there any reason that you do not know what your application can be expected to connect to? You wrote the thing, right?
I work with a lot of external vendors who offer self-hosted software and a really common refrain is "it must be a network problem" but these guys are universally unable to describe their application's networking requirements.
> it will give you the commands to fix it
Usually it's a crap you should not apply to the system.
SEL seems to work under the premise that if it’s too complicated for you to use, the attacker has no chance.
A system which is easy to use securely will stay secure, a system which is difficult to use securely will be insecure.
And I don't agree with the article that containers do not add security. Container runtime implements namespace isolation, seccomp filters, etc. and that reduce the attack surface, comparing to running the software directly on the host OS. More importantly in this discussion, it is convenient for sysadmins.
There is no perfect security anyway. And I don't sacrifice convenience for national security level security :)
These days that's just laziness/fear, and it's a shame to see it. It doesn't even make sense any more.
We don't disable SELinux.
> The policy language and tooling is cumbersome, obtuse, and is about as appealing as filling out tax forms.
If a security framework is so terribly complex and hard to use, then people won’t use it.
OTOH, look at how OpenBSD or OpenSSH approach security: simply primitives which are well documented and easy to understand.
The only reason SELinux even works in the few scenarios where it does, is because the operator had immense amount of resources to pour into it. This itself is another sign of how bad the design is: it’s so complex that no small team of humans have ever been able to use it.
SELinux does better or worse, depending on perspective, by actually enforcing controls
Sidenote: I don't like the implication that community-driven projects are inherently less secure.
> Lack of Resources: Debian as a community-driven project lacks the resources to develop and maintain comprehensive security policies comparable to those provided by Red Hat.
I don't like it either, but it may be true anyway. Although I don't think it would be resources so much as focus. The Debian community is not that small.
RedHat can declare that everything on the system is going to have SELinux policies following consistent guidelines on what to lock down, and all employees will work with the security team to make this happen. That is harder to do in a community driven project like Debian where ownership and work is widely distributed and entirely voluntary. It can really only happen when the goals are already a strong part of the culture and there is buy-in for specific rules to achieve those goals. For example, Debian's strong free-software requirements have been there from the beginning and so most Debian volunteers are self-selected to agree with or at least tolerate them, and even that has frequent arguments. Security culture is much more mixed, and there are a lot of people in the free software community who think that security starts and ends with fixing bugs when they are found, and push back hard on suggestions that anything more is needed. It is going to take a long time to change that culture.
The former strongly implies that, if you're using it for the latter case, then you really better know what you're doing. But this capability/competence versus task-fit gets glossed over in the paragraph where the author basically says; because Redhat chose to be a bag of dicks, jumping ship to Debian is the "logical move". It isn't if you don't know what you are doing. And it's sad that RH exited this space leaving a civil cybersecurity hole. The lack of a truly Free and "OOB secure" OS seems the case in point.
There are other reasons to doubt the security of Debian, but "you're using it wrong" isn't the best one to discuss.
For shared server usage. Most servers are single-use, what makes SELinux mostly useless again.
And on those shared servers, you have to define your actual policies for it to be useful... What a total of 0 people do.
It's hard to completely dismiss the idea that SELinux was a NSA plot to keep userspace capabilities out of reach on consumer OSes.
Indeed, since the dawn of virtualization and automated deployment, shared servers are a legacy behavior. Well, on Debian's world, at least : for RHEL, you may pay per instance, so there is a financial incentive to share said instances.
Ergo, RHEL and friends are inherently less secure than Debian.
For several versions of the OS, this worked quite well, but once dual-sim devices2 started coming out, this became more problematic. Furthermore, when SELinux3 became common on Android, this became more problematic since the radio SELinux context that rild started with was too restrictive for the implant to function. - RoidRage Bootstrap Methods (https://wikileaks.org/ciav7p1/cms/page_28049453.html)
It should be trivial to dismiss given the widespread usage and real world advantages it provides.
And no, a single use server doesn't make SELinux useless. It still means SELinux can lock down whatever services are offered on that box better than pretty much anything else can.
As a heavy open source contributor, I don't like it either. But I'd be kidding myself if I thought volunteers approach all aspects of software development with the same rigor as someone doing it professionally. I'm guilty of that myself; I do the things I find fun, and often don't do the things I find tedious (or have to force myself to do them because I know that future-me will be pissed off at present-me if I don't).
Still, though, there are plenty of for-profit organizations out there that don't feel it's cost-effective to be rigorous about security or some other thing. And many (most?) developers and ops people are evaluated not on how bug-free and secure their work product is, but by how quickly it gets done and shipped to customers.
Does, like, anything on mainstream Linux distributions really sandbox applications by default? Let's say I run a browser, a mail client, Signal, Discord, whatever on my laptop. If one of them has a code execution vulnerability, does anything prevent that app from reading/writing all of my home directory, take screenshots, send keystrokes to other applications etc?
I haven't used anything but Linux on my laptops and PCs for at least a decade, and I genuinely don't know the answer. Back when I started with Linux, the answer was surely a "no", but maybe anything something has improved in this regard?
I don't know much about the specifics but I think Wayland fixes a lot of the security problems related to keylogging and screenshoting.
Given that Google uses Debian internally for their workstations [1], employs a number of Debian developers [2], and has discovered and fixed security issues in Debian [3], I find this argument to be entirely disingenuous.
Sure, Red Hat has a well funded security team. But so does Google, and all of the other Debian users in "big tech".
[1]: https://en.wikipedia.org/wiki/GLinux [2]: https://www.reddit.com/r/debian/comments/j4liv4/comment/g7mm... [3]: https://lwn.net/Articles/676809/
The point made in the article is that security is hard and often thankless work. So it's not something that's conducive to volunteers doing in their free time often. It does take funding to move the needle on this here, and I think Red Hat is proof of that.
To the extent that containers are a software distribution method outside of a single authority, they are a security nightmare. They are the exact equivalent of shipping a developer's laptop off to the datacenter and replicating it as a production image.
If you're building your containers on a developer laptop and then pushing them to the registry from there, yes.
You can also not do that and instead have all builds happen on a CI server that isn't ever touched directly by anyone, like you should really be doing to build any artifact that gets deployed to production, container or otherwise.
Using images from untrusted sources is a not-quite-exact equivalent of downloading code directly from npm and shipping it off to production.
I hear that a lot, but it's not really true, or it is true only if developer created the image manually. Does anyone do that?
As soon as you use a Dockerfile you have reproducible builds, allowing you to use a different base image, or even perform the installation without containers at all.
That is extremely optimistic. As soon as you do anything involving an update - `apt-get update` or similar - it's not reproducible any more, and of course you do need to do those things in most images. And if you don't need to do that, you can probably avoid doing the whole Dockerfile thing in the first place (although that may not be so easy if you're not set up for it).
But I stand by my point - container images are still way more manageable than dev laptop images.
Depends on how you build your containers. If you have a build step, which pulls your dependencies from a trusted source and versions are locked down, then MAYBE. I've seen developers have all that in place, then in their deployable container they start by doing "apt-get update && apt-get upgrade" in the Dockerfile and install some runtime dependency that way.
There is also another problem, which I believe is what OP is referring to: People will write docker-compose file, Helm charts and what-have-you, which pulls down random images from Docker hub, never to upgrade them, because that breaks something or because: It's a container, it's secure. Fair enough if you pull down the official MariaDB image, or KeyCloak, you still need to upgrade them, and often, but they are mostly trustworthy. But what happens when your service depends on an image created by some dude in Pakistan in 2017 as part of his studies, and it has never been upgraded?
I had this discussion with a large client. They where upset that we didn't patch the OS right when new security updates came out, which to me was pointless when they shipped a container with a pre-release version of Tomcat 8 that was already 18 months out of date and had known security flaws.
The reason docker containers are absolutely everywhere is that it's a convenient way to ship software that skirts around the notion that most Linux distributions are spaghetti balls of needless complexity with distribution and version specific crap that you need to deal with.
Back in the day I had to package up my software as an rpm to give it to our ops department who would then use stuff like puppet to update our servers. I also got exposed to a bit of puppet in the process. Not a thing anymore. Docker is vastly easier to deal with.
From a security point of view the most secure way to run docker containers is some kind of immutable OS that only runs containers that is probably neither Red Hat or Debian based because having package managers on an immutable OS is kind of redundant. Which is more or less what most cloud providers do to power their various docker capable services. And of course the OS is typically running on a vm inside another OS that probably also is immutable.
Docker removed the need for having people customize their servers in any way. Or even having people around with skills to do things like that.
Being container focused also changes the security problem from protecting the OS from the container to protecting the container from the OS. You don't want the OS compromised and doing things it shouldn't be doing that might compromise the one thing it is supposed to be doing: running your docker containers. Literally the only valuable thing it contains is that container.
And it indeed matters how you build and manage those.
As for the efficacy of the two, I'm less interested in the feature sets of the two. I think what'd be more interesting is replicate exploitation scenarios with their default policies and see which subsystem succeeds in mitigating the exploit and which fail.
The feature set is exactly what dictates which systems are more likely to prevent exploitation, though.
App Armor simply isn't as granular, and simpler to bypass (e.g. by making a hardlink to a file to override AppArmor policy).
AppArmor may be good enough in many situations, but SELinux gives you much more control, so you can be much closer to perfect to protect against unknown situations.
For example, I'm seeing that SELinux didn't mitigate ShellShock where AppArmor did (despite being an attack vector that isn't really common). But these are the things I want to know.
"I don't care to understand the differences in security thees systems provide"
That's what you're saying here. Which means you're not going to be evaluating these systems in any way that matters.
> If SELinux prevents a hypothetical that's great, but security is a tradeoff and I'd opt for convenience and simplicity to sacrifice potentially negligible risk.
You're asking for benchmarks, but already here your willing to dismiss the results because you don't really care about them, you care about "convenience and simplicity" and security being good enough, right?
In that case, sure AppArmor is good enough for most people.
But so is a flimsy chain look. If you want to actually secure something and not just deter, you'd want a deadbolt and sturdy door, right?
The point is, the feature sets matter, precisely because so many attacks are hypothetical. You have to see and speculate what attackers might do and have things in place to prevent that. SELinux facilitates that a lot more than AppArmor does.
That's the fundamental point here, and not something you will likely find a nice graph to support. If you ask an AI to generate one for you it might be able to though. If you really still think you need it.
> For example, I'm seeing that SELinux didn't mitigate ShellShock where AppArmor did (despite being an attack vector that isn't really common).
Well that's nonsense. SELinux can protect against anything AppArmor can since AppArmor provides only a subset of features.
I sense you might not be interested in this since you've said you just want benchmarks, but here's a page from RedHat not only explaining how SELinux prevents Shellshock from being able to do damage, but even walks you through exploiting it on an SELinux enabled system so you can test it yourself[1]. There's also a blog post from Dan Walsh explaining how SELinux constrains shell shock [2].
I'm also less confident AppArmor is as effective against container escape exploits like these [3] [4]
> But these are the things I want to know.
Mmm. Well, there's no benchmarks. But if you do research you will find the examples you want. SELinux will have substantially more examples because it's employed in wider use due to RedHat and Android.
If you really want to compare, look at the Debian and RedHat security advisories, and look how many RedHat has saying that SELinux provides protection if enabled, and how many Debian and Ubuntu have being unable to say the same for AppArmor.
But really, again, you should bother to understand the feature sets and actual technology. This approach you want looking for benchmarks, it's not going to necessary be accurate or representative, and I say that confident that your methodology would still show SELinux as the better option (in terms of security, not usability/convenience).
The key is that you should strive to understand systems you want to use, not just look for a blog article that can provide justification for an intuition or desire.
[1] https://github.com/RedHatDemos/SecurityDemos/blob/master/201...
[2] https://danwalsh.livejournal.com/71122.html
[3] https://www.redhat.com/en/blog/latest-container-exploit-runc...
[4] https://www.redhat.com/en/blog/selinux-mitigates-container-v...
This is from Dan Walsh who is quite famous in the SELinux community [1]. See the part "Why didn't SELinux block it?"
Sure you can configure SELinux with a stricter policy, but virtually nobody does that in practice, they use the defaults, mostly because SELinux policies are really only used to fix stuff SELinux default policies break, usually using the audit2allow or whatever its called.
> I sense you might not be interested in this since you've said you just want benchmarks
If the EDR/antivirus industry can have various test suites and testing organisations, the same methodologies can apply to OS security subsystems. Let's see what happens with your default RH/SELinux, Ubuntu/Debian & AppArmor, and for shits and giggles OpenBSD, and see how they fare against exploits and vulnerabilities harvested over the years, and make reproducible labs. That's what I'm asking; I'm not going to scour security advisories to do tit for tat comparisons, I would rather see a well thought approach to this.
We do this kind of comprehensive benchmarking with all sorts of software such as compression, cryptographic libraries, compilers / languages, etc. Traditionally this would've been harder in the days when virtualisation and utility computing was nascent, but the infrastructure part is pretty achievable these days. Just need to someone to expend the effort (and I'm not volunteering).
Yes, I know who Dan Walsh is, it's part of why I linked to his article in this case.
> See the part "Why didn't SELinux block it?"
Yes....did you understand it? Or just read the title and assume you were right?
SELinux didn't prevent the exploit code from running, which is exactly normal and exactly the same as AppArmor.
What it did was prevent the code from being able to do anything.
So in that sense, SELinux absolutely stopped Shellshock to the same extent AppArmor would have.
> If the EDR/antivirus industry can have various test suites and testing organisations, the same methodologies can apply to OS security subsystems.
There's fundamentally less need because thees are open source systems and the design is sufficient to judge. You're just being incredibly lazy and justifying not wanting to learn these designs, or you otherwise don't want to put in the work to do so.
Imagine you see the design for a speedboat and a submarine. It's sufficient look at the designs alone to see that one cannot operate submerged underwater for an extended period of time. There is no need for tests to demonstrate that point.
The reason for EDR/antivirus test suites is because most solutions were closed source black magic and some kind of comparison is needed. That isn't as true in this case.
> Let's see what happens with your default RH/SELinux, Ubuntu/Debian & AppArmor, and for shits and giggles OpenBSD, and see how they fare against exploits and vulnerabilities harvested over the years, and make reproducible labs.
This already exists in the form of security advisories, as I said you are just being very lazy. As I said, feel free to check or ask an LLM to do it for you, to ask how many Debian security advisories were mitigated by AppArmor vs how many RedHat says were mitigated by SELinux. OpenBSD doesn't apply since it has no type of similar system.
> I'm not going to scour security advisories to do tit for tat comparisons, I would rather see a well thought approach to this.
Right, as I said you're being lazy. The information exists, you just don't want to put in the effort, you want a nice blog post you can blindly refer to.
> Just need to someone to expend the effort (and I'm not volunteering).
No one will because there is no need. The designs of these systems show what they can and cannot restrict and that is enough. You should put in, at least, a minimum of effort to understand why these abstract systems are not really comparable to antivirus solutions and why the testsuites you think make sense, don't.
As a datum, I have a laptop that's running Fedora, the install is on the order of ten years old (routinely upgraded to new releases), and it's never had SELinux disabled.
> Still. Many in the open source community have interpreted Red Hat’s decision for what it really was: A dick move.
I've had a short essay in draft for a while about the difficulty of a small business trying to make money using The Red Hat Model (https://opencoreventures.com/blog/2023-04-red-hat-model-only...). Red Hat seem like an outlier who're doing well with that model, but smaller places like Sidero or Bitfield had to find other ways to monetise their open source efforts, and sometimes that had pushback from the community.
Red Hat, though, were acquired by IBM, and IBM made it harder for an otherwise thriving ecosystem to exist. Not impossible, but harder. IBM makes money hand over fist (billions according to https://www.ibm.com/annualreport/). Was there really a reason to make Red Hat harder to redistribute? The interviews I've read come down to "our Red Hat team works hard and we don't want to give that away to low effort projects", though if you've got an interview with a different perspective I'd love to read it.
Ubuntu and Debian are for hobbiests and lawyers, and you should never run a public server on debian/Ubuntu if you care about security.
Was it professional when they let SSH vulnerabilities exist in RHEL7 forcing perfectly useable machines to upgrade to 8 for remediation?
Don’t get me wrong, they’re the new “nobody got fired for” company (technically still the same). That doesn’t imply Debian and Ubuntu are less secure except in name. Go to Google cloud and see what CIS hardened images exist.
Your perspective is an oversimplification if not completely wrong.
And Linux in general has less resources to develop and maintain comprehensive security policies comparable to those provided by Microsoft.
Yet here we are, with Microsoft products so "secure" that they're insecure unless you have a PHd in b****, being so convoluted and over-built that people have to migrate away from it just to recover the actual security they used to enjoy back when they were able to wrap their head around the whole stack.
If devs want things to be more secure, stop developing more acronyms and just educate the userbase on the acronyms they already have.
https://arstechnica.com/security/2024/01/microsoft-network-b...
Wasn't caught for two months and wasn't fixed until months after. How is Microsoft allowed anywhere *near* the bidding process for gov contracts anymore?
Certainly SELinux has its place but I never found the value it offers to be worth the complexity it adds.
Signal has to provide 100% of the features and convenience of Whatsapp and some more without compromising security for it to be a viable alternative.
Source: Just did it a couple weeks ago.
This is akin to someone writing an article about how Oracle and Microsoft got databases wrong because they didn't embrace some security feature that only DB2 has and that more than half of DB2 users out there think is a giant pain in the neck.
> Learn the basics of SELinux, including type enforcement, Multi-Category Security (MCS) Enforcement, and Multi-Level Security (MLS) Enforcement, with the help of some friendly cats and dogs!
Do I see what looks like the Penguin dogging the Dog on page 7?
There is explanatory security text for each image.
Image semantics can be derived from text semantics.
Complexity is generally really bad for security. It results in people working around the system or just turning it off. Security is not just "in theory" - a perfectly secure system that most users disable is an insecure system.
It reminds me a bit of the idea of making people change their password every month. Sure, in theory it reduces time a compromised credential can be abused for. In practise though it means nobody can remember their password, people start using really poor passwords and writing them down on post it notes. The net result is much worse security practically speaking, even if its better theoretically.
Debian uses AppArmor by default, probably because of the Canonical influence (there are more Debian developers and maintainers paid by Canonical than by RedHat).
But you can run Debian with SELinux (as well as with other LSMs, MACs, etc like Tomoyo).
At my last jobs, we disabled any of SELinux, AppArmor and Auditd on Debian/Ubuntu, just for the sake of performance. And we never detected any security issue for our usage and requirements. So I'm not an expert in this field.
Not sure what the purpose of the article, or the whole blog, is. You want to influence the choosing of Debian Vs RHEL Vs Oracle Linux in some place? As I'm not sure, will stop here.
I do a lot in Kubernetes, and there's been more than one CVE with a line like "Affects all versions of docker/containerd, unless running SELinux," which gave me a lot of reassurance that the effort put into making SELinux work was worth it.
Now that I'm on Debian, I'm slowly building a new set of policies for the OS. Thankfully SELinux has an excellent reference policy[1] to base off of. I'm hoping my new debian base images for my homelab & elsewhere will have a nice restrictive default SELinux policy by the end of the year. I hope there's more community effort here as well, SELinux really can't compare to AppArmor, and is absolutely necessary for proper security.
Honestly I'd love if the wider community took another stab at replacing SELinux with a KSM that had similar functionality but better tooling and design. I'd pick it up in a heartbeat, but right now SELinux is what we have.
SELinux can be frustrating without the proper background about what it is, how it works, and how it helps you. There is a surprising amount of tooling for it actually.
I've seen this too, but I usually see AA mentioned in the same situations as an equivalent mitigation to SELinux.
- CVE-2016-9962 - Bypasses Apparmor [2], mitigated by SELinux
- CVE-2022-0492 - Apparmor and Seccomp also protect against
- CVE-2019-5736 - Mixed, blocked by the default SELinux policy in RHEL (not Fedora), not blocked by the default AppArmor policy[3]
- CVE-2021-3156 - This one is not a good one for RedHat to put on the list. SELinux by default doesn't protect against it, Debian 10 at the time had a Linux security feature enabled (fs.protected_symlinks) that helped mitigate it, and additionally CVE-2021-23240 came out which had similar effects but only occurred on SELinux systems.
- CVE-2019-9213 - Not mitigated by AppArmor, mitigated by SELinux
- CVE-2019-13272 - Not mitigated by AppArmor, not mitigated by default SELinux policy, but easy to mitigate by enabling boolean. I'd consider this a win for SELinux, but only just.
While digging into this more, I came across this BlackHat talk[4] which really quantifies how SELinux improves security (though doesn't contrast it with AppArmor). I also came across a paper on usability of SELinux and AppArmor[5] which brings up an interesting point: If the tool is too complex, even if it's more powerful, more often than not it won't end up having better results.
That's all to say, I think if you're willing to invest a lot of time into it (say you want to make security your niche in your development career), SELinux is still the best. But I can see why many may gravitate towards AppArmor so as to not make perfect the enemy of good. That said, I still wish Debian had a choice between the two, right now SELinux isn't really doable without a lot of work.
[1]: https://access.redhat.com/solutions/7032454
[2]: https://github.com/opencontainers/runc/issues/2128
[3]: https://www.cloudfoundry.org/blog/cve-2019-5736/
[4]: https://www.youtube.com/watch?v=EkL1sDMXRVk
[5]: https://researchportal.murdoch.edu.au/esploro/outputs/journa...
“Red Hat owned making this policy apply to most of the popular software they distribute. On Debian the users have to set everything up.” — this sentiment is directly parallel to how BSDs see themselves as providing a whole consistent operating system, Linux meanwhile just wants to ship a kernel.
“Debian doesn't care enough about security.” — says everyone who runs OpenBSD.
“With SELinux policies, containers are isolated from the system.” — you could almost say they are “in jail,” maybe we could package this up as a syscall, hm, but what to call it...
IDK what BSD looks like in 2024, but in ~2004 you would have seen this exact same article about Debian, but comparing to FreeBSD instead of RHEL.
Linux is just an OS kernel. If you want a consistent OS, use RHEL, Ubuntu, Fedora, Android or something else.
> The ugly truth is that security is hard. It’s tedious. Unpleasant. And requires a lot of work to get right.
I use Red Hat-based distributions at work and Debian/Ubuntu in my personal life. A few years ago, I bit the bullet and learned enough of SELinux to run my workstation and all my servers in enforcing mode. The author of this article is right to credit Red Hat for all the work they’ve done to provide users with default SELinux policies that work out of the box. At one time, I considered installing SELinux on my Debian system and modifying Red Hat’s policies to work with the Debian packages. I realised how much work would be involved so I chose the path of least resistance: AppArmor (which does the job).
These aren't "attack surfaces left exposed" this is "users allowed to control their own computer and decide for themselves". And I notice the vast majority of this complaint about insecurity is not about running applications on Debian or RHEL, but instead about the systems built up for running things containerized and trying to mitigate all the problems that causes. Debian concentrates more on actually having an OS you can run applications on rather than a system for deploying containers.
>In the end, the choice between Debian and Red Hat isn’t just about corporate influence versus community-driven development. It’s also a choice between a system that assumes the best and one that prepares for the worst. Unfortunately in today’s highly connected world, pessimism is a necessity.
In the end it's about weather you think you should control your computer or weather someone else will control your computer. Pick appropriately for context.
Debian is many other things as well!
I suspect Debian is used on more server installs than desktop ones. While it doesn't come with enterprise support options like RedHat it is most certainly used on servers, many of which are in corporate environments and are running multiple services (in containers often) or are otherwise multi-user.
TBH I've always considered Ubuntu (and by inference its derivatives) more of an “inspired by” in relation to Debian, given it is generally closer to Testing then Stable and has many notable changes on top, more so as bigger changes have increased over time (snaps being so ingrained that they are almost required, for one).
I don't think you understand how Ubuntu is used. It is huge in the cloud space as a server.
The real risk comes from network-facing services and they are much better protected by seccomp and cgroups, usually configured in systemd, and Debian uses that extensively.
Seccomp can even protect vulnerable system calls. SELinux is not able to do that.
`systemd-analyze security <service unit name>` gives a nice list of things to consider tweaking. You don't have to fix everything or pay attention to the exposure ratings, just use it as a guide.
I did this for chrony, haproxy, nginx, tor and unbound on my Debian router. I also have some timer units to run scripts to update DNS blocklists and such, which have the same kind of hardening. For the services, some of them have caveats and can't be fully hardened, eg unbound needs PrivateUsers=no because it fails if it can't find an unbound:unbound user to switch to, even if it was already started as unbound:unbound by systemd. And SystemCallFilter makes it easy to get overzealous and only allow the exact set of syscalls that you see the service making, only to have a service update or glibc update that starts making a new syscall and requires another round of debugging, so do it in moderation :)
If an attacker gets execution in userspace, it's best to assume they can also get into the kernel via some 0-day local privilege escalation...
There needs to be better graphical tools for this, like a "profiler" or similar that watches a process for a specific time for errors in the config and that incrementally adds features while the process is running.
In my opinion, systemd sandboxes are where it's at. [1] They are seccomp based sandboxes, but have a lot of isolation and sandboxing features that are very easy to use, and they can also be incrementally enhanced with both SELinux and AppArmor profiles.
[1] "man systemd.exec" or https://manpages.ubuntu.com/manpages/bionic/man5/systemd.exe...
That way if someone does manage to break out of a container they have the privileges of a dummy user that doesn't exist on the host, so unless they are using a kernel exploit, they don't have any privileges to be able to do any damage.
That's not to say on very specific systems that need to be hardened, I do enable selinux and am glad it's an option. And if I have to use a security layer, I take the object based selinux over the path based apparmor.
RedHat have such bad moves, deploying unfinished, instable and unsecure softwares.
Today most security breaches came from crappy applications with an immense set of dependency put into production because someone want them, there is no protection for them, adding long and painful system stuff is only a way to have also badly configured systems.
Debian issues are more in a complex custom setup, preseed it's a nightmare compared to NixOS, which is much more important than SELinux, regularly disable on most deploys.
I wonder how many people that agree with this nonsense position also agreed with the keepassxc nonsense position.
You have to do 100% coverage testing on whatever program you're using. (Good luck if you don't have the source code.) Otherwise, you don't have any guarantee that your program won't seemingly be killed randomly.
Good luck, x2, if you have some snake oil "endpoint security" that keeps overriding your SELinux policy changes.
"Generate a SELinux policy for daemon X. This daemon accesses it's config file in /etc and it's runtime data in /var/x. It listens on network. All other activities should be disabled"