Hacker News Reader

Comment by fefe23 | Hacker News Reader

The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand.

The whole IT ecosystem has become a hail mary. Even admins usually have no idea what a certain program actually wants to do. If the admin knows how to install the app so that it actually runs, you call them a good admin.

From a security point of view, an application is like a nuclear power plant. It's good if it works as planned, but if something blows up it endangers your whole enterprise.

The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl. That way the radiation hopefully stays in, but history has shown that it really doesn't. Also, the wheel of history has just turned one more iteration and now admins just view the sarcophagus as something you deploy as you previously deployed the app. Who is responsible that it is air tight? Well, uh, nobody, really.

You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.

Obscure files like /etc/localtime, /etc/resolv.conf, /etc/ld.so.conf, /dev/zero ... how can you expect devs to build well documented and well sandboxable applications if they don't know which files their library functions will open?

You may have heard of /etc/resolv.conf ... but have you heard of /etc/gai.conf? /etc/nsswitch.conf? /etc/host.conf? Wouldn't it be great if the man page of getaddrinfo mentioned those (mine only mentions gai.conf)

View on HN

> The problem is not so much that selinux is too complicated (it is as complicated as it needs to be)

I disagree.

> but that we all run software we don't understand.

I fully agree.

My disagreement lies in the fact that you've described the problem, but are proposing that some software (SELinux) that fails to solve the problem is somehow good.

SELinux might be a perfect tool in an ideal utopia where everyone understands all the software they run, but that isn't the real world, and having a tool that works in a theoretical world isn't particularly useful.

Either SELinux is applicable to the real world, or it's not useful and not really fit for purpose.

This isn't a "blame game" - it's not about figuring out why things are bad (no-one understands their software components), nor blaming SELinux (it hasn't caused these problems, it's just failing to mitigate them). It's about figuring out how to improve that situation. Does SELinux do that effectively?

>My disagreement lies in the fact that you've described the problem, but are proposing that some software (SELinux) that fails to solve the problem is somehow good.

My opinion is that there's a cultural and policy problem and you're simply not going to solve it with technology. I don't think SE Linux is bad, it just isn't the answer here. One (clearly not the only) main reasons people just run a bunch of software they don't understand is due to expectations in productivity. In an idealistic world with caution built in, we are willing to sacrifice some productivity for security, stability, understanding, etc. I think many engineers like these ideas because they like to work with and build solid systems they understand very well--otherwise they wouldn't be practicing any form of engineering.

What we instead have is a world of engineering with no sort of guard rails. Unlike Civil Engineering, there's no licensing, no sorts of inspections, no oversight, liability is completepy waived in a 300 page agreement people just click through, etc. Even when things go horribly wrong (e.g. Equifax) the repercussions are limited. We see abuse in other engineering fields where actual life and death are factors and they're still just amortized in a cost analysis.

Ultimately though engineers simply don't lead the world (and arguably, maybe they shouldn't) , people with wealth lead the world (arguably, they shouldn't either) because it underpins pretty much all other systems anymore. As such, productivity is always prioritized over safety and security. This issue isn't limited to software, it happens everywhere almost ubiquitously at this point. Software is one of those areas where most people just care less, so you can get away with more and more of this. The issue is that I believe we're building a house of cards that will be difficult to revert to a reasonably stable state once it finally collapses.

> otherwise they wouldn't be practicing any form of engineering.

This is incorrect. The dirty little secret here is that software engineering is an extremely lucrative career, and a ton of people have gotten into the field for that reason. I'm not saying they're wrong to do so, but I have never met a wizard who didn't love the craft. I have met tons of people faking-it-till-they-make-it who are there for the RSUs and free food. The latter tend to practice resume-driven development, and tend to have very little idea how their code actually works.

> ... and tend to have very little idea how their code actually works.

Or they know how their code works, but do not care. These are the min-max'ers who will MVP their way on every conceivable angle to grab the checkmark from the project management office and run off to get the next checkmark as quickly as they can to rack up enough for the next promotion. They look awesome on paper to metrics-staring management. They are accelerated bit rot to any organization's codebase. And to rein them in, you have to challenge them in such excruciating detail that it starts to appear like bikeshedding to outsiders.

Except the difference from bikeshedding is the code they leave behind is utter crap to maintain, extend, operationalize, debug, instrument, monitor and only tested under the happy path with maybe the handful of sad paths you dragged them kicking and screaming through code reviews to implement. They know they can out-wait you and the impatient business users in these reviews, and will miserly implement only what you force upon them in these painful, political battles.

The only way you can out them is within an organization where operations teams have the power to write out in detail the challenges with their code after taking on operations duties for awhile, and return operations to the developer with literal pager duty attached until the gaps are addressed.

> Unlike Civil Engineering, there's no licensing, no sorts of inspections, no oversight, liability is completepy waived in a 300 page agreement people just click through, etc.

This! I have come to the same conclusion. If you want to construct a building you have to comply with multiple standards and industry regulations that serve as limitations and safeguards that follow your project from the initial idea up to (sometimes) the point of destruction of the constructed object.

In our industry even with the standards that we have they are oftentimes treated as guidelines not rules. Of course there are exceptions but those are just a byproduct(or proof) of the rules themselves.

>If you want to construct a building you have to comply with multiple standards and industry regulations that serve as limitations and safeguards that follow your project [...]

Also, don't forget that the laws of physics also weigh heavily on the minds of most engineering disciplines. Stuff has density, strength, thermal and kinetic properties that reflect omnipresent constraints on most decisions.

SELinux is a perfect tool in its niche — where that niche is "secure-installation system integrators." It's not a tool for regular people; it's for e.g. defense subcontractors who want to certify their virtual appliance as meeting certain standards demanded by certain customers; where meeting those standards requires them to actually go through their virtual appliance from top to bottom and prove the whole thing secure and sandboxed.

It's not a tool for application developers (who at most can supply SELinux templates to start with); nor is it a tool for sysadmins (who really don't have the time or the incentives to deal with it.) It's a tool to be wielded specifically by people occupying a role that, in most organizations, doesn't exist.

SELinux is applicable in the real world, but you must choose if you want it to be Fast (use globs of rules written by unknown people who don’t know your requirements), Cheap (start from zero and develop policies specifically for your app), or Good (do both while learning a substantial amount about SELinux).

If you’re using SELinux rules off the shelf for a handful of Fedora boxes for personal use, maybe that use case doesn’t work. But is that more “real world” than an institution like the NSA trying to completely lock down its production systems?

I think what people are missing (and perhaps the author is at fault for not calling this out more explicitly), is that the complaint is effectively: SELinux is more difficult to configure than it needs to be / should reasonably be.

Perhaps due to unnecessary internal architectural complexity, the lack of a good configuration interface, and/or bad config docs (which of these is the biggest culprit I'm not sure, but they all seem to have some level of truth to them).

The gp posited that it's necessarily complex - subjective, but I don't buy it. The author provides a good selection of examples that seem straightforwardly non-sensical in their operation: why doesn't audit2allow integrate properly with the overall SELinux architecture instead of requiring separate knowledge of semodule for application. Why doesn't the system keep track of label deprecation as system dependencies are changed & updated? These seem like pretty obvious out-of-the-box basics you would design in, if SELinux had been "designed", but the answer to these questions seems likely to be that it's more of a bundle of un-integrated tools and loose-text configs/blobs without well-thought-through centrally architected database of labels, maps and APIs.

One could even take this blogpost as a constructive set of ideas to improve SELinux, but the scale of the issues do seem overwhelming to change on such a large project, and based on the comments here on HN there seems little will to do so. To the point that perhaps a separate / new approach would be easier.

Honestly, the bigger issue is that most SWEs just aren’t very good. It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…

System design, though, is the actual point of SW ENGINEERING. That’s the part that is responsible for creating a foundation of quality to build on.

The other side is that sysadmins have largely become DevOps or SRE roles, which means most folks left as a sysadmin are those that couldn’t hack it in the other roles. In the end, as we grow the number of people in tech, the number of software and systems, and the complexity we experience a regression to the mean across the board.

It’s always been a pet peeve of mine that I can count on my hand the number of SWEs in my ~20yr career that had similar understanding of the system they worked with as their SRE counterparts… but this should be table stakes.

> It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…

Yes. It's absurd that anyone that can talk about this stuff at a basic level is seen as some sort of ultra-skilled professional. See also: the relatively small amount of chatter about this stuff in the blogosphere, where there's a million tutorials that amount to "here's how you run create-react-app" and about 3 that talk about even moderate system design topics.

> Honestly, the bigger issue is that most SWEs just aren’t very good. It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…

I think this is a bit unfair, the software industry is so big now there are lots of different sub-sectors. Big Tech companies have huge scale problems that are kind of unique. Most developers work on products that only have a few hundred users at a time.

Your 20 years experience is a bit telling. I'm the same age, back then SWEs usually had full access to production environments. Now everything is segregated and devs can't experiment in prod and end up getting detached.

Maybe we're just in different circles, or perhaps have different definitions of system design, but that's not been my experience. Usually I see developers being very excited about system design and quite good at understanding system design questions - nothing's more fun than architecting a new system and thinking about stuff like scalability.

Where people tend to fail is when they have to implement these systems, and they've planned for all these high level building blocks but then actually have to configure them. Redis, RabbitMQ, GraphQL, Docker, NGINX, the list goes on - there's so much to configure and you really don't truly know how they work unless you dive into the source code, so you inevitably get bitten by subtle bugs and errors where your expectations of how something should work based on the docs are violated by how it actually works.

Should a SWE have a deep level understanding of all this stuff? I'm not sure. Usually you can still hack it into a working system after a little bit of firefighting, and that's likely more time efficient than understanding source code or rolling your own infrastructure. But I definitely think there's room for more comprehensive and transparent documentation to bridge the gap between the source code and barebones docs.

Agree completely. I've never known a software engineer who didn't love designing new systems on paper and starting the initial development. It's when it comes time to stabilize and turn a system into a usable product minds quickly wander to designing the next system that fixes the flaws of the last system.

>But I definitely think there's room for more comprehensive and transparent documentation to bridge the gap between the source code and barebones docs.

This is usually provided by technical books, which are available for a much wider range of systems than you'd think.

> It’s extremely telling that when you spend time in tech forums most people dread system design questions as the harder side of interviewing for senior level SWE roles…

> System design, though, is the actual point of SW ENGINEERING.

I mean, I would say I'm pretty good at engineering and system design (10 YoE, senior/principal for most of them). But I dread the system design interview questions because (in my experience) they bear almost no relation to the system design experience I have and almost no relation to the actual requirements of the job. Things like "design an ETL flow" or "build a distributed wikipedia downloader" are just not in my area of experience. (I work mostly in the lower layers of single-host operating systems and file systems.)

In the real world, if I was tasked with "build a distrbuted wikipedia downloader" I wouldn't just start randomly guessing.

I'd get input, feedback, discuss, narrow down options.

Yet for some reason, that's an actual interview question. Start randomly guessing based on your knowledge of something, in a way that you'd never do in a real scenario.

Edit:

I feel like I should do an interview like this. Refuse to explain how to build it, and instead explain the detailed process on how I would successfully build and complete the project. lol

I totally agree that system design questions can be silly sometimes. When I do system design interview questions for senior engineers, I ask a really broad and complex questions and let the candidate drive us towards the system. The point of the interview is evaluating if the candidate can complete those soft skills in a reasonable way. I try to answer basic questions such as these:

Did the candidate just go for design without asking questions?

What questions does the candidate ask?

What areas are they asking questions about?

Can they take critical feedback on their design well?

There is a little bit of discussion on technology choice, but typically if they can justify an out of the norm choice it is fine.

That makes sense. I guess I've delivered incredibly complex systems in a tremendous number of diverse use cases, and yet I have zero confidence I could pass a systems design interview.

The few times I've had them, they've focused on how much knowledge of a particular choice I have.

When I build systems, it's incredibly common I don't know much about the topic. Especially in innovative spaces where you're literally the first team ever building a solution at this scale.

Take for example critical feedback, I can't imagine having to judge someone on that in an hour phone call.

The best teams argue quite a bit. That's healthy and reasonable. The question is if they accept, and respond to the argument, with positives and negatives, and how basically how they dance.

I don't know how to put into words what that dance is like, but you know it when you feel it. Maybe that's what you're looking for, a feeling of what that dance is like, but I'm glad I'm not making that determination on a quick basis.

Just random thoughts, thankfully I get all my employment through glowing referrals, and try to avoid interviewing anywhere without them so I can skip all of this questionable stuff.

I think most devs see that the jobs are asking for overloaded responsibilities from multiple other jobs and they're trying to live up to that. It also became the norm to see job postings which would have been 3-4 distinct positions a decade back.

True, but even in the 90's and early 2000's, before cloud took off, most developers weren't sysadmins, and they didn't want to be. That wasn't their specialty. The "Unix wizard" developer was always a rare breed.

> System design, though, is the actual point of SW ENGINEERING

Software Engineering is about problem solving using software and hardware (we'll just say computers for simplicity). Whatever solution system you propose has inherent constraints and tradeoffs. That's the Engineering. System design is a catchall phrase for a subjective problem scale.

Speaking personally, the reason I dread system design questions is because it's asking me to give an off-the-cuff answer to something I would prefer to take weeks to consider and research.

SRE = Site reliability engineer?

SWE = Software Engineer (obviously)

>System design, though, is the actual point of SW ENGINEERING.

I thought system design was the point of system engineering?

Software is a system.

> The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand.

There are many many problems.

One of the biggest problem with SELinux is that it is trying to graft Mandatory Access Controls on a userland that is not designed for it.

Unix, frankly, is not designed for security. It is designed to get work done by writing a bunch of little buggy C programs that you string together in novel ways.

Security is something that was grafted on it. And it shows.

How many decades of security vulnerabilities have occurred because of a shared /tmp space? 40 years?

How do you graft access controls on a system designed with no access controls and potentially billions of combinations of programs, paths, and various other resources without breaking anything?

The answer is: You don't. You can't.

Were as you have a system designed for security, like Android, and literally hundreds of millions of fully SELinux-enabled fully locked down user-facing Linux devices are out there being used by people who haven't the faintest clue what "audit2allow" is and wouldn't understand it if you tried to explain it to them.

So it's less of an issue of "we all use software that we don't understand". SELinux is complicated enough that you can devote your life to trying to understand it and still fail to craft good rules for other people.

It's more of an issue of "Linux userland follows the basic Unix design from the 1970s which is kinda shit if your goal is security".

It is just bad design. Pure and simple. That is all there really is.

However there is a way out.

The way out is to give each process and each user their own little itty bitty special Unix environment where they can do whatever they want. And then you use strong divisions to keep them mostly unaware of each other. Use a default deny policy and only poke holes in it when required.

My android phone came without a calculator app. I recently realized this in a setting where I needed a calculator. I commented out loud about the lack of calculator and got some responses from the group I was in at the time.

"Just download one from the app store," I was told. "But be prepared to grant it network access, file access, contacts access, camera access, and email privileges."

The alternative, which is Linux, is to grant your calculator app the permissions to read and write to the same resources that your browser uses to store the password for your bank.

Well "grant" is too strong of a word. "By default and there is nothing you can do about it unless you are exceptionally skilled" is more accurate.

Also your calculator app can read your sudo password as you type it, which you do a dozen times a day to carry out complex and security sensitive tasks such as "Connect to printer to print out mom's recipe for brownies" and "restart bluetooth because it's buggy and you want to listen to spotify on your wireless headphones".

> Also your calculator app can read your sudo password as you type it

True with X11. Fixed in Wayland.

Wayland fixed one problem, but there's no shortage of local privilege escalation bugs with which the calculator can still read the sudo password.

I want to learn more. Do you have some examples?

Just yesterday, Microsoft found a few[0]. There's no shortage of these, but more important is the haphazard way fixes are backported to longterm (e.g. [1]). This reached the point that Google's security advice is 'always follow the latest kernel'[2], except most users and distros simply cannot afford it, so they are stuck with a vulnerable system.

Linux is not unique here. In the longterm, all the typical desktop OSs will need significant structural changes far beyond 'chase vulnerabilities and patch everything all the time'.

[0] https://www.theregister.com/2022/04/27/microsoft-linux-vulne...

[1] https://nvd.nist.gov/vuln/detail/CVE-2019-15902

[2] https://security.googleblog.com/2021/08/linux-kernel-securit...

Which is still not the standard - probably will take at least a decade before we can even talk about this issue being fixed.

standard or not, it's already shipping by default in ubuntu and fedora except for when nvidia drivers are involved.

The problems will be worked out.. one way or the other.

Oh, I agree. I'm just noting that access controls are useless in a culture that ignores them. I think that goes for the original topic of SELinux.

That's really an app store problem.

Nothing says that Google couldn't prioritize applications that need the least permissions, or make it easy to filter by permissions, or even have an acceptance policy that said "A calculator app can't look into your contacts".

The system could favor the user in this regard, Google just doesn't want it to.

That is quite an exaggeration. My calculator app has requested 0 permissions. That's right, none.

Even though I use the calculator app from a famous ad company that loves gathering data - Google.

a.k.a container.

You really shouldn't rely on containers for this, unfortunately.

Single page of instructions from google apps has me download a tarball, add a hello world command, and Presto! I have a webapp on my computer! Then change this one line and PRESTO! My webapp is in the cloud!!

No telling how, or with what components, or what the dependencies or security implications are, or how to see the logging for the app. Just a single paragraph of instructions and you can be the next internet startup IPO!

'Hail Mary' is the best way to describe the situation I've yet heard.

Containers are more like a trash bag. Nobody expects it to be air tight, just good enough to make it to the dumpster.

I always felt containers were always about packaging and deployment, not security. Any "security" was a byproduct of isolation, not an end goal.

I fully agree with your last point. In my experience, the all-versions-packaged approach of containers actually leads to a worse patch state than a simple Ubuntu or Debian installation with unattended-upgrades.

I've tried to explain to our Devops guys that they need to automate their patching so that a new container is fully patched before deployment. They had no idea what I was talking about. I said, "you have containers that run for months, right? And they aren't modified at all after deployment, right? So how do you address new vulnerabilities? How often are you patching your base image?"

Just clueless looks.

How would you accomplish this exactly? Your use case may vary, but in my mind dependencies shouldn’t just be upgraded by some automated system without some sort of feedback mechanism (like tests).

If you’re going to run tests against the new dependencies, then why not just shift that whole activity to the development process itself (not the deployment system)?

When dependencies are sufficiently different from the existing prod dependencies, you can release a new version, instead of expecting ops/devops to take care of your dependencies for you.

I'm actually agnostic as to how it's achieved. I just think that the idea of running a container for three months without patching the underlying OS is nuts. Generally our guys only redeploy a container when there's been an update, so that OS is pretty stale compared to what I'm running on traditional VMs. It really was more of a reflection on how DevOps can become too focused on application development and deployment, as opposed to deploying holistic systems. We have coders writing apps that feed into our CI/CD system, we have admins who manage kubernetes/Swarm, and we have traditional sysadmins managing VMs. All with differing priorities and focuses.

I kinda stopped worrying too much about it because the "OS" (really userspace) that lives in a container isn't in a position to be exploited like the underlying VM OS is. Nothing in the container is privileged and there are no security boundaries that exist within a container. You should care exactly as much about container image updates as you do gem/npm/pip updates and I guarantee you have months old deps pinned in a lockfile somewhere.

What people hear is "Oh my god you're still running Ubuntu 14.04?!?" but what is actually happening is all the defenses and security boundaries are being enforced by a fully patched RHEL 7 instance with SELinux, the completely unprivileged application code just happens to have an old version of libz bundled with it. So for sure the application is in a position to be compromised depending on how old the libs and the vulns that might exist but that's the developer's concern exactly like their NPM deps.

So an application runs in a container that has a vulnerable OS. This has network access, DB access, NFS or SMB access. All valuable stuff. And then the container's OS gets breached due to the vulnerability. Sure, the VM hosting the Docker containers is secure, but that doesn't really matter if the breached container had access to PII or other valuable data.

This idea that people have about containers being 'the "OS (really userspace)' is fundamentally wrong. It's the equivalent of a VM, just (hopefully) stripped down to the bare essentials. There's no magic that protects it. The Docker host sure doesn't protect it, and in fact the Docker host can be vulnerable to exploitation from its containers, just as a vSphere host (or any other hypervisor) can be exploited if its VMs are insecure.

> And then the container's OS gets breached

This can't happen. There is no "OS" actually running. It's a bunch of binaries and files sitting in a tar archive that the process running can access and link against. What are you going to breach? All the networking, firewalls, kernel, remote access like SSH, and privilege management is the hosts. The only process running in the container is the application.

What you mean is "what if the application running in the container is compromised?" For example what if there's a RCE vuln in the version of libcurl that the app links against? Absolutely! Now you've got a huge problem but this is exactly the same as if there was an RCE in leftPad. In container world that's now the developer's problem instead of yours. Everything in the container is vendored, just pretend it was all statically linked. It's not like an VM where ops patches it.

> It's the equivalent of a VM, just (hopefully) stripped down to the bare essentials

No, no 1000 times no. It's in all ways the equivalent of a process (or tree of processes) running on the host machine. From a security perspective a container is no different than any other binary running on the host. You have it exactly right, there is exactly zero magic and no inherent security.

You should lose exactly as much sleep over outdated container images as you lose over outdated rubygems in your developers' Bundle.lock. Whether that means a lot or a little depends on the company and how security critical the application is.

Thank you for this clear explanation without any attacks! I hate to think how many people I've misinformed about this at work. I think what caused my confusion is how our devops team has been using Docker. They've basically treated it as a vm environment that doesn't require approval from my team. And some of their practices just seem to cause me stress that I should just free myself from.

Would it blow your mind to hear that some devops people actually run more than one application in a container? And that one of those apps might just be sshd? That's one of the many that I know about. Why they wouldn't just run docker exec?

Ok you got me (I laughed)

I'm just so sick of the container security shills; in particular I'm tired of people selling debian advisories with zero value add on enterprise contract.

Ah, the OS itself. I thought you meant the application dependencies.

If you're using an upstream image from a distro, pull that down, update it, push it back to a registry, then use that as your base image. Do that every day in a CI/CD (or just cron) system, such that you've got logs, auditing and fires off alerts if something breaks.

Add update steps to the builds of your final image containers too at the start, to catch any delta that may have happened during last base image patch.

There are also container scanning tools but they can produce some false positives sometimes but useful nonetheless.

You still run tests as normal against the new images before promoting, just like normal.

Exactly what I've been trying to push to the team.

If that matters why hasn't the internet exploded yet?

Hardly anyone is doing that thing and it seems to not matter. People who panic only when they see $STUPIDLY_NAMED_ISSUE seem fine.

Ransomware incidents are rampant and PII breaches are quite common. Remember Equifax, who lost the PII of basically all adult US citizens?

Sure, there are extremely sophisticated, targeted hacks. But those are the minority. There are also hacks where hacker groups accidentally (!) shut down an oil pipeline. The reason the internet hasn't completely burned down is three-fold:

- We accepted breaches as something "that just happens" and that "nothing is 100% secure" - which is true, yes, but applied very broadly.

- Our operating systems are generally well secured and most exploits are very situational.

- The hacker groups (at least most of them) don't put in too much effort either, since they make a very good living of the lowest common denominator.

I work for a stodgy company with over $4B under management. In a highly regulated industry where you have to publicly announce breaches. Sure it may not matter to a quick startup where someone is re-inventing Uber for gokarts, or other small industries, but when you have to satisfy auditors and regulators across different jurisdictions, you can't take a laissez faire attitude towards patching and compliance.

And considering how many breaches occur on a weekly basis compromising customer PII etc, its foolhardy to say the Internet hasn't exploded yet. It's been in a continual burn for decades.

Unless there's a vulnerability in something internet facing, you can go a very long time without updating anything. So, yes, it doesn't matter... except when it does. I've logged into production systems with multi year uptimes, with updates that haven't been applied in just as long.

Many orgs have a if it's not broke, don't fix it policy. Applying needless updates are just as likely to screw something up.

Many orgs get breached (or have no way to detect if they've been breached).

It depends on both your industry and your risk tolerance. We generally apply every security patch that comes along for RHEL. Application patching is different, it depends on the severity and what the vendor will support. If we can't patch, we try to apply compensating controls to mitigate the risk.

And the pride we all used to have with long uptimes is really a poor badge of courage. It shows a misplaced trust in either your security, or the threat model you face.

OS security patches (at least for RHEL) pose little risk of breaking applications. In the last 15 years, I've never once had to roll back a security patch from RH.

Security patches can be applied even to the kernel without restarting anything though, so having a high uptime is not mutually exclusive with staying up to date. Of course, you do have to pay extra for live patching.

I'm still leery of ksplice etc. And the cost is a factor too.

/me nods, but it is so very _very_ convenient.

I would argue that the fundamental problem is that the companies selling software "engineering" products do not actually take any responsibility that the product they have engineered works as intended. (see: https://www.snopes.com/fact-check/car-balk/ )

And of course, the main reason they do not take the responsibility is that the customers won't pay for it.

It is kind of interesting. We are very good at making bridges that do not fail unexpectedly, even if there is unlimited amount of unknown failure modes when working with physical materials. And on the software side, well, managing to come up with a fizzbuzz without failure modes is used to screen people on interviews. What would be similar for bridge engineers? Here is a hammer, nail and two pieces of wood. Can you make the two pieces of wood stick together?

There are a limited number of known failure modes when working with physical materials. We know the properties of materials and we know the physics.

The equivalent to fizzbuzz would be something like a free body diagram.

The engineer building the bridge is liable. Hence, you get engineers with qualifications (instead of a random person with X years in Y), and those qualifications certify they know what they are doing.

Which in my opinion brings up the certification argument, but I, as I’m sure others, have found that most certifications mean nothing in regards to whether someone can actually do the job. Is the key component missing in hiring in our industry a vastly more rigorous/extensive/difficult testing and certification protocol?

I'm an engineer in physical disciplines, and I don't think it's a certification issue. It's more that "engineering" in the software world is conflated with craftsmanship. The word "engineering" as used in relation to software is basically interchangeable with "technical work". Let me hasten to add, this is not intended to be pejorative or haughty, rather an observation.

Whether it's bridges, aircraft, or other complex engineered systems outside of software, there is a fundamental commitment to correctness, as assessed against physical principles such as conservation laws and both high-level and low-level verification criteria. Some software orgs (I think mostly in safety-critical systems) apply this kind of rigor, but most software development does not. Just look at the pushback on this site for development systems that increase rigor at the expense of "development velocity", such as memory-safe languages, powerful type systems, formal methods, etc. No civil engineer is going to be OK "shipping" their product without stress analyses, material property knockdowns, and the like.

My suspicion is that the root of this lies in an inability to systematically reason about software the same way we can about physical systems today. The tools and principles for correct-by-construction software are more primitive and harder to use than their equivalents in the physical realm. Also, lack of rigor in software still gets you a lot of powerful software and society seems remarkably tolerant of shitty software outcomes.

All software doesn't need to be engineered, the same way no one engineers a hand-crafted piece of furniture. But our software infrastructure ought to be, and it surely is not today. It will be awesome when software engineering starts living up to its billing, as I hope it one day does.

Yes, and in other engineering disciplines this level of rigor does exist. https://en.wikipedia.org/wiki/Engineer_in_Training

which is why bridges and buildings certified by qualified engineers have never collapsed

Both types of structures have collapsed.

I'm pretty sure someguydave was being sarcastic.

indeed.

I don’t believe any particular test or qualification wards off mistakes

I guess, that person is just being sarcastic. However besides certification, qualification etc I think huge factor is cost of IT project is variable. And even worst failures are mainly delays, extra budget or moving work to different vendor/product. So companies can save money without endangering lives.

A bridge, even a fancy modern one, is pretty simple compared with the software running on a 90s-era computer, much less a modern computer. Fizzbuzz is probably about equivalent to building a simple engine out of a box of parts.

The problem is that we have built all our software on Unix. The Unix security model is based on an attack model where users have to be protected from each other on a terminal server. Code is implicitly trusted and exploits were an unknown unknown.

That security model is almost completely useless now. Terminal servers are an extreme edge case. Services implement their own security model between users. Special “Users” for services is just a hack that tries to use an inadequate system as much as possible.

The Unix security model is deeply baked in everywhere and it’s nearly impossible to tack on a security model fitting to todays requirements afterwards.

I agree with your points about the Unix threat model being designed around protecting users from each other, but for completeness I'd like to point out that while modern Linux/BSD systems ship with Discretionary Access Control (DAC) by default, SELinux implements a Mandatory Access Control (MAC) system which is much more fine grained. SELinux is not limited to the traditional Unix security model.

Short simplified example highlighting DAC/MAC differences: DAC asks "is user Alice allowed to read Bob's files?" while MAC asks "is the SMTP server (subject user/role) process allowed to read private keys (object type) of the HTTP server (object user/role)?"

And if you're really motivated (read: want to have fun diagnosing unexpected file permission issues), you can associate files and processes with different security categories and levels (MCS/MLS), implementing horizontal and vertical separation. For example, "is this software update service allowed to read confidential files owned by the accounting database or keys used to encrypt classified information?"

There are other MAC systems besides SELinux as well.

The point of the article is that actually using a fine-grained MAC model is inconsistent with the way we actually use our tools.

It requires deep understand of both the application and the model, which requires a lot of work.

Is Windows much better with its VMS legacy kernel? It has a complex permission model with ACLs and everything but that trades inadequate permissions for something like the SELinux problem. There are too many knobs. (For this discussion leave aside the backward compatibility baggage which is another issue.)

The only approach that I see as viable without rebuilding the entire compute universe is VMs, either the web browser (JS and WASM sandboxes) approach or the qubes type approach.

"We have an isolation problem. Let's use a VM!"

Congratulations, you have just introduced an L1TF attack surface, and the bad guys are now freely reading all physical memory.

You have traded an absolutely enormous attack surface of hundreds of syscalls and a whole system full of files and apps for a much narrower attack surface involving the CPU and whatever the hypervisor exposes which can just be virtio-style pseudo-devices. Besides if the CPU is vulnerable than regular apps can probably exploit it too.

Also I didn't say the best approach would necessarily be hardware VMs, though that is one option and is valuable to let you just plug and play existing applications unmodified. I personally think we need to get away from shipping all software as raw hardware-tied binaries in favor of something like WASM. Java and the .NET CLR had the right idea decades ago, but these were insufficiently versatile and too tied to just one language or platform instead of being a general purpose VM bytecode. WASM is heading in the right direction but doesn't seem to quite be there yet.

Only apps that actually need to use e.g. special CPU features or run tightly optimized ASM code should ship with native binaries or native modules/libraries for the high-performance parts, and that should be something you have to approve.

I hear a lot of, "Let's solve isolation with VMs," without taking into consideration that if you put two fully-patched kernels side-by-side, very few people (i.e., only those sitting on 0-days) have any idea how to exploit the workload running directly on the fully-patched kernel, while everyone knows how to exploit the (Intel) host when they have root/CAP_SYS_ADMIN in the VM and the host has HT enabled.

VMs are far from magic security sauce, and especially if you need HT on Intel hosts, you actively want to avoid using VMs as a "security boundary." You're far better off with the attack surface of a fully-patched OS image from a reputable vendor.

The problem I see with wasm as the sole basis for a secure app platform is that then a web browser can't be just another app on that platform, unless the performance of the browser's JS engine is hobbled or perhaps wasm gets some kind of JIT compilation support. So maybe the web browser engine needs to be the platform, as in Chromium OS, although even Chromium OS has now compromised that original purity with support for Android and desktop Linux VMs.

Using VMs like this means the VM itself becomes hard-core security-critical but if apps have stupid bugs their damage is quite limited.

It also means you can mitigate things like Spectre or RowHammer by just tweaking how the VM JIT compiles bytecode into machine code. You don't have to update apps or even the kernel per se (though the latter would be a good idea).

I agree with you, but if we look past the hack of dynamically creating new users per process, isn’t the security model “sufficient”? E.g. I believe android’s security is quite elegant with the different user per-process model, communicating only through IPC and with heavy use of SELinux (which is much more useful in this model).

Some of the problem is that historically we've built systems badly engineered for security.

Take for instance something like xscreensaver. Something in there needs to be setuid so that it can verify your password to unlock the screen. That something is fortunately a dedicated binary, and not every single screensaver, but still, that's bad. Writing that executable is a delicate thing. Get one of them wrong, and it's a glaring hole.

And /usr/libexec/xscreensaver/xscreensaver-auth is quite the thing. It links to a whole bunch of graphical libraries and talks to X11 to read your password, so that thing has a huge potential attack surface. Far more than seems comfortable.

What we should have instead is some sort of authentication service. A program confined by SELinux to only interact with a socket and the password database, speaking a simple but well designed protocol.

With that in place we'd have much simpler security. Policies get simpler because random stuff doesn't link to PAM anymore, and so doesn't end up touching a whole bunch of critical security related files. There's one thing on the system that deals with that, and it ends up with a small, understandable policy.

And the policy for the users is now expressed in terms of "Do we want this program to be able to validate user passwords?", which is also far simpler.

With that kind of redesign over time we could have much smaller and more understandable policies, and a more secure system as a result.

I get your overall point, but your example is not a good one: on Linux, at least, xscreensaver does not have a component that needs to be setuid root.

xscreensaver-auth uses PAM to authenticate, which (typically) runs a small program called unix_chkpwd, which is setgid shadow (not setuid root), as /etc/shadow is owned by root:shadow and is readable by group.

xscreensaver is one of the very few screen lockers that is reasonably secure, because the bit that locks the screen is pretty minimal and only links with a few fairly-low level X11 libraries, not a full-scale GUI toolkit.

> What we should have instead is some sort of authentication service. A program confined by SELinux to only interact with a socket and the password database, speaking a simple but well designed protocol.

That sounds pretty overengineered. All you need is a small binary that is setgid shadow, that can take username/password on stdin, and exit 0 if auth succeeds, 1 if auth fails. But we already have that: unix_chkpwd.

Oops. Yeah, serves me right for writing from memory, pretty sure it was setuid at some point in the past.

> That sounds pretty overengineered. All you need is a small binary that is setgid shadow, that can take username/password on stdin, and exit 0 if auth succeeds, 1 if auth fails. But we already have that: unix_chkpwd.

That's probably too limited actually. Because you can have a whole bunch of stuff in PAM: LDAP, OTP, Yubikeys, and all kinds of other fancy modules. Doesn't seem that unix_chkpwd handles any of that.

Also, I still think it has to be a network service, because the unix fork model offers lots of avenues for attack. The parent process gets to mess with a whole bunch of things before exec'ing anything, and that list keeps getting longer as the kernel adds features. It's not possible for the child to reliably defend itself against anything the parent might do.

Thus I think for security, a network socket is the best way as it doesn't allow the client process to manipulate the environment of the security service.

> That's probably too limited actually. Because you can have a whole bunch of stuff in PAM: LDAP, OTP, Yubikeys, and all kinds of other fancy modules. Doesn't seem that unix_chkpwd handles any of that.

Yes, you're right. unix_chkpwd doesn't handle any of that and, in fact, was never intended to handle any of that -- and that's the entire point!

The entire design of PAM is, well, to have separate "modules" that are "pluggable" depending on how you need to handle "authentication" -- LDAP, OTP, Yubikeys, etc.

That is, the pam_unix module (which uses unix_chkpwd) is used when you enter in your (local user account's) password. If you're using something else -- LDAP or NIS or whatever -- for user accounts (i.e., your "passwd" database) there are separate (PAM) modules for that!

> Also, I still think it has to be a network service, ...

No, it really doesn't and, besides, there are other alternatives that would be much better to use instead of a network socket (such as a local UNIX socket, for one).

I really try not to make such remarks here on HN, but in this case it does seem that you have a fundamental misunderstanding of just how this stuff works (which is almost certainly why you're comment has been so downvoted).

Well I have been playing around with Wayland + Sway the last month. I like it but there are some caveats. For example I end up modularly picking the tools I want to use with it. So by default I think there is swaylock and then there is waylock but a few times I got back to my seat and the locking application had crashed, exposing my session to my co-workers. That ain't the desired effect. I resorted to physlock which swaps to tty and allows to disable things like echoing of kernel messages and sysrq. It has yet to crash. Also, all of this relies on PAM and according to the OpenBSD devs PAM is a mess.

What's the difference between a service that you send a string to and get a string back and a binary that you execute with a string argument and it prints a string argument back? I quite like the latter, as you're free from keeping state and thus have a smaller attack surface or potential to leak resources. Of course one difference is the execution environment, with a systemd service you can have it set up exactly as it should be so no changes to LD_LIBRARY_PATH &c can poke holes. I wonder if its socket activation feature can be used like a sort of CGI server - for each connection on a socket, run the binary in a controlled environment and connect its stdin/out to the socket, with stderr going to a log file.

You said it yourself. LD_LIBRARY_PATH, and a myriad other knobs.

Think of all the stuff you can do: mess with filehandles, signal handlers, chroot, resource limits, seccomp, capabilities, program arguments... and more appear over time. You can't defend yourself from things that didn't exist at the time the code was written.

Polkit recently got exploited this way: https://blog.qualys.com/vulnerabilities-threat-research/2022...

Putting it on a socket is a good way to make sure the client has no control over the environment.

Yeah but that means we need to have a way of securely running binaries, not move a simple program to a service. Something like taking a snapshot of the system at various stages during init and have the binary start in a known context. This would also help a lot for desktop users - many programs need to start in a specific environment and having the option to configure per-user per-program launch environment would help a lot.

> The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl. […] Who is responsible that it is air tight? Well, uh, nobody, really.

The people who designed the container/sarcophagus system. If it's not secure don't sell it as secure: see the difference between Linux containers and FreeBSD jails or Solaris zones.

> You can't even blame the applications for that. Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call? Library and framework functions tend to not document that properly.

As a developer / software "engineer" it is your responsibility to know how the components you choose work and how much you can rely on them. When a structural engineer selects certain types of steel or concrete to be used in the construction of a bridge, it is their responsibility to know the characteristics of the raw materials.

When you ship a software product, enable SELinux or AppArmor and audit what it does: then ship an SELinux/AppArmor profiles.

See also OpenBSD's pledge(2) framework where the developer embeds system calls in their (C) code to promise to only access certain OS features:

* https://awesomekling.github.io/pledge-and-unveil-in-Serenity...

* https://man.openbsd.org/pledge.2

> it is your responsibility to know how the components you choose work and how much you can rely on them

Unfortunately that is not always your choice and arguing against it most commonly costs you more than you like. My experience is quite often timelines get underestimated massively with software so you end up doing what you can with the resources given. If you pull this into the bridge metafore the bridge still stands but can collapse any time and as long as it doesn't collapse during rush hour and can be rebuilt before the next morning it's all fine.

Yep, this is accurate. People making decisions often value “works”/“ships” more than “secure” and the time allocations reflect it.

I wish it wasn’t the case, but it is.

It's pragmatism. If something doesn't work, it doesn't matter how secure it is.

If something isn't secure, it doesn't work. At that point, you would've been better off shipping a wireframe.

A door with a broken lock still provides value. An online PDF converter on a website without SSL still provides value. I'm not saying security doesn't matter or that it shouldn't be a priority, but things don't need to be perfect to be useful. It makes sense to focus more on functionality than security in many areas. I host a bunch of web apps behind a firewall that aren't secure enough to expose to the public Internet. Even commercial ones.

Unfortunately, that's exactly the problem. If it didn't provide value, no one would use it and its lack of security would be a moot point.

The PDF converter is a great example. It's functional, people use it - to transfer potentially highly sensitive documents over an insecure connection. At that point, it provides value in the same way that a Nigerian prince provides banking services.

For you, no risk seems acceptable when it comes to security. That is generally not the case for most people, who will accept some level of risk (ie an insecure program) if it provides value to them. That's the whole point of risk management.

Pardon me for being pithy. We've all heard the adage about perfect security being a logical impossibility. I will say instead that if a tool doesn't have security proportionate to its risk factor integrated in its design, it doesn't work.

There are lots of decision points. Some people are deciding what to sell; others are deciding what to purchase. An organization that de-prioritizes security in what they produce could be punished by organizations that prioritize security in what they consume. In this way, decisions are shared to some extent. Consumers might be misled over the short-term, but at some point they become complicit.

If a datacenter continues to purchase a model of generator after five high-profile failures of that model, they can argue all they want that it is the only model on the market that meets their needs. But clearly uptime is not their biggest consideration.

> When you ship a software product, enable SELinux or AppArmor and audit what it does: then ship an SELinux/AppArmor profiles.

It's not just sysamins -- developers don't understand SELinux, either.

As a dev, I find least-privilege sandboxing (e.g., look at how sshd is structured) and capabilities (e.g. FreeBSD capsicum) somewhat non-intuitive, but at least tractable to understand. SELinux is a blackhole.

>> The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl. […] Who is responsible that it is air tight? Well, uh, nobody, really.

> The people who designed the container/sarcophagus system.

Original comment misses two nuances with the gripe.

1. Container creators and software creators are not the same people. So they can (and should!) have different goals and priorities. For containers, security and reliability. For software, make it work.

2. There is a nuanced, different interface between software-system and container-system, created by encapsulating a system-like environment inside the container. Much of what software does can be encapsulated. Which allows a more controlled (and much smaller) subset to exit the container.

> 1. Container creators and software creators are not the same people. So they can (and should!) have different goals and priorities. For containers, security and reliability. For software, make it work.

If the container system designers did their job properly it won't matter what the software does inside of it, as breakouts shouldn't be possible.

It has been a few years since I looked, but last time I checked there hasn't been a vulnerability in FreeBSD's jails code that allowed someone to break out. The closest to it was an issue with devfs that allowed 'tunnelling' out, but not with the actual Jails code itself:

* https://www.freebsd.org/security/advisories/FreeBSD-SA-14:07...

Meanwhile, with Docker (including CVE-2019-5736, which allows for overwriting runc on the host system):

* https://www.cvedetails.com/vulnerability-list/vendor_id-1353...

> 2. There is a nuanced, different interface between software-system and container-system

One keeps things inside and other other often doesn't?

One side has a huge footprint to secure (software-system), because it's every way software could ever need to interact with a local system.

The other can expose a much smaller one (container-system), because it only needs to include things one would have gone off-local-machine for (i.e. networking).

So from a security boundary, containers are basically a machine-internal firewall between programs and the host.

> So from a security boundary, containers are basically a machine-internal firewall between programs and the host.

Which is only useful if they actually provide security.

You're missing my point.

It's effectively impossible to ever provide security at the program-system interface, due to the surface area. (I.e. the SELinux problem)

The very concept of having a container (that internally simulates a system) creates fundamentally different opportunities that allow both (programs that work) and (security).

Whinging about whether or not current containers do a good or bad job of it is a less interesting, short term quibble.

Ideally all apps & libraries would ship their own selinux policies with a common framework for combining them & base layer for privilege sets (eg, "let me open any file the current user can open" or whatever). If that was the case then your concern wouldn't be an issue. You'd just say "I use libc sockets" and you'd inherit whatever file path permissions are necessary for that to work, as defined & exported by the libc in question.

But that's not a thing. So distros are attempting to add it later themselves, which is a disaster.

> Wouldn't it be great if the man page of getaddrinfo mentioned those (mine only mentions gai.conf)

That's a huge part of the problem. Was looking for a monograph to pass on my successor that wasn't so familiar with linux - modern linux - as in systemd, docker, nssswitch, pam_homed, network-manager is penetrable but more often than not I'm just looking at the c source in github and I am in my 30ies, starting in my teens running linux nonstop and I'm still throwing up my hands every other day...

Add time constraints and pressure to deliver (don't waste your time understanding this, just do xyz) and here we are - on the other hand it's not okay that you have to devote years of trail and error to get a comprehensive undestanding of the system.

I guess a nuclear power plant has at least a training plan and complete reference book - that still needs to be read and grokked and trained upon but modern linux is often just undestandable by reading the source if you hit a problem - for some projects like things in the freedesktop ecosystem and partly systemd even that is kind of difficult because i.e. this bug https://github.com/systemd/systemd/issues/19118 explains the problem - there is no overview documentation, no clear way to look up what's happening, not even a good way to introspect and it's several components that interact with each other fail subtly. I've chosen this one because I've also hit it, not because I want to blame systemd, which is not so bad there are much worse things out there but it's part of a trend to introduce complexity - not sure what is necessary complexity and what is unnecessary - went to uni in compsci without ever someone slapping out of the tar pit (http://curtclifton.net/papers/MoseleyMarks06a.pdf) in my face and maybe that's part of the problem.

> there is no overview documentation,

Documentation on Linux has always been shit.

Huh? Compared to what exactly? Man-pages compared to MS provided info are fairly exhaustive..

Now if we are talking of a documentation of a random project on github..

The BSDs. When I admined Solaris there was also very good docs.

The man depend on the utility. Except for the GNU stuff, where the man pages often tell you to look at the info pages.

Growing up on FBSD, it was quite the shock when I started working with RHEL/Debian. The qualitative difference between the manpages was disturbing.

Let's say you want to build a good, secure application. How do you know what files your application will try to open? What syscalls it wants to call?

This almost looks like a hint to OpenBSD's pledge and unveil system calls.

I'm just a hobbyist, but regarding

we all run software we don't understand

what I like (again) in OpenBSD is that I feel I can largely understand/control what's happening in my system, it quite fits in my head.

Just my 2 cents.

Exactly. ‘ps ax’ on a fresh OpenBSD install gives you about 10-12 processes, each of whose purpose is either obvious or easy to discover.

Try doing the same thing on macOS or Ubuntu. Last time I looked carefully at the former I discovered, for example, daemons for classrooms (??) It’s a free-for-all.

You can use strace to get this information. Linux also offers seccomp and landlock which are very similar to pledge and unveil.

I would like to emphatically point out that: no, you can't. You can't use strace to get this information.

Let's say you want to use seccomp to whitelist allowed syscalls. Your code opens a file, uses stat to get the file size, then mallocs that many bytes and reads the file contents into the buffer.

Trivial program, right?

glibc will turn open into openat, stat into statx, malloc can become either nothing, or sbrk, or mmap, and maybe also munmap. If you hit an error, it will also call write to output an error message to stderr.

dietlibc will do open, stat, mmap.

This is also libc version dependent!

But can you at least use strace to know which files will be opened? No, not even that! Because the code may open some files only under certain circumstances. For example, localtime will open /etc/localtime -- but then it will cache the result for a while. /etc/localtime may be a symlink. If you construct a container you would also need the thing it points to.

What if you use a malloc that has explicit hugepage support?

Also note that seccomp and landlock are very much different from pledge and unveil. I am particularly appalled by landlock as compared to unveil. Go look up the landlock API if you don't believe me, and compare it to the unveil man page. If you thought the hoops seccomp makes you jump through are ridiculous, you haven't seen anything yet.

OpenBSD is not all gold either. pledge is not transitive. If your process pledges something but is allowed to execve something else, the process you exec is not bound by your pledge.

There is MUCH room for improvement all around.

However, the persisting idea that you can use a "training mode" or observe via strace to construct a whitelist IS WRONG and dangerously so. Unless you have 100% test coverage (and if you had, why do you still need a sandbox then?) you will miss error handling and optional code paths that you didn't enable in the configuration, and handling code for circumstances you didn't trigger and didn't foresee.

> Unless you have 100% test coverage (and if you had, why do you still need a sandbox then?)

100% test coverage does not guarantee absence of vulnerabilities in any way.

> you will miss error handling and optional code paths that you didn't enable in the configuration, and handling code for circumstances you didn't trigger and didn't foresee.

Having sandboxed many things, this is plain false.

Normal applications and daemons have no reason to call reboot() or stime() and so on in any obscure code path.

If they do, the sandbox should stop them and that's a feature and not a bug.

Their point about test coverage is that you don't know what syscalls will be made, or with what arguments, without running the program with enough coverage to find out.

This is probably the most well known issue with sandboxing - maintaining the sandbox. It's especially hard with seccomp, because you could upgrade your distro, or a dependency, and suddenly you're making a different system call.

>Also note that seccomp and landlock are very much different from pledge and unveil.

No, this is incorrect. They fundamentally do the same things. You're trying to say they're different because the API is different but that's completely missing the point. If you really prefer the API of them then you can go and use one of the emulations of pledge and unveil that have been built on top of seccomp and landock. They work because there isn't really anything special going on there.

Docker actually comes with a seccomp and an apparmor config that bans many things by default. while it is absolutely not noticeable unless you are trying something like docker in docker. There are so many syscall that shouldn't even be relevant to normal programs.

That's why I explicitly mention "whitelisting".

If you just want to ban some obscure syscalls and call it a day, you can do that. It will probably even be helpful to some degree.

I personally think our aspirations should be higher than "let's ban ptrace(2)".

It's actually a bit more than that. The syscall docker allowed isn't really fixed. It is affected by what linux capabilities the container had granted. Like: if you whitelist the container about CAP_SYS_PTRACE, you probably also want ptrace(2) to be whitelisted. Instead of a all or nothing/your program will still break even cap added model.

Then decompile the binary, bust out gdb, and do some forensics. Or realize you're about to embark on a multi-decade mission Learn It All(TM) which equates to maybe learning the quirks of enough niches of the programming community to maybe not be surprised by what you find... Eh.. 40% of the time.

How is the libc issue specific to linux? OpenBSD has a libc too.

OpenBSD doesn't support syscalls from alternative libcs, so you reliably get consistent syscall behavior.

But I believe GP is mostly talking about pledge(2), a pretty easy way to implement common sets of seccomp-like restrictions, and unveil(2), an easy way to limit path visibility. These are OpenBSD security features that Linux does not have direct equivalents of.

> The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand.

I don't think these statements are meaningfully different. "too complicated" implies "...for humans to manage". Maybe that's sort of your point?

Agreed. It's too burdensome for software developers to understand exactly what syscalls their program needs to make and the security implications of permitting those syscalls. It also doesn't help that Linux naming conventions and concepts are very counterintuitive (yes, dear Kernel Hacker, I'm sure they're very intuitive to you, but we lowly mortals struggle).

And unfortunately the SELinux policies are tightly coupled to the application such that you can't make SELinux policies the purview of a dedicated sysadmin expert and leave the appdev to the development teams. They have to collaborate which is swimming against the current of Conway's Law or else you make SELinux policies the responsibility of appdev and suffer their lack of expertise.

We had similar problems with operations in general, but containers largely solved this problem by allowing sysadmins to focus on a platform while developers focus on developing and operating the application. We need something similar for security. This is probably a rephrasing of your "sarcophagus" point?

> Obscure files like /etc/localtime, /etc/resolv.conf, /etc/ld.so.conf, /dev/zero ... how can you expect devs to build well documented and well sandboxable applications if they don't know which files their library functions will open?

Who the fuck invented that convention that fine-grained permissions must be file-based? It's insane. No, no developer will anticipate that he needs to read /etc/nsswitch.conf. No developer should. A developer should anticipate that the software needs permission to connect on network hosts.

As much as it is too granular, file based permissions aren't fine-grained enough. Asking to connect on random hosts is absurdly wide, most programs only need to connect in to a few of them, or connect to an user-supplied host (that can be a permission by itself).

Anyway, yes, the manpages should interlink better. What is a different issue.

> The whole container movement can be seen as putting the apps in a sarcophagus like Chernobyl

Reminder: containers are not meant to be security tools.

Fine-grained sandboxing (e.g. seccomp) is. And it can be layered upon OS-level VMs.

Additionally, bundling tons of stuff together with an application (like docker but also flatpak do) is not good for security. Same for static linking. They all increase the workload of updating vulnerable dependencies.

How does static linking increase workload?

By multiplying the number of potentially-vulnerable copies of a library by the number of ways to detect the presence of said library and by the number of ways to upgrade said libraries when found.

Plus, it makes updating an application way more difficult: you need:

- the distribution to recompile and test all binaries using a vulnerable dependency [x]

- users to download and update all affected binaries

compared to:

- the distribution patching a single package

- users updating that package

Imagine if OpenSSL was statically linked: even the smallest fix would require users to update half of the OS. And that would discourage both maintainer and users from doing regular updates.

[x] And this is assuming that thousands of automated rebuilds are possible. In reality many statically linked languages encourage locking the versions of build dependencies, making it impossible to do just one backport of a patch.

Why would you need to rebuild everywhere? Couldn’t you rebuild once in a trusted place and distribute signed binaries?

Distributions have their fleet of buildbots to do the latter.

Exactly! So what difference does it make if there is one patched package or 1,000?

You also forgot to divide by the number of binaries that were compiled with non-vulnerable versions of the library.

What if your vendor was responsible for the fix?

Then it increases the workload of said vendor?

I've been recently thinking along similar lines about libraries. In web dev, it's common practice when you have a problem to first look for a library that solves it for you.

I get the "don't reinvent the wheel" sentiment but, I think we take it too far sometimes. I've been looking at the source code for some dependencies at work lately, and many of them actually don't hold up to our own code quality standards. Kind of subjective, yes, but many of our dependencies would probably fail code review if actually reviewed by us.

Then when there is a bug in a dependency, nobody actually understands how it works and the code is often tucked away and not easily changed.

when it comes to JS libraries I'm usually a lot more interested in how good their testing is. Code quality can be very subjective in some areas. But if the dang library doesn't have a lot of good testing then I can't really trust any updates from it :/

Coming from marine and aerospace contracting, I'm always shocked by the cavalier attitude The Rest Of The Software World has about pulling in third party dependencies. "I have a problem to solve. Let's Google for: [problem] [language]. Aha! This library on GitHub has 12 bajillion stars! Pull that one down, build it, and YOLO!" No in-depth look into what other things the dependency does, how it increases the attack surface, what user data it gobbles up, what license it uses, what is the update cadence and how often do we need to pull from upstream, what is the contingency plan if it ends up not being suitable, nothing. Just git push and close that JIRA ticket!

Inadvertant bugs are the last thing I worry about in the javascript (node) ecosystem.

I'm more concerned with the fact that running anything requires transitive trust to hundreds or thousands of projects in node_modules, any of which can run scripts while being installed.

How many people out of a thousand would you trust running commands at your terminal?

/etc/resolv.conf is a funny one.

Used to be, you could update it manually. Then NetworkManager (or whatever its successor is) came along and would change it back on you. Oh well, you could still edit it for a quick test.

Then I set up a Wireguard tunnel. And it changed /etc/resolv.conf. Oh, but an entry is missing, I'll just add that and test. Nope, read-only. But I'm root! What gives?

Turns out the wg-quick script mounts a file on top of /etc/resolv.conf! I didn't even know that was possible until I saw it. Nobody messes with wg-quick's resolv.conf, and that's final! Until some other tool ups the ante and gets code to undo that.

I have, in moments of desperation, `chattr +i /etc/resolv.conf` before. I understand the rationale behind managing that with a more robust service, but in my experience that's much more prone to the black-box effect.

>>The problem is not so much that selinux is too complicated (it is as complicated as it needs to be), but that we all run software we don't understand

There is truth to this, and it nails the fundamental asymmetry of the bad guys vs good guys in the security war. To program selinux you need to understand the software you run at the syscall level. and potentially have a deep understanding of its file usage, particularly if it’s doing ipc.

In general, I think that is a good goal. More understanding is more understanding and that is Good. In practice? I equate it to the problem of writing secure and robust code in C, I don’t know how good you have to be todo it and I basically assume that anyone who says they do is full of shit. I have contributed to the Linux kernel, I have decades of UNIX and specifically Linux experience as a software engineer, and I am still surprised when I fire up strace from time to time. You look at something like the recent Dirty Pipe bug, and I have a difficult time accepting that many people can fully grasp it all. The cost of a fairly simple system interface is all the subtlety and edge cases.

I view the container movement as a way of managing interactions. It’s really hard to manage dependencies in such a heterogenous systems. Containers simplify it all, at the cost of owning ones full dep tree. But with a good release process, one can keep on top of that and put less effort into the combinatorial growth when interacting with all the other dependencies other programs require.

containerization is less about security these days, and is being used to work around the broken package/dependency management situation on linux which is caused by the wild west of libraries that make little effort to assure any kind of forward/backwards compatibility. Given the fact that distro package maintainers cant maintain 30x different versions of any given package to satisfy all the applications stuck on a particular library version, means that larger applications which can't be constantly churning their code bases to replace foo(x,y) with foo_bar(y,x) are stuck shipping their own pinned version of a library.

So, I don't think anyone seriously thought that containers provided significantly more security than simply running an application and trusting the kernel syscall and filesystem permissions were bulletproof. More software layers are unlikely to improve security.

It's the forward march of progress. As things get more complex they also get more fragile and inscrutable, and harder to solve problems for.

Containers are a reaction to the flaws in the system. Fixing the root cause of those flaws is hard, because they're systemic. We would need to re-design everything to fix those flaws. Containers are a stop-gap measure that makes things easier to deal with.

The thing that I've learned about systems is they are all crap, but humans constantly munge them to keep them working. We don't end up with the best systems, we end up with systems easiest to munge. Every once in a while, a spasm of collective organization redesigns the system, and then we keep going with that one. It's evolution. So however crap our current system is, eventually it will change, probably for the better. It just takes a really long time.

This seems like something that WebAssembly (misnomer) is set to resolve, to some degree. Sandboxing capabilities is easy because all of the wasm module imports must be provided by the process that initializes the module, so you can view imports as a kind of list of "requests for capabilities".

I think that's leaning even more into the parents point.

We make a sandbox (in this case a very inefficient one) -- we assume the sandbox is the only way to deploy software, but mostly it's not because there will be times you need more than the sandbox or the software doesn't work, and then you're running "privileged" WASM routines.

The sandbox itself might even require some kind of leaky behaviour, such as accessing files or shared memory.

The wheel keeps turning. Some new sandbox will come along.

WASM will probably be always slower than native but call it very inefficient is unjustified imho considering the compiler needs to be fast enough to run in browser from what I have seen its around ~1.5x slower than native code with there still being many low hanging fruits to be optimized.

1.5x slower for a weak sandbox that's already hardware compromised (spectre) with no plans to ever be fixed, along with a questionable portability story (cool the instructions itself are portable, but there's no APIs. Which are, and always have been, the actual portability issue) does not exactly sound compelling. It makes sense on the web where the browser is already providing API portability & permission sandboxing, and all that's necessary is actually just instruction portability and protection against eg memory corruption (actual security sandboxing still being done via process isolation), but beyond that? Like desktop apps? Pretty much entirely useless.

And it doesn't address any of the issues here anyway. As soon as you bind libc to it (or any other system library), which isn't exactly unlikely, you're right back into the problem of not knowing what the selinux policy should be. And you still want selinux policies since WASM is only a memory sandbox, not a permission sandbox.

prefer not to get into a debate about the efficiencies of x vs y but sufficed to say: Computers are only getting negligibly faster over the last 10 years and our software has gotten overall quite considerably slower.

A runtime overhead that slows everything down to 1.5x, even with low hanging fruits is only going to accelerate this issue.

From a consumer perspective: We all act as if everyone has 8-16G of ram, because that's what we're used to, but the reality is that the majority of people have 2-4G of ram, even these days. That's not counting the anemic CPUs that are often inside awful cooling solutions.

From a server perspective: we outsource our Ops to cloud providers and pay a significant premium for computational speed, which means things like runtime overheads have direct costs.

The reason I called it inefficient is because it's not adding anything we don't have, it's just "another layer" with a large runtime overhead.

And, anyway, I'm mainly referring to this talk: https://www.destroyallsoftware.com/talks/the-birth-and-death...

That talk makes the point that replacing the swiss cheese hardware security models with a software one can shift your performance. You run 1.5x slower, but you also run 1.5x faster by avoiding all the hardware checks. On the whole, you can theoretically run about the same speed, but with better security.

Even better, if WASM takes over the world, you can bake parts of it into hardware and remove a lot of other bottlenecks.

> and then you're running "privileged" WASM routines

That's pretty much where snap ended at, isn't it?

Why would a completely new compilation target be needed for that when “old” sandboxes can work just fine by hijacking syscalls? Sure, memory safety is somewhat improved by Wasm, but valgrind and the like are effectively giving you the same thing.

Wasm addresses memory and execution isolation from the rest of the system, however anything running in it maintains it's exploitable logic internally. If the application needs to read/write files, and someone exploits that logic, wasm won't help. The only thing that could is limiting the filesystem access in the first place and building secure software to begin with.

> "The problem is not so much that selinux is too complicated (it is as complicated as it needs to be)..."

Completely disagree, if the target users are advising each other to disable it... then the tool is definitely more complicated than it needs to be.

Using tools that don't hide any complexity are very painful to use. It feels like the creator doesn't care about the user and put no thought into the display of information or workflows of the tool.

Solving a complex problem with a perceived complex tool is relatively easy... don't hide any complexity.

Solving a complex problem with a perceived simple tool is difficult... hiding complexity at the right time, revealing functionality it based on the user's intentions and experience is not easy but greatly appreciated.

if the target users are advising each other to disable it... then the tool is definitely more complicated than it needs to be.

This does not follow. I you ask a 5-year to repaint the Sistine Chapel and the result is somehow less than stellar, are you similarly going to blame the paintbrushes used? There exists no situation where the task at hand is too complicated for the user?

A better version of your example would be a 5-year-old that cannot open the packaging to get to the pieces of the paintbrush that need to be assembled, in order to paint his first stroke... that's the (arguably) unneeded complexity of SELinux

Uhh isn't this precisely what the whole snap/flatpak movement is trying to fix?

I can talk about snaps, but at least they are offering a specific permission model which is usable for users to understand. i.e. Via the interfaces, I can know software x accesses my password-manager, home directory, camera etc... I can disconnect access to the given permission and have it enforced via the kernel and apparmor.

The applications themselves bundle only libraries that themselves are sandboxed/snapped.

Yeah, I came to comment on this:

>> It doesn’t help that no one wants to be told: “stop what you’re doing, spend a week to learn this other thing, make a tiny change, and then return to your real job”. They’re looking for solutions — preferably quick ones — and not a week’s worth of study.

If you're a system administrator running SeLinux why dont you already know SeLinux? It's not like some obscure command, it's a big part of the infrastructure you're running.

Pour one out for the Machine God.

From the moment I understood the weakness of my flesh, it disgusted me.

SELinux is so ridiculous.

If you want a real security on linux where root is not God !

Go to this site.

https://www.rsbac.org/

It's little difficult to implement (by kernel customisation) but there is a learning mode to secure all Linux structure

Be "root" is not be "God" after implementation. You will must ask to Security Officer (SecOff)

You can speak of "Evaluation Assurance Level" with this security solution and push SELinux into a trash.

root is not God under SELinux either. root processes are also confined.

Indeed, that's possible and a good idea - as long as you're not talking about user shells [1] running as confined processes. It turns out that having an unconfined root shell is actually not a bad idea unless you're interested in extreme levels of security. My experience is that confining user shells is a significant hassle.

Running daemon processes etc. as confined root on the other hand is a good idea, but if you're going to the trouble of confining a root process you might as well just run it as a normal user instead if that's possible.

[1] Meaning shells used by a real person attached to ssh.

Reply on news.ycombinator.com