Hacker News Reader

Comment by tluyben2 | Hacker News Reader

Until well into the 2000s we deployed all our services as chroots. I even sold a server side product successfully which was distributed as chroot so it would install almost on any Linux distro without having to care about dependencies. It worked really well and felt quite a bit more accessible and lighter than docker. We just had a script which basically did:

    wget https://bla/image.tgz
    tar xzf image.tgz
    rm image.tgz
    chroot image/start

and the product would be installed and up & running. A lot of clients thought it was magic. No Apache install? No MySQL install?? It works on Fedora & Centos & Debian & Ubuntu & ... ?

I use this currently still a lot to run new software on unsupported hardware/kernels. For instance, I run a lot of new (Ubuntu) software on my OpenPandora under the default OS on that device; I have an SD with a chroot with Ubuntu and things like dotnet core. Runs fine. Gives devices like that a second youth.

View on HN

Docker has mostly succeeded in making this simple concept complicated. A "container image" is just a tape archive of some files. The runtime is just like your chroot, but with namespaces and cgroups. That's about it.

On top of all this, they designed a new programming language / build system for making those tar files. They invented a container registry so that instead of just "downloading a file", you have to use their particular protocol to download a file. They have their own version of systemd so you can see what programs are running. They tacked on some very complicated networking stuff. They tried to build a couple of container orchestration frameworks on top of all this. I think all the "extra stuff" is where people run into problems, and it's isn't even the core of the value it provides.

People have realized that it's all a little much and the good bits have been distilled into their own projects. Many build systems can just generate an OCI registry file directly (remember, it's just a tar file, and you don't need all that 1970s JCL stuff to make a tar file). Various runtimes (containerd/CRI-O) exist for running containers. And there are orchestration layers on top, for actually managing sets of containers, network rules, etc. (ECR, Kubernetes, etc.).

(Worth noting is that Docker itself does use all these spin-offs. It uses containerd as the runtime, it uses buildkit for building containers now, etc. They even ship a Kubernetes distribution, at least on Mac and Windows.)

I never thought about that. Is it really just that? Then how come Docker is this huge innovation that conquered the world and nobody can live without it anymore?

The innovation was the layered distribution system. Suddenly it was more efficient to distribute images to hundreds or thousands of hosts

It was the right UX at the right time.

Personally, I preferred my home brew “Kube” that would ssh updates to cgroups from a central control server.

Which is basically the Chef model, and Salt model of clients pulling desired state.

Docker & Kube aren’t novel and weren’t as specific things. Tons of home brew options operating in roughly the same style existed around the net.

Their adoption in industry, use at scale, simply gives them a grander appearance of somehow being truly novel

So the explanation for that is in how moderately sized companies work. There are system administrators, and there are developers. The developer writes some software, tells the sysadmin what environment it needs to run, and the sysadmin builds and maintains that environment.

Docker allows the programmer to write more code to build and maintain their environment, cutting the system administrator out of the loop; the sysadmin just has to provide a server able to run generic Docker containers, and doesn’t have to care about what’s in them.

Basically it’s a bit of a power grab, and a bit of developers being upset about slow processes in their companies (“why does it take two weeks to deploy a new version of this dependency”) and finding a workaround. Then the rest of the world started cargo-culting them.

I think that's a pretty unfair assessment. It is likely that the underlying problem was that no Linux distribution had all the packages people needed. You need the latest version of Python, and a bunch of modules, and MySQL, and Nginx, and node.js, and... and that never happened. Debian stable had something from 6 years ago. Debian unstable didn't boot. Every time you got a new machine or new developer, you would have to painstakingly manually install all those packages on top of your OS base image, probably breaking other parts of the OS as you did that.

Docker lets you build all the dependencies once, and then run it anywhere. You can run your container on your Mac in a (managed!) VM. You can send it to Amazon to run on a random EC2 instance that you don't have to manually update. Or you can just run it on your normal server, without affecting anything else on that server.

It isn't about developers wanting to write more code. It's about developers getting annoyed with the difficulty of installing software.

The point is that the developers were not the people installing software in production in the environments that Docker gained popularity in.

I agree with the initial assertion as to how companies work when they start to grow to medium or larger. The rest, though, feels an awful lot like projection on the parent's part; someone's an insecure admin or programmer reveling in their superiority at a moderately sized company -- not meant to be an ad hominim but you're really specific there, buddy.

The enterprise is about supportability, scalability, and ease of maintenance. Officially support software and common tools/platforms/SOPs are a must unless we're building something brand new from scratch. DIY charoot jails may work just fine... until the two admins who really know it go on vacation or quit or get fired for mining BTC on company gear. Now Johnny-New-Hire needs to unravel it, and it may not be documented well, or even work that well without an admin kicking it occasionally.

Were I to put a job advert out for DevOps roles needing Docker I could usually get someone with a base level of competency, and get them reasonably quickly. Likewise if I need to outsource this to contractors or even offshore budget IT support it's a lot easier to find a Docker guy then it is running someone through whiteboard exercises to figure out how much they know about jails and makefiles. And if we're way out of our league with issues there is Enterprise Docker which is expensive but gets us official support from Marentis (or whoever owns them now), which may not be justified in terms of cost, but absolutely makes management sleep easier knowing we can escalate things if we can't fix it internally.

The parent explained it as if it wasn't novel but he included all the reasons why Docker became what it is.

Basically, "it's just chroot with [tons of useful features]"

It's also an ecosystem. The biggest innovation for me is being able to browse the docker hub and get official distributions to start from for basically anything out there.

Nowadays, creating a raw chroot is far from simple. Last time I tried, I had to handle the special mount points like /proc and /sys. I can't remember the details, but I believe there were other problems.

In my experience, systemd-nspawn provides a nice alternative. It's mainly a chroot that abstracts the low-level details, and it fixes some security concerns (with cgroups, etc) and adds some nice features: e.g. networking, or `systemctl status` from the host that will display the sub-tree for the processes of the sub-host.

Last time I tried to recover an ubuntu system by booting from a live sub, mounting the drive and chrooting into the target device i was surprised to find out that systemd refuses to work if it detects being run in a chroot.

I was left puzzled.

If you're talking about what happens when you mount the root filesystem, chroot into it, and run systemctl then you're essentially asking to break out of the chroot and communicate to the service manager outside the chroot which almost always doesn't make sense.

The only time this is reasonable is when a container needs to manage services on the host a la what Portainer does for Docker.

You have a lot of options for accomplishing what you want.

* To run a service with a given root filesystem add RootFilesystem=/path to your unit file.

* Run a service from an image of a root filesystem use systemd portable services.

* To run your service using the init system in your root filesystem (could be systemd but doesn't have to be) use systemd-nspawn --boot.

you know, I just wanted to run services from within the chroot, like you could do in the pre-systemd era.

I mean, as long as you have /proc, /sys and etc bind-mounted you should be okay, right ?

Right, but what you're describing as running a pre-systemd "service" is really just "run a shell script that sources some environment files and then double forks".

Running a systemd service means talking to the /org/freedesktop/systemd1 object on the system bus typically listening on /run/dbus/system_bus_socket and asking it to start your service. This is all that systemctl really does.

    mkdir ./rootfs
    curl -O  https://cloud-images.ubuntu.com/minimal/releases/bionic/release/ubuntu-18.04-minimal-cloudimg-amd64-root.tar.xz | tar -xJPC ./rootfs
    mount -o bind /proc ./rootfs/proc
    mount -o bind /sys ./rootfs/sys
    mkdir ./rootfs/run/dbus
    mkdir ./rootfs/run/systemd
    mount -o bind /run/dbus ./rootfs/dbus
    mount -o bind /run/systemd ./rootfs/systemd
    chroot ./rootfs bash
    # See that it works just by sending dbus messages.
    dbus-send --system --print-reply --type=method_call --dest=org.freedesktop.systemd1 /org/freedesktop/systemd1 org.freedesktop.systemd1.Manager.ListUnits
    # Now do the same with systemctl.
    export SYSTEMD_IGNORE_CHROOT=true
    systemctl list-units

THIS IS ALMOST CERTAINLY NOT WHAT YOU WANT. You're just talking to the host systemd. You won't see any of the services in your chroot since how could systemd know about them? Your chrooted root is also now just root on the host. Just use systemd-nspawn.

    # undo all the bind mount junk.
    chroot ./rootfs bash
    useradd ubuntu -G sudo
    passwd ubuntu
    exit
    cd ./rootfs
    systemd-nspawn -x --private-network --boot
    # login
    sudo systemctl list-units

This was very informative, thank you.

Nothing a shell script wrapper can't setup for you -- arch linux provides an arch-chroot which handles exactly that.

Most folks have two requirements:

1. Put files on a device

2. Start a process

The scale at which containerised scheduling of this outcome saves more time than it consumes is much higher than many have been led to believe. I measure it in sysadmin FTE. If you have >0.5 FTE dedicated to managing deployments then there may be a payoff, because (amortized over a reasonable period e.g. 12 months) it takes at minimum half an FTE to setup, manage, and maintain a PaaS properly.

I've become accustomed to folks claiming things like, "I spend 5 minutes a week managing my kubernetes cluster". Then it turns out it took them a solid month to get it going properly, two weeks to containerise their application reliabily, and then next quarter they're stuck for days on some weird networking or logging issue, and nothing else progresses. Basically, they either don't value their time, or they're misjudging how much they've actually allocated.

It's often the same people who boast about how cheap their bare-metal servers are vs public cloud instances.

I used that for build machines. Had a btrfs filesystem on a loopback device I could just download as needed to seed the agents and then make a subvolume to work in. When all was done, I could just drop it. A lot of was automated with "schroot" and definitely faster than spawning a new instance of a VM every time!

Yeah, it's very nice (and fast) for creating all kinds of build and test environments without messing up the machine. Language like Python recommend virtual envs like that, but this is more consistent and you can actually install whatever around Python as well without messing up the parent system.

A similar approach is used by distributions such as Debian/Fedora/Ubuntu/RHEL.

In these distributions, you have tools ("pbuilder/*builder" in deb base distributions, "mock" in rpm based distributions) which create a chroot, grabs the build dependencies and install them from the repo, build the package and then throw away the chroot.

It's really convenient for many reasons:

1) tracking and referencing build dependencies is far better as you have to declare them explicitly, no more "but it used to compile on the old machine... what am I missing".

2) you don't add kludge on the host running the build as you start from a fresh chroot every time.

3) you can target many distributions/versions from one building host (I'm regularly building RHEL 7 and 8 rpms and Debian 7, 8, 9, 10 debs from my Debian sid for example, without too much worry of weird side effects due to my particular setup.

The downside is that builds can take a bit more time, but that's clearly something I'm willing to trade-off for more systematic and (semi-)reproducible builds.

I even spent a few nights a month ago to have a newer version of mock that supports EL8 available on my Debian (for those interested: https://github.com/kakwa/debian-rpm-build-tools)

ISP's all over the valley did this in the 80's and early 90's as a way of supporting multiuser systems ..

We should never forget that at the root of containerization is a desire to keep everyones shit sorted. That's been a Unix thing forever.

EDIT: "Gives devices like that a second youth."

Hell yeah, long live the OP!

Anyone interested in buying a first batch OP? I ordered it early in the process and then didn’t end up using it much

Still got my two first-batch OP's, keeping an eye on the Pyra situation, but oh .. what a ride its been ..

Yeah, waiting for the Pyra as well. I got to test the Gemini which was crap (imho), the Pocket 1 (excellent machine imho), Pocket 2 (no nub and thus, quite shit; worse battery life too). I only bought the Pocket 1 because I liked it so much when testing, but the OpenPandora is still really one of my favorite machines for one the go hacking because I can just hold it in my hands instead of having to put it down on a surface.

hehe, yeah .. the Pocket1 is my current favourite handheld, there's just nothing quite like it .. plus, paired with a couple of iControlPad's, its kind of badass.

Been keeping an eye on GPD's other thingies, but I'm just jaded now. I just want a revised version of the Pocket1 that doesn't suck.. totally with you on the track-pointer.

Could do: drop me a mail; in profile.

> We should never forget that at the root of containerization is a desire to keep everyones shit sorted. That's been a Unix thing forever.

That's literally the point of operating systems: To allow many people to share the resources of one machine without being able to interfere with each other. Modern containers and VMs are just reinventing operating systems, and unfortunately they're doing it on top of existing operating systems rather than building -- you know -- a better operating system.

I have been interested in an OS that would better implement virtualization and containers better, "closer to the hardware." I know most users aren't terribly interested in that sort of thing, but I think a lot of hacker-types would enjoy easily sorting projects into different areas. How nice would it be to easily spin up a couple different machines to test a server-client relationship on the same machine, without a handful of slow VMs?

I feel I don't know enough to see the issues with an approach like that, or know why it's not already mainstream.

it's known as a "bare metal hypervisor" and is the most commonly used hypervisor. If you look into things like vmware ESXi you'll see it in action.

https://www.vmware.com/products/esxi-and-esx.html

Thanks for the link! Looks to be a little steep learning cliff, but I'll see what I can teach myself.

Any notable downsides to using chroots?

1) There were/are sometimes ways to break out of them. 2) The process table is mixed in with your OS process table - making it hard to tell what is running chrooted and what isn't. 3) The network stack is shared 4) They share an OS

You can make a "spectrum" of environments where you run code. One one side is everything running on a single server in a single OS, on the right is everything having its own machine and OS. In between you have chroot, docker, blade servers, virtual machines, and other isolation techniques. chroot falls somewhere between everything running in one system, and everything running in docker containers on one system.

I'm curious as to why blade servers are in your mix—aren't they essentially separate physical machines sharing a common backplane for I/O? Seems basically identical (from a software perspective) as servers hosted in the same rack.

They also often share power among some number of blades. When the spectrum is how co aolidated software and hardware are, it is a bit apart from separate physical instances, as the fault tolerance isn't quite the same.

Blade servers are purchased for their density.

1) should actually be "there are ways to break out of them, period". chroot isn't a security feature, it doesn't even try to be one.

This. If you want the security features, switch to a BSD (jails) or illumos (zones). These started spiritually from that same chroot place but were designed to incarcerate your software in a small subsystem.

The illumos ecosystem in particular got a lot of work from Joyent in this container vein—like how Windows can now run Linux binaries natively because they implemented the Linux system call table, illumos has Linux-branded zones that do the same “our kernel, Linux’s API, no virtualization” approach to Linux containers.

I mean you should throw in Linux filesystem namespaces in there too since I doubt too many people can just switch to a BSD for this.

Windows runs Linux binaries by running Linux in a VM. Something newish is that Windows also implements Windows by running NT and a Windows UI in a VM.

Running on a hypervisor originated at IBM, on its 370, and is very mature technology. Arguably, an OS running on bare metal is practically an embedded system, these days; There are just so many things that make a hypervisor useful or essential.

The key insight IBM had was that the hypervisor runs under the control of one of its VMs. That means the hypervisor doesn't need to provide a full-featured, comfortable work environment; that is the job of guest OSes. Instead, it manages resources according to policies implemented in an "executive" guest OS not used for, or vulnerable to mistakes or malevolence in, regular user programs.

A modern example of such a system is Qubes, security-oriented OS that hosts and orchestrates Linux, BSD, and even Windows VMs.

WSL 2 is virtualization-based (and likely Microsoft’s primary path going forward), but WSL 1 was not— it actually did implement the Linux ABI on top of the Windows kernel, allowing Linux processes to coexist alongside Windows processes (with no actual Linux kernel involved at any point).

It’s actually a pretty neat architecture— I’m on my phone right now and can’t track down a link, but it’s worth reading about if you’ve got the time. Kind of a shame that they moved on to the virtualization approach, but understandable— they’re trying to solve the same sort of problem as Wine, where you’ve got to mimic all the quirks of a foreign OS and it’s also a moving target (so you’re never “done”).

File system access is super slow on WSL. This was one the drivers. If I recall correctly it is because some common Linux syscalls (stat?) are missing/slow on Windows NT kernels.

The filesystem in general is known to be much slower on Windows due to it's extreme flexibility, but Linux design decisions assumed a much more performant filesystem. Hence why running linux on windows slammed into the performance problem.

> Something newish is that Windows also implements Windows by running NT and a Windows UI in a VM.

That’s not really that new. Windows 3.x, running in 386 Enhanced mode, was based on a 32-bit pre-emptive multitasking hypervisor (the VMM). Windows apps shared VM 0, cooperatively multitasked, and were mostly 16-bit. VM 1 and above were for DOS apps, including 32-bit DOS apps using DPMI.

(In Windows 3.0, it is possible to start a subordinate instance of the Windows UI in VM 1 or above. This possibility was removed in later versions.)

This architecture was introduced in Windows/386 2.1x and maintained through Win 95,98 and Me. (In Win 95 and later, most of the 16-bit code in VM 0 became 32-bit.)

The "chroots doesn't offer any security" statement needs to be a bit more nuanced. A non-privileged process in a dedicated directory can't do much damage, and certainly can't escape the chroot without some sort of privilege escalation.

That does not mean you can slap chroot syscalls everywhere and call it secure, of course. But it is still an important part of dropping privileges, together with seccomp-bpf, control groups and the various ACL systems.

It is an important part of things like the OpenSSH privilege separation, where the early protocol is handled by a dedicated process in a read only chroot. It proven both simple and effective in practice, contrary to the idea that chroots are escapable.

Here's a list of ways to break out, marked either as root required or not. Some are only vague idea, others are explained. https://deepsec.net/docs/Slides/2015/Chw00t_How_To_Break%20O...

I too thought you can't easily escape unprivileged chroot, but in reality you need to think about quite a few things to make sure it's true.

And once you need to think about those things... Why not apparmor/selinux instead.

None of the listed methods are novel. Most are elaborate ways of saying "privilege escalation".

If an untrusted process can escalate to root, ptrace to unrelated processes, or access memory outside your process, it was never really contained in the first place.

Do also note that none of the listed methods works in what the presentation itself calls a "reasonable chroot".

> Why not apparmor/selinux instead

It's not either or. Dropping privileges is something you preferably do in more ways than one.

> The process table is mixed in with your OS process table

Just wanted to point out that this is true for Docker containers as well.

Something that has actually come in handy from time to time trying to diagnose things.

I think from within the container you can only see that container's processes, but outside (as root) you can see all processes, even those inside the container.

It makes it easy to attach a debugger, too.

That's correct.

> 4) They share an OS

Unless you make your Dockerfiles `FROM scratch` you do have an OS too.

https://en.wikipedia.org/wiki/Chroot#Limitations

Apart from other comments same pro/con as containers: I seem to recall we had a commercial backup solution that ran like this - as suid root. And came with some quite outdated dependencies.

Vulnerabilities in libc (eg: dns resolver) bundled Sendmail, databases... They can be independently patched (good) or they can be independently left on ancient unpatched versions (bad).

If you use the system/distro library services, it can be easier to verify patch levels and know if any given hole (eg this week's sudo hole) affect your system or not.

For me, Docker is important because it lets me use Kubernetes. That's what I really want to use. If Docker used chroot instead of their own thing, I would be fine as well.

I've been using Linux vserver patch since times immemorial.

Hopefully not a dumb question, but why not use QEMU?

Also... can someone weigh in on why the likes of Linux VServer and OpenVZ became dormant??

Before KVM appeared, VServer performance was way better than QEMU.

After KVM appeared, it was a little better, so people who were already using it continued to do so.

chroot, lol. In 2000s FreeBSD already had jails, SunOS had zones. And I remember, to use Linux in early 2000s was very risky and weird.

So, where are those two (SunOS, FreeBSD) now? Still going strong? /s

SunOS became Solaris, which was forked into Illuminos which is the basis for SmartOS - Joyents virtualization platform used by many top tech platforms.

FreeBSD is probably used to serve more data over the internet than any other OS distribution. All Netflix streams go through FreeBSD.

So yeah, still going strong.

So, one got sold at an auction sale and now is part of a second tier platform, the other has as its major modern success story that Netflix runs its servers with it?

Second tier platform? What are you comparing it to exactly?

SmartOS is an open source operating system that is widely used commercially as a kubernetes node (among others). It has native ZFS and DTrace, in addition to zones, etc.

So is kubernetes second-tier? I don't understand your point.

FreeBSDs success is invalid because its Netflix that heavily use it? Get real...

That Linux has been eating the world (server, mobile, even embedded) whereas SunOS died, Solaris is mostly forgotten.

The SmartOS is just yet another OS with limited adoption. Whether it's "used commercially as a kubernetes node" doesn't mean much. Itself is second-tier, not Kubernetes (an OS doesn't magically adopt the success/adoption rate/etc of the tools it's used with. Most Kubernetes nodes remain Linux).

As for FreeBSD, which I use to run back in the day for a spell, it has hugely declined in use from the late nineties.

>FreeBSDs success is invalid because its Netflix that heavily use it?

You got it backwards. Your argument that "Netflix uses it" as proof for FreeBSD success in 2020 is what is invalid.

One company (however big) using an OS doesn't mean the OS is a success, or hasn't declined in adoption.

Doesn't FreeBSD run the very site you're on? Distrowatch recently switched to it (which IMO is a bit ironic, although they list BSDs as distros too).

There's bunch of companies that use and regularly contribute to the code:

https://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/n...

Also: https://www.freebsdfoundation.org/donors/ (I would imagine that company that donate have interest too keep it alive because they use it)

Apple doesn't disclose much about their infrastructure, but they did post jobs looking for FreeBSD admins so I would imagine they don't have a negligible number of them.

WhatsApp was running primarily on FreeBSD (not sure how it is now)

>Doesn't FreeBSD run the very site you're on?

So? As a feat is even less impressive that the one we just rejected as some major sign of it being a success (Netflix running FreeBSD).

I'm sure it also runs a lot more of servers. But it's less prevalent that it used to be, and Linux has eaten almost everything in that space.

Enumerating success stories is the first sign of a project with few of them (Linux doesn't have to enumerate how many companies use it). It's the same for language. C or Java don't need a "success stories" page. But niche languages will invariantly have one, and tout it as some proof that they're widely used...

Why are you being negative about a particular distro? You clearly have a chip on your shoulder or an ulterior motive. FreeBSD also powers my NAS (FreeNAS) in my house. It also features ZFS as a first-tier filesystem. Use whatever the fsck you want, but stop being a jerk please. People STILL care about these distros and work to improve them in whatever "niche" markets you think they're in, no sense in you being a little beach about it.

I provided individual examples. There are many others. Go do your own research instead of shitting on other peoples rebuttals to your clearly uninformed position.

Isn't FreeBSD heavily incorporated into OSX?

Kind of. OS X uses the userspace and maybe some pieces of the kernel (like the network stack) but never used the BSD kernel wholesale. AFAIK, it doesn’t have jails.

Sure; my favorite platform at that time was in fact Solaris and I had (and have) many Sun machines. However zero of my clients had either Solaris or *BSD. So what can you do...

This is basically AppImage, right?

AppImage doesn't chroot, or as far as I'm aware even use namespacing. Most of the dependencies are packaged with the application in a squashfs image that is then mounted in tmp when it is run, so that's the only similarity.

This sounds juicy - are there any good resources on this/is the product still available?

You had a checksum check there as well, right ? (I see it's over https, but still.)

Reply on news.ycombinator.com