LXD 2.0: Introduction to LXD | Hacker News Reader

LXD 2.0: Introduction to LXD

stgraber.org

226 pointsbpierre50 comments

> That is, a LXD container runs a full Linux system, exactly as it would be when run on metal or in a VM.

This is what makes LXD so nice. It's the power and flexibility of a VM with the speed, api, and runability of containers.

I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container. Even google runs ssh server alongside their applications.

With LXD you can easily port entire services over to containers without completely rewriting the way they behave.

> I always hated that docker forced you to treat a container as a single process and spoke about it as if it's a virtue. Let me decide how many or little processes I want running in a container.

You can have as many processes as you like running inside a Docker container, you just a need a single process to "boot" them, even if it's a simple shell script.

The big difference is that LXD can run, in an unprivileged fashion, a systemd based distro without issue, and with a sensible view of /proc.

Docker can't currently do that, largely because running a distro in a container isn't an important goal for them.

Docker can run systemd in unprivileged container: https://github.com/vlisivka/docker-centos7-systemd-unpriv

PS.

Try it:

    docker run -dt --stop-signal=$(kill -l RTMIN+3) -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name centos7_systemd vlisivka/docker-centos7-systemd-unpriv

My impression is that someone created a specific dockerfile to get around the issues. I'm noting that LXD can run, for example, stock debian.

This appears to be a fairly complex subject. See https://news.ycombinator.com/item?id=11216950 and the original LWN article that it depends from.

There are quite a few issues with running a systemd-based distro in a container. The first being that /sys and /proc are basically not namespaced. So if you're not careful, systemd might start modifying your host. There's some work to improve this (the cgroup namespace -- yes, that's a thing now) but the point remains that allowing a root process to start touching /proc and /sys inside a container won't end well. I believe that LXD tries to get around this problem by mounting a FUSE filesystem on /proc and /sys that does a bunch of fakery. I don't like that, to be honest. It should be the kernel's job to namespace things properly.

That's not a good idea, for the same reason you don't want a shell script for /sbin/init.

Indeed, LXD would seem to be a faster, more efficient way of doing what you would do with traditional VMs. I don't see a lot of overlap with docker, despite the superficial similarity.

There are still a few gotchas on the low level stuff, especially networking, but also other areas that involve kernel drivers more directly.

The security around the containers still isn't as good as a hardware VM, but I think hardware segregation is on the road map.

It's ideal for running container inside VMs, though, and offers nice management segregation on those larger VMs.

Can you name some of these networking gotchas you mentioned? And elaborate on the security remark?

The major security issue is that containers share the same kernel as the host, with the whole kernel syscall interface as attack vector. VMs run separate kernels that only interface with the host via the hypervisor.

And we haven't seen many hypervisor security holes in the past... :) Especially with Xen.

Well, there's also the issue that "Linux containers" are not a kernel primitive. So there's plenty of syscalls that aren't even namespaced (the keyring ones come to mind). Docker handles this by having a default seccomp profile that disables a bunch of syscalls that aren't namespaced.

However, the development of Zones and Jails (on SunOS/Solaris/illumos and FreeBSD) was much more security focused, with the default being "that's not allowed in a (Jail|Zone) until we can make sure it's safe". I really wish Linux had just ported Jails or inspired their security model on them.

I do a lot of work with the internals of Docker and runC at SUSE. Trust me, it's not pretty how you have to set up "Linux containers" and there's 1001 gotchas.

Congratulations on 2.0! I tried out LXD a few months ago after hitting a few walls with Vagrant, and I've been happy since.

Mark Shuttleworth's normally responsive on Google+, and he helped point me in the right direction with initial issues that I encountered. Stéphane and the team have also been helpful on Github with issues.

The first thing I struggled with in this containers business was "what's the difference between LXC/LXD and Docker et. al.?", and I am glad that he addressed it first in the post.

I've got north of 20 containers on our machines, and LXD allows me to create a container, play around with software config on it, and when I get it right; I just leave that container and update it regularly like the host it's on.

You can also use LXC as Vagrant backend to get away from the crashy and buggy VirtualBox kernel driver. Though ~all the Vagrant images you find online are tied to VirtualBox.

Yeah, it can be a pain to get images.

That said, my workflow for testing anything these days is inside a vagrant lxc box. it is well worth the lack of images for the improved workflow.

This looks really nice.

I build out major productions systems on top of LXC at Heroku, and to this day it is still working incredibly well.

So I am really looking forward to further iterations of the tools by Stéphane Graber and co at Canonical.

The main thing that jumps out at me is a REST API for this layer. A REST API with Golang bindings to run containers feels so much better than old-school systems programming with exec.

I'm currently all in on Docker in production for this very reason. Coordinating stuff with a golang client over HTTP is really really productive.

If I need to run Linux software, I can use an lx-branded zone inside of SmartOS, and run at the speed of bare metal while having ZFS, mdb, kdb, dbx and DTrace, CPU-bursting, and change kernel schedulers on the fly.

But then, as I am about to run Linux software inside of an lx-branded zone on SmartOS, I realize that SmartOS has 15,000 packages, with most the of the software I would have run on Linux, available natively on SmartOS:

http://www.perkin.org.uk/posts/building-packages-at-scale.ht...

...so then, which operational advantages would LX containers on Linux offer me, that I already do not have on SmartOS with zones?

You can think of it as competition. Other people can think of it as something that they can make people who are already familiar with systems that use the Linux kernel.

SmartOS is paranoid about correctness of operation and data integrity; Linux and LXC containers are a poor competition in that area, when the OS comes with memory overcommit turned on by default, and default filesystems do not include data write guarantees nor block checksums...

  What’s LXD?
  
  At its simplest, LXD is a daemon which provides a REST API    
  to drive LXC >containers.

Thanks, that clears it right up.

Not sure if you're being sarcastic or not?

No sarcasm. This is a simple explanation of where LXD fits into the stack.

If you don't know what LXC is maybe this isn't clear. But LXC stands for LinuX Containers, and is extremely fundamental technology in the containerization space.

(You should possibly have replied to the post above me)

> LXC stands for LinuX Containers, and is extremely fundamental technology in the containerization space.

Not exactly.

LXC was amoung the first pieces of software to take advantage of namespaces when they were added to the kernel; but LXC is not often used. Most of the various containerisation software now available does not use LXC, and it not based on the same codebase.

I like almost everything about LXD, except the attitude towards networking. The project has rejected simple solutions such as port forwarding, saying that managing the network shouldn't be LXD's job. Instead, they'd like the user to manually configure their own bridges or routes or iptables chains.

I can kinda understand their point of view, there's no simple solution that will please everyone. But most developers or IT folks aren't networking experts, and LXD won't be an intuitive tool for them without a simpler mode of operation.

I have just finished a small LXC deployment, and the networking did take a little bit of time to figure out what direction to go.

I ended up choosing to use bridging to connect all the physical network adapters to the ones in the containers. This is nice because I set up each container with its own IP address, which travels with it when the container is moved to a new host.

> [...] most developers or IT folks aren't networking experts, and LXD won't be an intuitive tool for them without a simpler mode of operation.

Really, no expertise is required. Just basic understanding how the heck the network works. If somebody can't grasp what bridge interface is or how NAT operates, he apparently doesn't have the qualifications for writing software.

Then I guess most of the developers I've worked with should brush up their sandwich-making skills and quit their jobs.

You do realize that this is not to their credit, right?

It simultaneously irks me and pleases me that the first result on DuckDuckGo for "lxd" is the Legion of Extraordinary Dancers.

(Google gets it right, presumably because of all my code searches.)

"[...] a 2010–2011 web series about two groups of rival dancers: The Alliance of the Dark who are the villains and The Legion of Extraordinary Dancers, the heroes, who discover they have superpowers referred to as 'the Ra' through their dance abilities."

I'm moving my home NAS back to Debian after a couple of painful months [1] [2] with FreeNAS 9. Would this give me jail-like functionality under Linux? The one thing I enjoyed about FreeNAS/FreeBSD was how easy it was to create and tear down jails for all kinds of uses.

[1]: https://blog.voltagex.org/2016/01/08/freenas-9-3-on-the-supe...

[2]: https://lists.freebsd.org/pipermail/freebsd-net/2016-Februar...

Plain LXC, rut, or docker are probably closer.

I like docker, but I prefer something closer to a VM. Why would you say go for LXC over LXD?

Also, Googling "linux rut" doesn't bring up anything useful, I'm assuming you mean rkt

LXC is very nice. It's very VM-like and why I prefer it for almost everything I deploy instead of using Docker.

I know how to manage machines. I don't have the same confidence for Docker-images.

LXD seems to be a construction on top of LXC to make managing it easier "at scale", so if all you want is to manually be able to construct "container VMs" on demand, I think LXC is closer to what you actually want: It let's you manage the containers directly, near the iron, without anything getting in your way.

Don't get me wrong: LXD seems to have a genuine value-proposal, but for this use-case it seems to be a slight step of indirection, and my guess is that this will cause you more troubles than it's worth.

Very Nice.

I want to understand how Linux Container works, started to study the cgroup code in Linux kernel and LXD. Here are couple of notes on Tracing the code of CGroup in Linux kernel and LXD I took.

http://www.srcmap.org/sd_share/7/0cc46512/LXD_Code_Trace.htm...

http://www.srcmap.org/sd_share/7/060b8d61/Linux_Kernel_CGrou...

This looks really nice. A REST API with Golang bindings to run containers feels so much better than old-school systems programming with exec.

Does LXD provide any type of cluster management behavior? As in if I have two host machines and a dozen containers and host A goes down, is it possible to configure a live migration of the latest known state from host A to host B using LXD alone? Or do I need something built on top of it to do that?

I think that's Juju's role. They have a beta LXD driver for Juju.

https://jujucharms.com/docs/master/config-LXD

There's an OpenStack driver for LXD.

LXD and any sort of cluster seem incompatible at the moment. LXD, for example, stores all the LXC config info in a local sqlite database. When you launch a container, it turns the sqlite stored info into an LXC config file, and spawns LXC.

You can control remote instances of the LXD daemon, but there's no syncing back and forth from Host1-LXD1 to Host1-LXD2...they operate separately. You can migrate a container from one host to the other, but it's pushing everything (config/image/etc) when you ask it to migrate.

So, it's not clustered itself, and it keeps the important info somewhere that's hard to cluster.

It would be interesting if they would replace the local sqlite store with something similar to etcd. That still leaves the issue of the disk images though.

Proxmox does cluster LXC, so there's that option.

LXD looks similar to OpenVZ or rather Proxmox(with LXC or OpenVZ)

With so many contenders stirring up the pot, this is an exciting time in the world of Linux Containers!

I would love to see an LXD container driver for Mesos.

Do any of you know if LXD is APPC-spec compliant?

It's not as the appc spec stands for "app container" and LXD doesn't run application containers but instead full system containers.

Is anybody making an effort to put the LXD environment on Centos 7? I'd really like to try it but all I have provisioned right now is Centos 7 (and below).

Wonder if I'll be able to use LXC/LXD on "plain" Debian, and be able to avoid Ubuntu.

Ubuntu, yes, I'd think so. Canonical, however, not. What makes you want to avoid Ubuntu?

I use "plain" Debian on my servers when possible. Personal preference, mostly. Not really a fan of Ubuntu Server.

if someone can create a boot2docker clone for this, i'm in.

Why not just use regular TinyCore? Or any other distribution?

LXC containers aren't app containers. It is intended to be used with regular installs, not with single applications.

Reply on news.ycombinator.com