vishvananda

Interesting. If I read this correctly, they chose to let talent affect lucky events but not unlucky events. I wonder what the rationale is for that and if it changes the results significantly to allow for probabilistically avoiding unlucky events.

This looks like a neat feature. Reading up on it briefly, it looks like it changes the mount namespace / chroots when running the binary in order to accomplish this. Unfortunately that defeats the purpose of crashcart, since the whole idea is to have access to the container file system while running your sideloaded binaries.

Unless there is some magic that I'm missing, I don't think this option helps for our use case.

we didn't look at using overlay. Might be possible, although that would introduce a dependency on kernel version and/or module. A custom fuse might be an option here as well but fuse in containers is a bit sketchy at the moment.

Ok. I will check it out. Thanks.

we could bind mount over /nix or /nix/store, but that means any existing nix packages from the container would not be available. The whole point is to have the whole container file system available along with the utilities. We could alternatively find each package that we need for our debugging utilities and bind mount each directory from the store individually. This would work due to unique paths in the store, but that means potentially hundreds of bind mounts and is an orchestration nightmare.

The point of crashcart is to be able to side load a filesystem with utilities into a running container. It is very important that the location we pick doesn't conflict with any path already in the container. If we used /nix as the mount path it would conflict with any container that uses nix. In order to prevent this (probably rare) conflict, we build our utilities in /dev/crashcart/ instead.

interesting I wasn't aware of that. I wonder why my builds are not using it.

Yeah I'm aware of the build mirror, but crashcart builds things under a different prefix so it must build things from source each time. I realize this doesn't affect most users (which is why it takes a while for disappearing sources to be found and fixed). A content addressed mirror for sources as well would solve the problem nicely.

That solution is not correct. Your boats on the top right are touching. They cannot touch, even diagonally

A caching http proxy would help me build the same things reliably, but it wouldn't help anyone else who cloned my repository and attempted to do their own build unless I also gave them access to the proxy. And it hides the fact that the standard from-scratch build doesn't actually work. I think the cached nix packages is why it takes so long for some of these issues to be discovered.

The difference with Debian (and other linuxes) is that the source code for the build is also provided. The upstream source might be from random place on the net, but Debian provides a source package that you can use to rebuild the binary package. Maybe the solution is for nix/nixos to provide something similar.

I have been using nix for a while to build binary packages for crashcart[1] and I really love the premise of isolated source-based builds.

Unfortunately, over time I've become quite frustrated with the pull-everything from the internet model. If you are building packages from scratch each time instead of pulling down cached version using the nix command, the build breaks quite often.

Mostly it is silly stuff like the source package disappearing from the net. A particularly egregious offender is the named.root[2] file being updated, which will cause everything to fail to build until the sha is updated in the build def.

I don't know that there is a great solution for this problem. Maybe there needs to be a CI system that does from scratch build of all of the packages every day and automatically files bugs. Alternatively, a cache of sources and not just built packages could ease the pain. This issue probably affects ver few nix users, but it has demoted my love of nix down to "still likes but is somewhat annoyed by".

[1]: https://github.com/oracle/crashcart [2]: https://www.internic.net/domain/named.root

I have been doing the big company thing for a number of years now after exiting successfully as employee number one of a small startup and having to shut down a larger and better funded startup where I was CTO. Since then, I've done quite a bit of thinking about how to be successful should I choose to do a startup again.

The conclusion that I have reached is that the best approach is to find a boring problem. There are so many legacy industries out there that haven't changed in decades. If you can improve one of those, you're basically printing money. This was what Uber did for Taxis. A couple of examples of startups that I've run into lately that are doing this:

ripcord.com: record storage is a multibillion dollar industry in the US alone and is largely controlled by one company that is a hundred years old and has not modernized. spruce.co: title insurance is also a huge industry that is dominated by old companies with zero technology. If you've gone through the process of buying a house you know how ridiculous the signing and title transfer is.

The last point about seed blowing has been debunked. It is listed as myth number 2 here: http://www.npr.org/sections/thesalt/2012/10/18/163034053/top...

I'm also a fan of minimal images. Cde is an iteresting solution, but for dynamic languages like python packing everything into a virtualenv and shipping that is a reasonable solution. To automatically grab linked libraries you can use something like smith[1]

[1]: https://github.com/oracle/smith

> Docker is basically the equivalent of a sign which says: "don't walk on the grass" as opposed to an actual wall which FreeBSD jails and Solaris zones have.

I think this is dramatically overstating the risks. It is possible to run containers securely, it is just much more difficult to secure containers on linux than on BSD or Solaris. It is significantly difficult to break out of a properly configured container (using user namespaces, seccomp, and selinux/apparmor), and I know of no cases where it has been done successfully.

I still separate tenants onto VMs because I don't want to be the first example of a breakout, but I don't think people who isolate with containers are crazy, just a little less risk-averse.

I agree that more implementations are necessary. My team released an oci-runtime implementation in rust a few weeks ago: https://github.com/oracle/railcar

We also released an oci-image builder that uses a different model from docker build: https://github.com/oracle/smith

Author here. I missed that this hit hacker news. Happy to answer any questions if people are still lurking.

i will update the readme to clarify that, thanks

Author here. This hit hacker news in the middle of the night for me. I'm happy to answer any questions if anyone is still lurking.

Well put in every respect. As I mentioned in a couple other comments, that sentence could definitely be clearer.

Author here. As I mentioned in another comment. I should have clarified that its shortcomings are specific to the container runtime (i.e. the part that creates and enters the namespaces). Go is a great choice for the command line utilities and daemons. I am a big fan of go in general, and even the container builder smith is written in go.

Author here. For more discussion on init handling you may find my article on pid namespaces[1] enlightening. In terms of differences from runc, there aren't many. The goals for railcar are threefold:

1: provide an alternative implementation of the oci-runtime so that the spec doesn't become too locked to a single implementation.

2: provide an implementation in a single language without some of the "baggage" of the existing implentations so that it is easy and fun to hack on.

3: experiment with new ideas to inform the future of the oci-runtime spec.

[1] https://hackernoon.com/the-curious-case-of-pid-namespaces-1c...

Author here. This hit hacker news in the middle of the night my time. I think the other replies already answered your question but it is worth mentioning that my point could have been worded better. Note that go works very well for other parts of the container system: docker client, docker daemon, containerd, kubernetes, etc. It is just the low-level namespace handling it has trouble with.

Author here. Usage of rust at Oracle is pretty nascent, but it is a large company and a few people are using it here and there.

One was a simplified runtime in c, similar to a stripped down version of systemd-nspawn. I can hopefully share the other one in a few weeks. I'm going through an open source approval process for it.

I agree that rust will have an adoption problem amongst part-timers. Rust code is unreadable until you spend a few days with it and writing it requires a week or two of wrestling with the borrow checker.

In contrast, one of the reasons I'm a big fan of go is it is extremely easy to read for almost anyone. People pick it up very quickly.

Author of the go netlink library here. I've run into this issue a number of times. There has been conversation in the past about adding some kind of new runtime command like LockOSThread to prevent new threads from being spawned, but it didn't gain any momentum.

Even though I am a big fan of go, I've personally built two container runtimes in other languages do to the namespace clumsiness.

Personally, I think rust is an excellent alternative for namespace utilities.

EDIT: there is more information and links in the issue in the netns library: https://github.com/vishvananda/netns/issues/17

My read of the documentation also indicates a large overlap with linkerd. Hopefully one of the creators is around and can do a compare/contrast.

I came across this recently. I had always wondered why I managed to find math so fascinating when most of my peers hated it. I think I was just lucky enough to see through the poor state of education into some of the magic underneath. For a (partial) solution to the math education problem, Paul Lockhart has also written a book called "Measurement"[1] which is a very entertaining read.

[1] https://www.amazon.com/Measurement-Paul-Lockhart/dp/06742843...

Debugging is a valuable skill that some of us were lucky enough to pick up by chance. A friend recommended an excellent book that very nicely explains the process: "Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems"[1]. I found myself nodding along while reading. I think this would have been a hugely valuable primer if I had been exposed to it before I figured out how to do it on my own.

[1]: https://www.amazon.com/dp/B00PDDKQV2