CDE: Easily run a program on any linux without thinking about dependencies | Hacker News Reader

CDE: Easily run a program on any linux without thinking about dependencies

stanford.edu

132 pointsgregwebs32 comments

Reposting the comment of chbarts, which has been killed despite being entirely appropriate. Moderating has gone completely out of hand lately:

chbarts wrote:

" I think the potted example on the web page is somewhat poor, as it seems to be a problem existing package management systems already solve.

The more interesting aspect is the sandboxing, and how this is great for one-off software runs where it isn't worth it to install the needed packages on your base system because they're logically just a unit with the software you're testing. You mention this, but I think you'd do well to make it more obvious.

This is also, and this is something you mention, a way of preserving historic software and data. Bundle a full CDE package with an emulator and you can keep running that program for even longer; in fact, it seems a CDE package could gradually grow layers as time demands, from moving a (user-mode) kernel in, to adding device emulation, to going the final step and making the package more-or-less a ship in a bottle running in full hardware emulation.

This could also grow into a way to run untrusted (or minimally-trusted) software in an iron box, not just a sandbox. But that is harder and probably not your preferred direction."

thanks for reposting, sfk. i don't know why the original comment would've been killed ... chbarts brought up a great critique about my presentation of cde.

It looks like they got capriciously hellbanned for comments in this thread: http://news.ycombinator.com/item?id=1320949 — everything after that was auto-killed, several dozen productive comments over the course of six months.

Items can be fully deleted or [dead]-ed, and users can be silently hellbanned so that all of their contributions are [dead] on arrival for everyone but themselves. AFAIK there are no other metamoderation mechanisms.

Common Desktop Environment? I have a couple old RISC boxes running it...

;-)

Sorry. Couldn't resist...

Now, seriously, this goes a bit beyond package management by bringing along what seemed to be the specific versions of libraries. The goal is to do reproducible results for scientific data but, in this case, I would prefer not to have the original libraries packed along but having the dependencies flagged in a way that any version-specific bug could be easily pointed out.

I also though Common Desktop Environment at first. At least from a 'marketing' perspective, it might be a good idea to not use an acronym that is already in use in the Linux/Unix world.

This is a very good point. I can't see CDE and not think about all the environment on all the Sun workstations that I have had in the past.

That was my first thought as well. My initial exposure to nix was a purchased copy of Red Hat Linux 4.2, running on an HP Vectra tower. and I remember paying extra for the CDE install CDs. An absolute noob when I started, by the time I'd learned enough to make my modem work I felt like I'd climbed a mountain. sigh* Good times.

Common Desktop Environment. That was also my first thought. We had it on our Sun E250.

Nice machine. Wicked fast on its day. I bet it still can hold its own despite the 2GB RAM limit.

It was and served well. But now it's not in use anymore.

hi everyone, i'm the author of CDE, so please feel free to ping me if you have questions, complaints, or bug reports. i'm in the midst of fixing a bunch of silly bugs now and improving the documentation.

I find the dependency-capture mechanism very interesting. I would love to see it used to help generating dependency lists for package management systems too.

I will probably have to build my first .rpm packages (I am yet to read the first docs) in the near future and CDE could be a lovely addition to my toolbox.

Thanks for doing it.

This looks awesome. My biggest initial fear (not having used it yet), is that I would inadvertently package up something like my private SSH key.

oh good point. the contents of the cde-package is simply a directory hierarchy, so you can feel free to use 'find' to skim thru it. also, i'm now adding a feature where you can tell cde to ignore certain directories (e.g., ~/.ssh/) so that it won't pick up those files.

Maybe you could output a warning on files with perms like 600?

Something like this has been sorely needed. I'm wondering if it's possible to generalize this over different OSes, if the code itself is multi-platform, or there is some way to specify hints for it (or should there be). I also go to Stanford so would be easy to discuss this in depth if you want.

please email me at the address listed on the website. looking forward to chatting more about this!

could you put up a doc or something explaining the difference to chroot jails ?

because you mention: . CDE runs packaged programs in a sandbox similar to a chroot jail, but truly malicious applications can circumvent this sort of isolation attempt.

Do you use similar techniques to chroot - essentially I am asking whether your code is basically chroot (in terms of its behavior AS WELL AS its internals) with some security features removed ?

no, it's not using the same mechanism as a real chroot (since you need to be root user to create a chroot). cde uses strace to intercept all the system calls that a monitored child process makes, and re-writes all file access system calls to re-route their paths inside of cde-root/.

This is fantastic. I haven't tried it out yet, but I'll give it a shot for the paper I'm writing.

BTW, even with a "uniform" Linux environment, I've been finding so many differences in the behavior of libraries that it's driving me nuts. This will be a nice way to get past, for example, inconsistencies in wxWidgets.

what research field are you working in? please email me personally if you'd like to chat more about how cde can help your workflow. i'm interested in collecting real-life case studies and anecdotes

I replied to you via email, but thought I'd post it here as well in case anyone else had suggestions.

I've been thinking about a general purpose experimentation framework for research. Over the last 5 years, I've been re-writing parts of this framework for different projects and wish I had spent some time generalizing and packaging it.

The workflow is:

dataset -> experimental parameters -> code+libraries (this is where the trouble often is) -> result files (plain text, hundreds!) -> analysis (generally in R) -> selectively plotting results

From my superficial knowledge, this is quite similar to the PC-BSD PBI format: http://www.pcbsd.org/content/view/20/26/ . They both allow you to distribute software in an easy non-hellish way on Linux.

Wouldn't this cause severe problems with copywrite and licensing? Just because you have a copy of some library on a machine doesn't mean you have the ability to duplicate it to another machine, let alone archive it for future researchers.

I could also see the possible issue of pulling in GPL licensed libraries, and the general theory is if you ship a package with GPL licensed code in it, your code must be GPL. There's a fine line between "sending along someone else's libraries" and becoming a monolithic executable containing someone else's libraries - and between those points is where GPL gets you.

There needs to be some kind of solution for the problem of really confidential files to be packaged. The second example in the video packaged the machines /etc/passwd for example (granted, it doesn't contain the passwords, but still).

The third example was showing some real bloat in that it also packaged the kernel and the initramd. I really don't see how these two can be of any use on the target machine.

Still. Very interesting idea and impressively easy to use.

It reminds me of the Ermine Statifier. http://www.magicermine.com/sesl.html. This tools is so useful when you have to for example run tcpdump or a QT app on an embedded system without installing a TON of libraries or messing with LD_LIBRARY_PATH.

Cool idea, and a worthy cause. Is there any provision for copyleft license compliance? Perhaps a distant-future version could use dpkg -S or the rpm equivalent to find which package owns a file, then apt-get source or something to optionally link or bundle a source package with the sandbox.

awesome tool, great job!

no like the name though. sounds like a Unix desktop system.

If anyone wonders why C/C++ programming is still so popular, here's a perfect case study. Most new fangled languages don't offer the option of static linking. This single piece of uncompromising idiological doctrine has led to billions of dollars in effort expended in engineering systems to workaround that choice and to effectively reproduce static linking in different form.

Most "newfangled" languages don't use linking in the same way as C and C++ do anyway (unless of course you're interfacing with C or C++). For instance, you can't statically link to a python source file. That's not because of ideology. It's because static linking doesn't even have any meaning.

Analog of static linking for a pure python library is including it within the library package. Another example: deploying python application in its own virtualenv environment.

dynamic linking is generally a functional superset of static linking because you can bundle the needed libraries with the app, as say loose files in the app's dir tree, for example. This lets you avoid Shared DLL Hell while still allowing you to surgically deploy a critical bugfix to one of those libraries.

Reply on news.ycombinator.com