Snappy-start: Tool for launching a Linux process from a snapshot | Hacker News Reader

Snappy-start: Tool for launching a Linux process from a snapshot

github.com

78 pointsingve32 comments

Condor [1] has done this since sometime in the 90's. This [2] paper contains references to a paper about it from at least 97. An interesting part about the Condor approach is that it can proxy file and network access, so the file-descriptors etc. can remain open (of course this may not always be what you want).

[1] http://www.sysproject.info/condor.php

[2] http://research.cs.wisc.edu/htcondor/doc/condor-practice.pdf

This goes way back, to 1960s and 1970s transaction systems such as CICS and Univac's TIP. The idea is that you have a copy of an initialized transaction program in a ready to run state, and when a transaction comes in, it's attached to a copy of the program and executed.

This approach comes in two flavors - in one, each transaction gets a fresh copy of the program. In the other, the program is used over and over unless it aborts, in which case a new clean copy is loaded and used. This corresponds to CGI and FCGI under Apache.

Programs which run as transactions have to be prepared for some unusual events. Time, for example, suddenly jumps forward when a stored transaction program starts. File and network connections have to be connected to the right things. Stateful connections, as with databases, may not play well with this approach. This can work well when the program doesn't have that much live state when the transaction program is frozen. A program that has a lot of startup overhead related to getting external resources may not be a good candidate. Unfortunately, those are the ones for which this is the biggest win.

If the main program is program loading overhead, as in Python and Java, maybe the solution is to go hard-compiled. C, C++, Go, and Rust are hard-compiled, and you can even hard-compile Java with GCC. Python and Javascript don't have that option.

(Nightmare of the future - precompiled Javascript snapshot blobs loaded into browsers.)

Fascinating. Imagine being able to (re)start Java, Ruby and Python processes nearly instantly... No more interpreter startup overhead, just right into the process. There's some interesting implications for deployment as well; snapshot a containerized process on one machine, reproduce the disk elsewhere, and restart the already partitioned process.

I would think you'd need rather robust error handling in the program being snapshotted: there would be several machine states which would change from run to run.

What Smalltalk did to enable all of the above you mention: Certain facilities like networking had to be quiesced just prior to pickling, and then specially revived on return from snapshot. There were also special hooks for return from snapshot. (IIRC, there is a Smalltalk method of that name.)

Snapshots becomes much more powerful when combined with a transactional log of all changes. This gets around certain problems with de-synchronization of source files with the image. Smalltalks that abandoned the change-log and returned to conventional source files suffered for it. (Enfin/ObjectStudio)

Getting this error while running ./make.sh:

   Hello, world!
   Hello, world!
   * Running test: ./out/example_loader
   Traceback (most recent call last):
   File "tests/run_tests.py", line 57, in <module>
    Main()
   File "tests/run_tests.py", line 40, in Main
    RunTest(['./out/example_loader'], -1, use_elf_loader=False)
   File "tests/run_tests.py", line 34, in RunTest
    subprocess.check_call(['./out/restore'])
   File "/usr/lib64/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
   subprocess.CalledProcessError: Command '['./out/restore']' returned non-zero exit status -11

Interestingly the project has Github issues disabled.

> Interestingly the project has Github issues disabled.

This is pretty weird. Certainly goes against the spirit of free software in my book. It's not like they have to answer any issues.

(Oh right it's Google. Well that explains that.)

Actually there is nothing in the definition of free software that requires a bug tracker, even by gnu.org and rms standards and they're pretty high.

gboudrias said “spirit”, not “definition”.

Having Issues enabled sets the expectation that they are going to do something about your issue.

This is also sometimes called 'checkpointing', and has been a long-studied feature necessary for distributed processing systems. There's a lot of ways to skin this cat. Kernel modules, application libraries, userland tools, and features to support various kinds of applications (single-threaded, multi-threaded, preserving TCP connections, etc). Find the one that fits your use best.

Descriptions of how checkpointing works: https://en.wikipedia.org/wiki/Application_checkpointing http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.117... http://www.anchor.com.au/blog/2013/02/overview-of-checkpoint... https://lwn.net/Articles/478111/ https://www.usenix.org/legacy/event/usenix01/freenix01/full_...

Individual solutions: https://ckpt.wiki.kernel.org/index.php/Main_Page https://code.google.com/p/cryopid/ http://criu.org/Main_Page http://crd.lbl.gov/departments/computer-science/CLaSS/resear...

An almost-full list of solutions here: http://checkpointing.org/

I've always wanted the ability to take a snapshot of a running process, say a ruby repl, that's thrown an unhandled exception, waiting for user input. Send that snapshot to a friend, who can debug the issue, send back, etc. Collaboration on a live system over the internet.

Yes, exactly like smalltalk. sigh.

I used http://criu.org/ a while ago to do this for Clojure REPLs

Looks really nice except having to compile custom Kernels I guess.

but it is in user space, why the need for compile kernel?

Some distros (e.g Arch) don't have CONFIG_CHECKPOINT_RESTORE enabled by default.

Did it change the startup time significantly?

Yeah, it was much faster than starting up the normal way, at least for a small heap (213 ms to restore 128MB on an E5645). I didn't pursue it further because it required root (not sure if it still does) and would've required writing some kind of snapshot manager.

They're adding integration in with Docker too for Criu: http://criu.org/Docker

Here's a fun example of it in action: http://blog.kubernetes.io/2015/07/how-did-quake-demo-from-do...

Very cool. Would be even nicer if the snapshotting process created a self-contained ELF executable, rather than a snapshot file that needs a separate executable to start it back up. Seems like it should be feasible.

Since the README mentions kentonv, I wonder if this would be useful within Sandstorm.

Sandstorm is the motivating use case. :)

This is a nice PoC tool, but as a checkpointing solution the idea of using ptrace(2) just reeks of being a huge hack.

If you want to do syscall replaying, check out the Approximate Replay-Trace Compiler (ARTC).

For a full checkpointing suite on modern Linux, see CRIU or DMTCP.

Sounds like an ideal tool for debugging.

Trace while running, take a snapshot when there's something wrong

It's a cool hack, but only useful for fast startup just at this point. It can't do network sockets (for obvious reasons), or things like gettimeofday or even fds just now.

Reminds me of a Lisp image dump.

If you want a debugging tool, try http://rr-project.org/

I'd imagine that it has some overhead. So running it on production all the time would not make much sense. Because sometimes restarting a process makes it very hard to reproduce the bug. But snasphost / restore would handle that case. The problem is - what to do with file / network descriptors?

Lots of Smalltalk production systems did exactly this. One could also log into running production systems and inspect live objects.

This is really great tool! I wonder if it could work with Java programs... Can't wait to try it...

I'm personally excited to see if I can get this to allow me to persist tmux sessions over a reboot.

Thoughts?

how does this compare to digitalocean's loading from snapshot? It takes 30 seconds or more to load your snapshot. Is this any different? I'd love to not rely on DO because this is one of the key selling point of DO, saving a snapshot and loading it.

This is about snapshots of a single program. What you are describing at DO is for an entire virtualized computer.

if I wanted something like DO, what can I use?

if this works on a single program, what sort of use cases would this come in handy? Perhaps a java program that has a bit of startup time?

VirtualBox has snapshots. Here's a page with an image: http://ehikioya.com/moving-virtualbox-vm-snapshots/ The page is talking about cloning, but you can save/restore snapshots within the same VM too.

Reply on news.ycombinator.com