Misadventures in process containment
apenwarr.ca
apenwarr.ca
Of course, there were always systems that asked a lot from you before they gave anything back. I think everyone's first experience of SQL was probably "what?" followed by "no, really, what?" Kneel at the altar of the cartesian product, my child, that you may be reborn into the kingdom of relational algebra. But, hey, if apostasy is your bag there's still nothing stopping you from putting all your data in one big table. If you're good enough at it you might even snag a job at Google.
But the Dockerverse is really something else. No shade on the technology – it's really cool stuff – but boy does it love to tell you how big and complex your problems are. You want to build images on one machine and run them on another? Sounds like what you really need is a Secure Container Registry! Want to have services with dependencies? Forget init^H^H^H^Hsystemd, try Docker Compose! Just kidding, I mean Docker Swarm! Just kidding, I mean Kubernetes! Actually you kind of want all of them maybe!
I don't think this is a weakness of the software itself; you can find simple ways to use containers if you try hard enough. But that difficulty really speaks to the motivations of the cloud services industry. Using Docker for simple problems is asking a car salesman for whatever will get you from A to B. Hey, buddy, you don't know what you're missing. Everyone thinks they want something simple until they see our amazing upgrade options. And what about safety? Think about what would happen to your family if you were caught in a tragic Byzantine Fault.
And, yes, absolutely, if you have Big Cloud Problems, the cost of finding the right cloud-native service discovery mesh is near-zero when amortised over thousands of servers worth of existential dread. But I just can't help but feel like I'm preparing for cloud scale like I'm preparing for a zombie apocalypse. It's an interesting problem space, and a great excuse to hang out on the tactical flashlight subreddit, but I'm not sure these tomorrow solutions actually help me with my today problems. The people selling the gear sure are doing well, though.
I've worked with startups that have invested more and more money in an attempt to get the assumed benefits of containers. The double-or-nothing aspect is what had me worried when I wrote "Docker is the dangerous gamble which we will regret":
http://www.smashcompany.com/technology/docker-is-a-dangerous...
--
Anyone familiar with the mentioned `bupdate` tool and zsync¹? I'd love to read a comparison, as I've only used the latter.
Flipping quickly through the zsync docs, it looks very well done. I'm not sure if the way they adapted rsync is better or worse than the bup/bupdate way of doing it; it looks like they take more time to do the initial encoding of the index file, but that's not very important since you only do it once, especially if it saves bytes.
They also have a (complex and potentially error-prone) way to look into .gz files and sync them extremely efficiently even without gzip --rsyncable. That's really cool, but risky, and of course only works with gzip, not other compressors. Not sure if that's a good idea or a bad idea, but nobody forces you to use it.
tl;dr zsync has actual documentation and an actual release, so you should probably use it instead of bupdate.
(People should feel free to ask me any questions about redo, bupdate, the redo container builder, etc in the comments here if you like.)
See also:
* I've abused webseed support in torrent clients to do similar things - a downloader based on the torrent chunk hashing format would be quite interesting
* rsync can do it, maybe
* http://xdelta.org/ - have used this to patch Raspberry Pi images
FWIW, the last release of `zsync` predates `gzip --rsyncable` by about six years¹.
> People should feel free to ask me any questions
"Thanks" isn't a question, but thanks for creating `redo` and the great articles. I always seem to come out of them having learnt something new, and often end up rabbit holed in the interesting side topics that are raised too.
The zsync docs clearly talk about "gzip --rsync", which I guess must have been an earlier version of "gzip --rsyncable".
* Dependencies are best represented as a tree. Dockerfile forces to linearize that tree.
* It's not possible to compose two or more Dockerfile together.
* If `COPY` or `ADD` instructions are being used, all the files are being send by the client to the daemon, including timestamp and UID. This breaks caching badly as two different users would not produce different images.
* In general Dockefile are not bit-reproducible; two developers building the same Dockerfile will get a different output.
The underlying v2 image format has content-addressable layers which is great. It means that in theory it's possible to upgrade the base OS layer without rebuilding the rest.
PS: Nix's dockerTool.buildLayeredImages fixes all those issues and can take advantage of the CAS format.
bupdate avoids that by just letting you post only the latest file (and .fidx), then the client figures out which blocks it doesn't yet have.
I've never seen debdelta used in the wild, but I know Fedora's delta RPMs work quite well.