Kraken, an Open Source Peer-to-Peer Docker Registry
eng.uber.com
eng.uber.com
I wish more companies used BitTorrent internally for distribution of tools and software. I feel like homogenous environments like those that benefit from centralized config management like puppet would probably see performance gains with P2P technologies introduced.
This looks like a great contribution from Uber engineering and I look forward to playing with it!
In practice, internal networks are such a mess that leveraging p2p is often more trouble than it's worth.
[0]: https://chihaya.io
Under the hood, a tool like Artifactory could take advantage of this means of distribution, be responsible for signature checking, etc.
That doesn't have to be the case. You can certainly have private peer-to-peer networks that are authenticated and/or restricted to your internal network.
Facebook did this back in 2010: https://torrentfreak.com/facebook-uses-bittorrent-and-they-l...
That's different from applying a 50,000 node solution to a 50 node problem though.
Unfortunately you'd need something to manage the pinning settings, but it feels like a relatively small addition to some other registry that could be hacked together in a small amount of time.
I was more thinking of a harbor[0] like cobbling of these technologies together -- harbor combines a bunch of F/OSS tools into one powerful registry solution. Here are a few:
- Distribution for storing images
- Notary for signing
- Clair for static analysis
Based on this I was thinking of was basically Distribution + ipfs cluster + a small management daemon. The daemon is only there to tie the other pieces together and present a unified interface but the bulk of the work can be done with the other pieces.
[0]: https://github.com/goharbor/harbor/wiki/Architecture-Overvie...
[1]: https://github.com/docker/distribution
"... a test where a 3G Docker image with 2 layers is downloaded by 2600 hosts concurrently (5200 blob downloads)"
they show this finishing in 20s, so that's 7,800 GB transferred in 20 seconds. That works out to 390 GB/sec. A 10Gbps interface can transfer about 1GB/sec, so you'd need 195 dual attached fileservers, or about 100 if they were running at 40gbps.
This line is far funnier than it should be. It reminds me of the time that I was told about how awful some architecture was because it would overload the network.
Their improved architecture sent lots of smaller files over NFS instead.
That was novel.
This is total nonsense. This would mean that instead of using 1200+ byte packets, bittorrent uses 120 byte packets to transfer data. This is easily disproven by looking at the implementation of any client or just by looking at the traffic and seeing that it uses 1200+ byte packets for sending chunks around.
I'm always wondering though, if those optimizations are really needed or if they are created by engineers that want to have fun and do premature optimizations. Ever wondered why a "WebApp" like Uber needs so many engineers and new projects?
But when I look at companies like Airbnb and Uber for example, I cannot really understand why they require that type of tools. Yes they are big, even very big, but nowhere the size of a Facebook or a Google. Most of those projects seem to be engineers having fun at work, and marketing themselves by publishing cool blog posts.
MPM traffic was low priority, so for large enough binaries (and most of them were, because of static linking) you could easily see a Borg task spend seconds or even tens of seconds in "downloading" state if the borglet hadn't already seen and cached locally that version of the package.
It was about time this became available also for Docker and Kubernetes.
I guess what IFPS is missing and Kraken has are "pluggable storage options, and instead of managing data blobs, Kraken plugs into reliable blob storage options like S3, HDFS"
#1 - Dragonfly started November 2017 where as "Kraken was first deployed at Uber in early 2018". Which presumes Uber was working on this probably much earlier than Dragonfly even started.
#2 - Dragonfly is still listed as a Sandbox CNCF project, where as Uber is running this in production for roughly a year it seems.
#3 - Dragonfly was just refactored completely from Java to Go. So its kind of hard to say it's "battle tested" at all, at this point in time. It's basically an entirely new project as of 2 days ago.
What got into CNCF may be a newer iteration based on what Alibaba has been running but that means it should at least hold a candle to what is described in the post. With that said, one can surmise the Java version is the one they used in production.
Apart from that, P2P image distribution is probably limited to internal uses that are not critical. Kind of missing indications of much difference in reliability to say an internal CDN this would give.
Same is the case with 'Discourse'. Almost impossible to find relevant content cos any search for 'discourse + keyword' invariably shows content related to "online discourse (conversation)"
But now I realise it is a separate project by Uber that is called Kraken.
But as it is a project I am okay with the name clash. It kind of suits the distributed nature of its arms.
I know there will be some confusion but hopefully, people's google-fu and search terms will be able to distinguish from Kraken the cryptocurrency company, Kraken the docker registery tool and Kraken the mythical mega octopus, and more.