Nix × IPFS – Milestone 1
blog.ipfs.io
blog.ipfs.io
P2P, such as IPFS and recently BitTorrent 2.0 with its hash tree per file, is the only free (as beer and speech) reliable way to host things online forever - at least as long there is the last of the veterans who keeps a copy and seeds it.
2 of them are run by commercial entities. I don't think those are all that interested in reducing dependency on them.
Docker is also removing some old images and putting limits on existing images, it's too expensive for them to host.
On top of that, Bit torrent has a tendency to overwhelm routers and make them slow to a crawl.
Those are a lot of downsides to ask your users to accept just so you can cheap out on download servers.
1. the fairness in TCP is done per TCP connection, bittorrent opens many connections so they take over the whole bandwidth. You can limit number of connections or throttle the bandwidth used by BT (of course you will get slower speeds), maybe there are some other ways.
2. when you maximize your throughput ACKs might get dropped which slows everything down (I suspect that might be the issue you're talking about) again you could throttle or enable traffic shaping and give highest priority to ACKs. This worked well for me.
Not all 1st world countries have data caps. I pay ~20$ for month for 37mbps (15 upload) with no data caps (it's literally the cheapest plan I could find, if there was cheaper available, I would take it).
P.S. until 2009-2014 russian internet was completely out of government sight. an average internet user in big cities had a choice of 3-10 ISPs competing prices etc
The current bittorrent company is interested in chasing ICOs, I don’t think they work on new versions of the protocol anymore?
edit: yeah it’s from 2008
"work on new versions of the protocol anymore?" made me think of the BitTorrent protocol in general, sorry for the misunderstanding.
I think you're right that the BitTorrent company aren't directly involved in the protocol work, they seem to be focused on trying to push their plagiarized cryptocurrency project.
It seems like you could implement a p2p solution as a pull through registry... as in host a registry (even per node if you like) that Docker will use as a pull through cache, still speak the same distribution protocol, but have the registry use p2p to get the content.
IPFS is a distributed CDN; but not very good for storing things persistently or reliably from my experience.
At the moment; the nixos cache is stored in very durable and reliable S3 storage; with very high durability guarantees. Why is that not good enough?
sure it's centralised. But IPFS doesn't offer distributed durability; it offers a CDN. It doesn't seem to address this issue the authors seem to claim it solves. (To me)
I still think it's _super cool_. And once builds are also content-addressed and not just fixed-output derivations; having trustless content delivery is a valuable addition. But IPFS doesnt' seem like a robust answer for the "what if we lose access to the source code problem" given it's not a durable storage system
In this case; projects like https://www.softwareheritage.org/ and https://sfconservancy.org/ seem better bets to solve the source code access issue
With distributed storage like IPFS or BitTorrent the availability of resource is proportional to its popularity. So as article explains, initially there will be group of seeders. I assume anyone who downloads an keeps a package on the system becomes a part of sharing swarm. This dramatically reduces the burden of hosting from package creators as the cost gets distributed among all available peers. As long there is one peer who has the complete package it will be available even forever.
Usually a project like Nixos would not use the simple standard ipfs daemon but something like cluster.ipfs.io and i think i have also seen ipfs servers on top minio or s3 for storing the data. So the actual databits durability and reliability can be the same or better than the s3 comparison with ipfs.
A discussion like this is best split up into the api and storage system parts: so S3 becomes the S3 Rest API and an S3 object store and IPFS becomes the IPFS access protocol (and IPFS gateway REST api) and whatever IPFS storage system a projects choses to use.
The cool thing about IPFS for package managers lies more in the protocol and architecture part than the low level data storage.
- content addressable storage as core design principle solves a lot of problems that need to be manually build on top of s3 anyways if you want to build a CDN or Package manager directly on S3, but in a standard and documented way . I would argue even without the p2p distributed part of ipfs, it would be a great fit for a content addressable storage system.
- "discovery": finding packages across different servers, this is such a game changer. Traditionally you would have 1 or 2 repositories configured and packages are fetched from those, if these servers do not have the right package or versions or are offline you are screwed (having to manually find alternative mirrors or finding copies of the package elsewhere). In ipfs, if your peers don't have a file, it can just ask the network for available "mirrors" and even users who have a copy and contribute to the network can provide you their copy automatically.
On Skynet, pinning doesn't mean hosting the file from your machine, it means paying a bunch of service providers to host the file for you. When you pin content to Skynet, you can turn off your computer 5 minutes later and the data will still be available globally. Just like IPFS, data is content-addressed and anyone can choose to re-pin the content.
Service providers are held to 95% uptime each, which means doing something like 10-of-30 erasure coding can get you 99.99% uptime on the file as a whole. The low uptime requirement for individual providers dramatically cuts costs and allows amateurs to be providers on the network. The erasure coding algorithms ensure data availability despite a relatively unreliable physical layer.
The other problem Skynet set out to solve is performance. In our experience with IPFS, if you aren't talking directly to a node that's pinning the content, IPFS is very slow. We've heard stories of lookups taking greater than 10 minutes for data that isn't pinned on a major gateway.
Skynet uses a point-to-point protocol rather than a DHT, which not only makes it faster, it's also more robust to abuse. DHTs are pretty famously fragile to things like DDoS and active subversion, and Skynet has been designed to be robust and high performance even when malicious actors are trying to interrupt the network.
Other than that, we've tried to make sure Skynet has feature parity to IPFS. Content-addressed data, support for building DAGs, support for running applications and static web pages, and then we've added a couple of elements of our own, such as APIs that allow applications and web pages to upload directly to Skynet from within the application.
https://siasky.net if you want to give it a try for yourself.
There is no requirement to buy or receive coins to use IPFS. And multiple different implementations of persistent stores can use the same IPFS network to ensure durability.
This means that FileCoin, other coins, static hosting providers, self-hosting can all coexist and strengthen the same network.
Every other distributed content addressable file store seems to only be interested in supporting their own coin only. For that reason alone, I find IPFS the most interesting.
The tendency of funded / for-profit software is to eschew layering, so it's easy to try out but ultimately ill-fitting and inflexible. This is a core problem of capitalism with IT.
IPFS, libp2p, IPLD, Filecoin etc. all resist that temptation, and I think it will help them greatly in the end.
I am wary of crypto coins as they tend to have wild swings in value over time as they are used primarily for speculation... can I be sure that if I upload something today worth $2/month, in 2 years I will still be paying this much or less?
If you are doing business in India, you get payed in rupees. If the workers are in India, you just pay them in rupees. If you have to exchange currencies you end up with several types of friction that just create headaches and potential losses. If you periodically cash out or inject cash it's easier to deal with than on every transaction.
Denominating file replication services in a "coin of the realm" just seems like the same sort of rationale.
One of the problems with capacity planning is that you get punished for being wrong in either direction. You bought too much hardware or not enough, too soon or too late. With an IPFS or a Skynet, putting your hardware online two months before you need the capacity at least affords you some opportunity to make use of the hardware while your Development or PR team figures out how to cross the finish line.
A blockchain gives a way to make a payment and provide identity/ecrypt functions to keep the resource private while it is active.
Yes, one could create a system that attaches other authentication (user/pass or oauth), but then one has to create/connect a payment system which then uses that login information in conjunction with credit card information. To sell a product online taking credit cards requires an excess of 10 pieces of information that must be provided by the user.
In the case of compute resources, I may want to deal with 100s of providers to host my resources (blog, images, video, code) and I'd need to use an intermediary if I wanted to be efficient about it.
With something like Lightning payments, a system like this can provide resources without the need for a "signup" process or intermediary.
> can I be sure that if I upload something today worth $2/month, in 2 years I will still be paying this much or less?
What does the future value of a fiat currency have to do with the current rate of storage on something like AWS? Would those prices not go up if the currency was undergoing deflation? Would you not have paid less integer values before the deflation? Would you not pay more integer values after? Where does something like AWS provide cost protection, other than spot instances?
If you create a crypto contract and put the funds in escrow, then there is ZERO ways for the cost to go up over the life of the contract. Other than a bad actor scenario, which is why having multiple providers is the way to go.
Why multiple providers? With no-features data storage, S3 can easily provide that without an intermediary and in a pretty well automated way.
> Would those prices not go up if the currency was undergoing deflation?
Unless you're in a country with very unstable currency, the exchange rate change will be very slight. Sia changed ~60x over the last 2 years, which is significant.
The price of course won't change during the contract, but what happens after is not trivial to plan for.
Crypto is the only means of payment I am aware of that does not have a centralized middleman with the power to deny a transaction.
Decentralization aside, there are efficiency gains as well. Every transaction on the Sia network is point-to-point, and in some cases we've had nodes that average more than 1 million discreet transactions per day for over a month. The total cost of doing that was something like $100 (including the cost of all the resources bought with those millions of transactions), I struggle to imagine a traditional payment system providing that kind of value.
There's also a lot more flexibility to innovate. For example, every single one of our payments is accompanied by a cryptographic proof that the accompanying storage or computation (not many people know, but the Sia network does support a limited form of computation) action was completed correctly. The payment and computation are fundamentally/cryptographically tied together in way that we could not reasonably achieve on a traditional payment system.
Also do you have Swift bindings for use in an iOS app?
We don't have swift bindings but other developers have been able to make iOS apps without much trouble - the API is pretty clean.
Example: https://github.com/tetek/skynet-ios
Meanwhile, we're still waiting for Filecoin to launch, and networks such as Sia have seized that opportunity and created great things like Skynet [1]. Skynet itself still has some overhead if you want to ensure data persistence and availability, but the cost is orders of magnitude lower. In addition, new layer 2 providers have emerged to address those gaps, such as Filebase [2]. They provide S3 compatible object storage that is backed by decentralized cloud storage. You get high availability (of the storage layer), geo-redundancy, and less than S3 pricing out of the box.
It is this type of offering where we are going to see the most impact and adoption as the underlying technology not only makes things more efficient, but cheaper too.
Just a heads up from someone who's been bitten.
On my network of random Pis and other stuff, I use only high-endurance SD cards which can withstand lots of writes have durability much closer to SSDs.
A thing people tend to forget is temperature. Keep them cool and they have higher chances of surviving longer.
I just had it happen so I figured I'd bring it up just in case.
I absolutely agree it's hard to go head-to-head with s3 in the short term --- this is why I am most excited about sharing sources. Once the ecosystem is bootstrapped, it will make more sense to use IPFS for binaries too. (e.g. if you wanted to build some fancy multiple build farms and reputation system.)
> https://www.softwareheritage.org/
I do really want to work with then!
- IPFS as CDN means software heritage can be "seeder of last resort"
- Original authors uploading to CDN, using IPNS or similar for git tags/versions, should make it easier for software heritage to archive the code in the first place.
No, none of these distributed P2P networks (that I've seen) do. The problem isn't just building a DHT (kademlia-based networks have existed for years) the problem is incentivizing people to seed - ideally people with high-bandwidth and massive amounts of storage who are seeding data that people want. In other words you need to build an economy on top of the network.
Cryptocurrency could be used for this, so long as its a cryptocurrency that supports instant micro-transactions (ie. you don't want to be writing to a blockchain every time you download a 10KB gif). So maybe someone will get around to building an IPFS-clone but on top of bitcoin's lightning network or something like that. Clients would need a way to decide which peers to send requests to based on who's offering the best speed/reliability vs price. Servers would serve higher-paying requests with higher-priority and would drop any requests that are too stingy to even cover bandwidth costs. Using the network wouldn't be free but it would be extremely cheap and fast and reliable.
One problem this doesn't solve is getting other people to backup/seed your data for you. An idea I think would work for that would be a prediction-market based reputation system for peers acting as storage hosts. That is, peers could advertise themselves as storage hosts, you could upload your data to them (for a fee) and they'd give you a cryptographic receipt promising that they'll still be able to deliver the data at some later date. People could then make bets on whether a host will fail to uphold any promises before a certain date, and the betting odds would be a measure of a hosts reliability. Clients that are uploading their data to the network would take that reliability measure and the price into account when choosing hosts to upload to. At any point a client could publicly challenge a host to provide proof that they still have the data, and if the host fails to provide proof the bets would close in favor of the punters who betted against them. Otherwise, once the bets expire, they close the other way. This would all need to be built on top of a blockchain though you couldn't use bitcoin for this until/unless they add support for covenants or sidechains.
Also, I'd argue more important than even having the hosters is having the content addresses. We need well-known immutable data for people to want in the first place. And traditional system bury data under so much mutation/indirection that it's hard to know what that content is, or that content-addressing even exists.
I highly recommend https://www.softwareheritage.org/2020/07/09/intrinsic-vs-ext..., which is about software heritage trying to get the word out to the larger library/archival/standardization community that content addressing and other "intrinsic" identifiers are possible and desirable.
Git and torrents I think is the best counterexample to the above, and there is probably more legally-kosher git and bittorrent usage, so I am especially bullish on Git hashing being the bridge to a more distributed/federated world.
Can someone enlighten me as to real-world examples where actually reproducible builds are critical?
One problem with NixOS currently is that certain dependencies in our package tree are very painful to change. If we touch glibc, we need to recompile 60.000 packages and it takes A LOT of compute power to do that.
With reproducible, content-addressed builds we can do things like early cutoff optimisations; making these changes less painful. SImple example; if somebody just changed a source code comment in glibc; then we get the same build artifact, and can skip building 60.000 packages.
If you do this at the source code level instead of the package level (like Bazel) then somebody can change how glibc does domain resolution; but packages that don't depend on that don't need to be recompiled either.
This is wonderful because it's totally sound. All efficiency methods (basically, hiding things, as the other comment says) is left to the build plan itself.
Switching to Nix for batch jobs is switching from DOS to an OS with actual process isolation. A complete paradigm shift.
Of course, since builds are reproducible you can catch this if you build it yourself and compare the actual content-hashes.
The caching model is about when the drv changes ((drv, output) is the cache key, basically), the trust model is one one can download something to avoiding build to fill in the value for the new cache key.
Note also that with the new floating content-addressed derivations, the trust stuff is much easier to think about.
The example was given for linking but this can be done for many steps of the build process.
Most people do not compile their own software, they used signed binary distributions being given to them. If you have reproducible builds, third parties can more easily verify independently that the code being distributed is the same as the code published in the open source repositories.
This doesn't just keep dev teams honest, it also provides defenses against situations like those where a hacker gains control of the website being used for distribution. I believe reproducible builds have helped to catch this real life issue at least once in the cryptocurrency community (with Monero).
To me, that's enough to justify their existence.
If you push something to CI that needs some new dependencies, the CI machine will build them and when you want to get a local dev environment or deploy, the binaries will be fetched from the CI machine. We can also send the binaries between machines to save compile time. This is possible because we have confidence that it doesn't matter which machine built them because the environment is so strictly controlled.
Reproducible builds (as in bit-for-bit) means I can trust a (secure) checksum of a binary I downloaded somewhere, let's say from IPFS. This is defense-in-depth, adding another layer of security to prevent sabotage.
Reproducibility means the whole system can reliably be built from source and I don't need to manually make sure the system has the correct constellation of implicit dependencies and their version numbers. This saves time and money in deployment and will pay off in a very big way 10 years down the line when that code is still in use somewhere.
NixOS doesn't solve reproducible builds as-in "bit-for-bit identical binary distributed files", but it does solve "same input" -> "functionally same system". This is the core idea of NixOS and what makes the package manager and the spawned ecosystem (like Guix) quite neat. I also think this isn't any more controversial then say having your infrastructure reproducible with terraform and ansible/salt. It gives you machines as cattle, and not pets, with the added flexibility that gives you. NixOS is just a complete package where in other systems there are several components accomplishing the goal.
As to why reproducible builds is important is to ensure a strong connection between the upstream source code and the distributed binary build. This gives you confidence that everything going into the build has been declared and that the binary can be reproduced bit-for-bit identical if you so wish.
This can be both important in a supply chain process. I recently saw tailscale provided pre-compiled binaries without signatures. What does those binaries contain? They are not signed, so the releases could very well be unauthenticated and someone could have compromised the server distributing the binaries. So I tried reproducing the tailscale binaries without much success as the build process is proprietary.
https://github.com/tailscale/tailscale/issues/779
So what does the tailscale binaries contain? I can't get a bit-for-bit identical binary without putting into a great deal of effort so for all intents and purposes they can very well be proprietary.
Now I don't think the tailscale people are malicious, nor that the binaries have been replaced. But it's a real-world example I recently went through.
When you have binary reproducible builds such a thing would immediately be obvious and hell would break loose if they attempted it.
See these:
And as an outsider I'm unable to read the RFC and understand what is going on and what tangible issues it solves for Nix.
Until that ideal is reached, you could call "leaving it to the plan writer" a cop-out, but what else is there to do? It's the stop-gap.
The benefit of the new system is that all store paths can be content-addressed. That means the inputs of all build stems can be content-addressed, whereas before they might also be "input-addressed" paths whose contents are also a matter of trust. This means one takes "small step" rather than "big step" trust steps, and also that upstream non-determinism won't "pollute" downstream builds.
However one problem i never see solved from any "normal" distros is reproducible systems. Hypothetically if i used Nix as a Desktop my config would be bulletproof. .. but then i'm going through a fair amount of work configuring everything when, as established, i want none of it.
So i (as a user) seem to want some middleground between no effort installations / configurations of my Desktop, with reproducible snapshots/states/configs/something.
You can use Nixpkgs on other distros/OSs, so maybe that is enough. But is there a better way?
And yes, the fact that I can configure all of my systems with a shared set of configs and have consistent and reproducible environments is fantastic.
I wrote up my thoughts about the switch here: https://kevincox.ca/2020/09/06/switching-to-desktop-nixos/
While i've dabbled in Nix in the past some things felt more odd to me than perhaps they should have. Primarily the fact that it felt like all these configs were spread out and i didn't understand how Nix wanted me to version them.
Flakes (from early tests) seems to make this very clear. As it starts from a repo, so i'm attempting to make all my configs, including Home Manager (of which also feels weird lol) start from the Flakes installation.
Flakes also solves the reproducibility problem that i didn't get why NixOS had. So far it's really neat.
I would imagine that you could use Home Manager on a non-NixOS system to at least create reproducible configs for the programs you use, although the OS as a whole would of course still be non-reproducible. However, I do not know how well Home Manager works on non-NixOS systems.
As you mention, just using Nix itself can be sufficient to get a reproducible set of packages on your system. I recall reading a blog post about someone who does this on both Ubuntu and MacOS.[1] The way this person does it is interesting because it’s more sophisticated than spawning the occasional ‘nix-shell’ or something. For example, they get the benefit of Nix “generations,” with a new generation being created each time they modify their declarative config files.
[0] https://github.com/nix-community/home-manager [1] https://www.nmattia.com/posts/2018-03-21-nix-reproducible-se...
* Config files aren't always stored in text (e.g. KDE Plasma, Gnome)
* It's all-or-nothing usually, unless you can find a way to make the home-manager config your "base"
* Changing settings isn't integrated, and requires editing/rebuilding/reopening (e.g. I can't just hit Ctrl+ to increase font size in VS Code, I have to edit a config file, run home-manager switch, then usually reopen)
* Plugins, especially things like extension stores, are a hack and a half. You have to hunt for hashes, then change it every time your old version falls off the CDN.
It's a nice idea, in theory, but it only works in a vacuum.
Edit: Oh, and obviously I keep the playbooks+roles in version control.
If anyone wanted to block a file in IPFS, could they generate node ids close to the filehash and return empty peer lists?