IPFS support got merged into curl
twitter.com
twitter.com
This lack of verification is expected with HTTP, but not with IPFS. curl should verify that the resultant output conforms with the IPFS address or else just have users input the gateway/ipfs HTTP address as you always could.
curl can operate in a pipe mode and that adds additional complexity in respect to verification.
IPFS gateways can serve you in a manner that allows continuous(?) verification: <https://docs.ipfs.tech/reference/http/gateway/#trusted-vs-tr...>
This would in theory allow curl to block the pipe until it is able to confirm that a piece that arrived is verified or abort a tampered-with file early. This would take quite a bit of work to implement however - since it seems like there is no maintained IPFS implementation in C: <https://docs.ipfs.tech/concepts/ipfs-implementations>
Searching for “Merkle” on https://ipld.io/specs/transport/car/carv2/ gives no results.
There’s an intro to IPFS content identifiers here: https://docs.ipfs.tech/concepts/content-addressing/.
Why a DAG rather than a tree, though? I can't seem to find an explanation for why a DAG is needed — I would think a tree should suffice.
(But I'm new to IPFS)
From https://ipfs.tech:
> the innovation of content addressing: store, retrieve, and locate data based on the fingerprint of its actual content
Edit: well it seems that I have absorbed his downvotes, better this way ahah
P2P just flat out doesn't work for mobile devices too which are pretty much the entire internet userbase now.
Kind of like linking to the newest song, the link is static the content changes
I know my local ISP caches the Google homepage (I assume this is the reason).
Are there any ways that could be mitigated?
I don't have a great write-up published (only working code!) but if you know Graphviz, see https://github.com/bazil/plop/blob/main/doc/crypto.dot -- or study Tahoe-LAFS.
Of course, shared secrets scale poorly beyond individual users and groups of friends. But there's fundamental limitations there, you need to do access control by some secret knowledge.
Perhaps in 99% of pages you want the latest version, but I think in 99% of requests you want an immutable thing.
Generally speaking web pages are made up of vast numbers of immutable resources.
Bitcoin’s competitors such as Ethereum and Monero did not allow a software or discussion forum monoculture and instead have scaling block sizes, and still hard fork to add new scaling features (https://ethresear.ch).
Under 10 tx/sec is glacial, it makes Bitcoin useless for the people with the least access to financial services, and it seems starkly in opposition to Satoshi’s written comments about scaling when adding the 1MB block size.
I do think Bitcoin should be considered a failure if it becomes purely an exercise in hoarding.
I will say!I’ve been too happy with Ethereum and Monero to test Lightning, but I should. I had misgivings with its interface and availability requirements while reading of it in the past years.
Why not just curl the gateway then? You already could!
It’s a first step, though
At least the rule is invoked safely. Curl doesn’t do localhost pings by default; instead it checks for the use of an IPFS_GATEWAY environment variable or a ~/.ipfs/gateway file, and will fail with instructions, if neither are present
After all, curl already supports dozens of other obscure protocols: DICT, FILE, FTP, FTPS, GOPHER, GOPHERS, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, MQTT, POP3, POP3S, RTMP, RTMPS, RTSP, SCP, SFTP, SMB, SMBS, SMTP, SMTPS, TELNET, TFTP, WS and WSS…
One challenge of doing this correctly is that curl is intended to be a “one-off” call that downloads and exits without participating in the swarm the way a good BitTorrent client should. Granted, presumably this IPFS gateway solution also has this problem.
Also, the Internet Archive uses web seeded torrents to accomplish the problem of insufficient seeds rather than throwing them into the ether (non-crypto variant) and hoping for mirroring by wishful thinking.
Conclusion: Centralized, unicast-like support for p2p is possible and useful for download-oriented clients.
IPFS Desktop in particular: One shitty thing they did was auto re-opt-in to telemetry without asking the user. With that kind of sneaky bullshit, I refuse to run it. IPFS lost trust and is unable to communicate how their "solution" is useful or make it usable by humans who aren't IPFS Desktop developers.
P2P protocols like IPFS are built with the expectation that clients are persistent. They take a nontrivial amount of time to start up (e.g. to discover peers), and require continuous maintenance from the application to keep running (e.g. to chat with those peers and keep them happy).
These characteristics are incompatible with clients like curl. They're written with the expectation that connection setup is cheap and that connections don't require maintenance while they aren't being used. These expectations are all true of existing protocols like HTTP(S) or FTP; they fail with IPFS. And there isn't really any way to solve that without introducing another process to act as an intermediary -- which is exactly what an IPFS gateway does.
There are stable nodes to bootstrap a new node into the network. More initialization that if you were using http, sure, but it isn’t as against the grain as you suggest.
It’s almost as if I copy+pasted from the curl man page or something!
Imo that’s better than it being married to crypto. IPFS has its uses.
I like that they’re picky about who to let in their network… at least they were last I checked.
You need to buy crypto (at least someone has to)
It’s exclusionary and insidiously viral by nature.
Without being tied to crypto, the barrier for entry and usage is lower.
On top of avoiding all the politics around crypto.
I'm confused on why such a shallow abstraction was put into something present on every device, and why this seems to be such a big deal to the decentralized community.
Even with it's automatic gateway detection it's purely used to rewrite the URL, which seems like something the operator could easily do themselves.
I could easily be missing something here though.
<img src="ipfs://WHATEVER"/>
I wonder what changed?
From the above link;
>>The InterPlanetary File System (IPFS) is according to the Wikipedia description: “a protocol, hypermedia and file sharing peer-to-peer network for storing and sharing data in a distributed file system.”. It works a little like bittorrent and you typically access content on it using a very long hash in an ipfs:// URL. Like this:
ipfs://bafybeigagd5nmnn2iys2f3doro7ydrevyr2mzarwidgadawmamiteydbzi
Default behavior in the merged curl PR got adjusted and now the only gateway that is used implicitly is the localhost one. Using an external, potentially untrusted public gateway requires explicit opt-in from the user via IPFS_GATEWAY env variable.
FWIW, in recent years IPFS ecosystem made content-addressing over HTTP viable and useful. Standards and specifications got created. Verifiable responses have standardized content types registered at IANA.
For practical info, see:
"Deserialized responses" vs "Verifiable responses" at https://curl.se/docs/ipfs.html
"Deserialized responses" are designed to be used on localhost, "Verifiable responses" are something one would use with a gateway they don't trust
Client docs at https://docs.ipfs.tech/reference/http/gateway/#trustless-ver...
Server specification at https://specs.ipfs.tech/http-gateways/trustless-gateway/
I understand that implementing the IPFS protocol in a tool such as cURL does not make sense. But I don't really see the point of a fake support like this.
> Meaning: not only do you use and rely a total rando’s gateway on the Internet for your traffic. That gateway might even, on its own discretion, redirect you over to another host. Possibly run somewhere else, monitored by a separate team.
> I have insisted, in the PR for ipfs support to curl, that the IPFS URL handling code should not automatically follow such gateway redirects
I wonder what decision was made here.
I see there's some doubt here about the usefulness of this patch so I'll go over a couple examples as to why this is great to have.
First an example of the same image in multiple centralized places. Say you have a the same "cat.jpg" on 3 different sites (same exact byte size, same image). In a centralized world that would be: https://someurl1.tld/cat.jpg https://someurl2.tld/cat.jpg https://someurl3.tld/cat.jpg
Which link do you pass to someone when sharing that? It's going to be one of those, if any goed offline your image will be perceived as offline even though it's there on other sites.
In a decentralized world with IPFS we can do the same thing through gateways. The gateway itself is a centralized endpoint through which you access content on the IPFS network. Here however the content of "cat.jpg" is hashed to a CID (Content ID). Let's say the content of "cat.jpg" has CID QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxXfwfby9TpuuigE
An example to get this same file on form the IPFS network through a gateway looks like: https://gateway1.tld/ipfs/QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxXf... https://gateway2.tld/ipfs/QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxXf... https://gateway3.tld/ipfs/QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxXf...
On a first glance there isn't much of a benefit. You need a gateway url (a central endpoint) and you need to know the filename (QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxXfwfby9TpuuigE).
Now for the curl approach and why this is awesome. Your image (QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxXfwfby9TpuuigE) is accessible through any gateway. But when you pass this information to someone else, you don't want to tell them to use one specific gateway. You only want to tell them the content hash (QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxXfwfby9TpuuigE) and let them use their own gateway. You want that because it promotes the data to be propagated over the IPFS network and be better accessible.
The curl patch _prefers_ that you use a own local IPFS gateway. So when you receive a link like "ipfs://QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxXfwfby9TpuuigE" it would use the gateway it has locally. And yes, this is just url syntactic sugar as it's all eventually been rewritten to a full http-url. If you have a local node with this patch it will probably will rewrite it to "curl http://localhost:8080/ipfs/QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxX..." But you could also be using an IPFS node in your local network but not on your local machine, meaning it could also rewrite it to: "curl http://10.0.3.3:8080/ipfs/QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxXf..." (or whatever your network is)
But to you, as the user of curl, that all doesn't matter. You just do: "curl ipfs://QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxXfwfby9TpuuigE"
Granted, if you use a node that isn't a local node the curl won't find it and you'll have to manually specify it in your local "~/.ipfs/gateway" file or in the "IPFS_GATEWAY" environment variable. But these are one-time configuration steps.
Another thing i want to highlight is malicious gateways and verifiable data. This patch really should be considered as a low-level syntactic sugar to access IPFS, not as full IPFS implementation. One can build on top of this and do data verification. This is, at least that is my opinion now, not a place for inside of curl itself. (assumption here) You can do a streaming approach and verify it block by block with another application, which should be a valid usecase. Also it's very much in the philosophy of building 1 tool for 1 job. Curl is to get data from network resources, something else is to verify that that.
Moral of the story is that passing and IPFS url like ipfs://QmbjWyHqUyjxfq7KSiSLBi59CdpqwaAxXfwfby9TpuuigE becomes completely agnostic from central points. It becomes a local user setting to handle these urls. With IPFS still being young and still very much a niche technology, this is still somewhat of a hurdle to pass. But as IPFS gains more adoption like this, the hurdles become more and more streamlined to solve. At some point in the future people know that to use ipfs they install something that implements the ipfs protocol to use such links. Just like people know to use a webbrowser on http/https links, magnet links to be opened in a torrent client and mail links to be opened in a mail client (to name a few examples).
> #IPFS and content addressing is here to stay.