Webfs: A Filesystem Built on Top of the Web
github.com
github.com
The RFC is full of that kind of crazy gotchas, not to mention the overuse of “MAY” or “SHOULD” which will drive you crazy if you try to implement a client / server.
If you want to go further in insanity, just look at how crazily over engineered the locking mechanism is. I have no words for it.
https://tools.ietf.org/html/rfc4918#section-6
Unfortunately, even if you think you implemented the whole rfc correctly, your implementation will work with almost nothing as not that much implementations are any good in the wild. A useful WebDAV implementation must be full of vendor-specific workarounds.
It's interesting looking back but I think developing a standard has a higher chance of success coming from some dude's GitHub than it does with $1T of market cap behind it.
Of course it would still make sense for server to tell client about this (there are files/no files/I don’t know).
Maybe it's on the decline, but I'd hardly put it as something that "never really found its footing". It is still a decent way to do fileshares over the public internet without sshfs, etc.
That's not really what the link is, though. It's an adaptation layer to turn a random network resource into something that looks like a filesystem.
At the time, I worked for a large porn company and we couldn't host stuff 'in the cloud' because they didn't allow porn there.
We invested a ton of money into an Isilon NAS to store our image/video content and the best way to get stuff off it over HTTP was via webdav. Unfortunately, there wasn't a good Java client.
So, I built a simple proxy that would accept regular GET requests and on the back end, use webdav to retrieve content from the Isilon. In front of that proxy was our CDN.
Since then, Sardine has been the basis for quite a few other projects.
Now I see that Wikipedia also lists several alternatives to WebDAV too, but I think httpdirlist is good.
1. Thanks for the direct mp4 link.
2. Consider reducing your desktop resolution for webcasts. I can see that there are dialogues open. I can't for the life of me see what's presented in them.
https://en.m.wikipedia.org/wiki/9P_(protoco)
I've been kicking around similar ideas (nowhere near implemented) for a while:
https://old.reddit.com/r/dredmorbius/comments/6bgowu/what_if...
The name here comes from using web resources (referred to by url) as building blocks for a file system. I guess "Web for FS" rather than "Web as FS".
- NFS, particularly with the Solaris-originated concept of automounts over a /net mountpoint.
- Various virtual filesystems. Midnight Commander ("mc") offers several of these, including archive formats (tar, cpio, afio, rpm, deb) and remote (FTP, SSH).
- SMB/CFS/Samba
- Various FUSE filesystems, including again ssh, ftp, and others. These generally require specifying in advance specific mountpoints.
The notion of on-demand access to remote resources over protocols (e.g., http/https, or others), under filesystem dynamics, is interesting -- you can use any general tool, utility, or application for access, mediated through the filesystem by way of drivers, rather than a specific application (e.g., Web browser, FTP client, etc.)
There are numerous issues. In particular, applications tend not to respond well to remote resources disappearing, changing, or failing to return from change requests -- NFS's behaviour with nonresponsive remote hosts is ... notorious.
Consistency, availability, and partition resistance (CAP) are long-standing concerns, and there's no way to solve for all three. I'd add latency as another major consideration.
https://en.wikipedia.org/wiki/CAP_theorem
The general notion of managing and tracking changes locally, and pushing them to remote, has merits. I'm not aware of a "gitfs" ... though of course, one does exist, TIL: https://www.presslabs.com/code/gitfs/ The notion of using git (or other versioning system) as a mediator for remote/local revisioned access seems to have merits. Obviously not viable for very-high-variability systems, but adequate for many occasionally-modified resources.
I'm not sure if you're looking at using your WebFS itself as a publishing mechanism, though in general I think I'd recommend against doing this. For small-n peer-to-peer distribution that's probably workable, but for large-scale provisioning-and-request systems, relying on HTTP or other established transports is likely more sensible.
One area I've recognised as being particularly fraught is the whole notion of security and privacy. Providing unfiltered local access to remote resources which may change arbitrarily is a great way for allowing malware onto local systems -- your transport layer should probably implement some level of security and mounts deny direct execution of content. The fact that remote content could be copied to an executable mountpoint remains, and would make numerous attacks possible.
Similarly: access, update, write, and/or publishing actions all leak considerable information which could be of concern to specific users or organisations. Hash-based indexing (already addressed in this thread) being only one of several such vectors.
The point isn't whether or not these are protocols that are utilised on the naked Internet, but that they offer access to network services via filesystem semantics.
That is, rather than use a specific client or API to access remote content, or copying it locally as a separate step, you simply open a file in an existing application, or, within a program, using fopen() or equivalent operators. The networking is ... translucently ... handled in the background by the filesystem interface and/or driver(s).
The reasons SMB is not generally used or advised over the Internet are worth looking at, as this touches on many of the security / privacy concerns of any such service.
> If you have ever thought "x can probably be used as a file system"
...you might also want to take a look at Storage Combinators[1]. Not quite the same problem space, but abstracting away a bit from both concrete filesystems and other storage mechanisms to get to a composable abstraction.
Note: I am the primary author, and also taking a good look at WebFS for further inspiration... :-)
[1] https://2019.splashcon.org/details/splash-2019-Onward-papers...
Can another participant in the network tell what files are in my store? (like a party providing disk space, or someone intercepting my traffic)
All a storage provider sees are many small encrypted blobs, so the size of large files is not leaked either.
Where is the data for all of those IPFS hashes stored?
And what ensures that it will persist?
If you or a friend run the HTTP or FTP server then it will persist the data for you. IPFS doesn't incentivize data persistence so if WebFS is working on top of IPFS it inherits that problem. You could run WebFS on top of one of the storage networks and persisting your blobs would be incentivized.
WebFS is storage layer agnostic. Give it a Store, and it will give you a file system.
So if WebFS is running on a system with access to Tor SocksPorts, can Stores be onion URLs?
Edit: If not, one could presumably route WebFS through OnionCat's IPv6 /48. But that only works with v2 onions, which are deprecated. However, tinc works with v3 onions. And either of those gives you UDP transport.
The data encryption keys are generated using a secret and the hash of the data being encrypted. That key is stored in the reference to that data. This continues recursively to the superblock which is not encrypted.
What sort of content do you say Tor onions can't host?
"Tor onion" just means that a server is (ideally) only reachable as an onion URL, which is only accessible via the Tor network. There is the limitation that Tor only handles TCP. Otherwise, one can route anything over Tor. In my experience, that includes HTTP(S), FTP, Tahoe-LAFS, SSH, RDP, Mumble, OpenVPN and tinc. And others, if I spent more time remembering what I've played with.
What you're describing is a Tor hidden service. Hidden services are separate from the Tor relay network itself, which is what I thought you were referring to as "Tor onions".
Hidden services are optimized for confidentiality over performance. Using them for bulk data storage would place a lot of load on the relay network, and it's not clear what security problem this arrangement would solve.
As far as I know, "hidden service" is deprecated, with "onion service" the current term. And it does tend to get shortened to "onion". But I admit that it was confusing. Because relays used to be called "onion routers". Which is also more or less deprecated, I think.
The security problem is Stores being physically located and compromised, based on IP addresses found in traffic logs.
I mean, if it's securely encrypted, does it matter if others can see it? And indeed, if it's online, you must assume that others can see it.
My privacy concern is about IP addresses. So I'd want to use IPFS with Tor onions as stores.
w.r.t. quantum computing: it is possible for WebFS to use symmetric cryptography for all remote data. Although, many Cell implementations in the near term will likely use elliptic curves or RSA.
I was addressing jerry292's concern about access to the encrypted data in Stores.