File Systems: The Original Hypermedia
jon.work
jon.work
It kind of has. URLs have the notion of paths, which are obviously strongly associated with the notion of file system hierarchies. People sometimes (sometimes accidentally) put their file systems directly on the web, see DirectoryIndex in apache for example.
> Why isn't a website just a remote directory on someone's computer that we can explore via a file browser?
Well now we are getting into the meat of it. To be a hypermedia requires the presence of hypermedia controls in a media. Hypermedia controls can be as simple as links, but the web introduced more sophisticated controls such as forms, and allowed HTML authors to specify more significant interactions beyond click-to-link.
IMO the uniform interface is the most interesting aspect of hypermedia, and that really emerged post Web. I like the authors concept of a file explorer enhanced with hypermedia ideas though and would be interested to see more details on it.
It's one of the things I love about IPFS, it has native directories along with fuse services, so you can just `cd /ipfs/...` and browse around. That has a lot of beautiful side effects in that you no longer need .zip to package directories and for a lot of things you no longer even need to download anything, you just access them directly via your file system.
Especially with FTP being removed from browsers, we could really need a proper official successor that can act as an online file system.
I one time tried to make a form that works with iphone auto fill just for name and address fields. Apple insists putting the house number on the end of the street name. House numbers may have multiple letters and streets are named after people who may have a single letter second name. Boulevard Peng Da nr 1 vs Boulevard Peng nr Da 1. This doesn't work.
During corona I needed a document to work my night shifts. I received a pdf with some form fields. After filling those out the document changed it self into a static permit. I had never seen such a thing before.
When writing blog software I ponder organization a good bit. My conclusion was that depending on the size, quantity and nature of the writings different methods would work best. however! As it grows it is hard to tell which one is currently the best formula and it is very easy to maintain the full set that I define as: Hierarchical tree, Categories, tags and search.
How to do the hierarchical tree is obvious.
A limited number of Categories that should be defined in advance (if possible)
You should have as many tags as possible. This comment could be tagged #history #writing #organizing #ideas #search #link #interface etc. The user can be exposed to a small sub set of sufficiently populated tags. The tag pages can be sorted by the amount of tags per word.
Search is just search but could have topical filters and use all of the before mentioned but could also take one (or more) articles as queries (and list the results under the text)
How and when to stitch on an LLM I don't know.
You just need to rethink the form, which is also an inheritance from print media, as a possibly fuzzy dialogue. Incorporating multimodal input, extracting information from uploads, etc., is a huge accessibility win. This is totally achievable right now, it’s just a matter of viewing the "form" as an interaction designed to fulfill a predetermined goal. What actually complicates forms is handling conditions, state tracking, and synchronization, which can become arbitrarily complex. This sort of dynamic behavior can never be fully standardized. So special cases are always going to need special solutions.
> Hierarchical tree, Categories, tags and search.
I think it's important to highlight the distinction between exploration and retrieval. Tags and categories are perfect for powering indices and lists, enabling opinionated exploration interfaces. However, when it comes to open-ended retrieval, arbitrary conventions don't really help. Retrieval requires the searcher to have some prior information on patterns, which are media dependent.
To make the retrieval problem more manageable, information needs to be structured in a predictable manner. Patterns are powerful tools for enabling and enforcing memorization. For example, if I want to retrieve a specific academic paper, my search model will align with the typical information pattern of academic papers. This pattern encourages me to memorize key metadata, such as the publication year, author names, institutional affiliations, and keywords in the title. These elements form the standard metadata body for this type of content.
With some js I could easily add field sets and sub sets. For example click add address, add house numbers, add additional employee/function/phone numbers sets, they can each have 100 email addresses and 1000 websites etc
xpath isn't sql but if the data looks this much like xml?
Converting posted values into table rows and back makes me feel like I'm doing something I'm not supposed to. Tables do not fit our mental model. What is the right solution if you only have one customer who has 6 phone numbers? The enterprise thing to do is add a table with phone numbers and give each number a number. lol?
DB performance is of course godly at scale but if it is just one form that lives on the users computer.... it might not be a nail?
> Hierarchical tree, Categories, tags and search.
It's a fun puzzle, no idea how far we are solving it. We should probably record what one looks at and have a button to mirror a publication locally.
I've often pondered how discovery can be pushed so that things of interest to you may gravitate towards you (and to those with similar interests). 100% accuracy at that would be a game changer. Someone out there is writing precisely that what I need or want to be reading.
But it needs to be dynamic. The ordering on the front page is dynamic. The vote counts are dynamic. What does the voting? These are all easier to have a database with values that query and then render the page.
The other issue adding new content. How would someone post a comment? How and where does it get written? How do you make sure they write to right place? How do you make it easy to use as typing in box and hitting button?
Dynamic responses are the special sauce of the web, they are why it is a success. Without it, it would be good-looking Gopher.
Hacker news is a web application, with a client-server architecture. Indeed would be impractical to replicate with the file system model.
Still, I think it would be useful to have standardized hypermedia documents. It would allow for content that's naturally multimedia to be much more easily handled and distributed. I find it super weird that we need to create a 'website' just to have a multimedia, responsive document. It practically must be hosted on a server because nobody sends HTML around. Mind you that I used the file system just as a metaphor for what an offline-first hypermedia document model could look like.
The remote filesystem needs a server. HTTP is a much better protocol for serving multimedia documents. The remote file system protocols are slow because they are based on RPC and make lots of requests.
It is dangerous to use filesystem for sharing cause have to carefully limit write access. None of the current filesystem protocols do that well.
If you were designing something new, it would probably end up like WebDAV or S3, an object store that can be used like a file system. There is a place for distributed storage, like IPFS but better, but that wouldn't be a file system.
There is a standard for hypermedia documents, HTML. There were attempts to make it more semantic, like XHTML 2.0 or using RDF, and they failed. There were attempt to transform XML, XSL, and that failed.
People send HTML around, they upload it to servers like I did a few days ago. But HTML is usually generated, for me because Markdown is easier to edit and can use templates.
I think you're failing to see the point. Having hypermedia documents follow a file system model does not imply that you would need to basically give up hypertext protocols. I'm sure you’ve at least once visited an Apache directory index on the web. Heck, just type file:///Users/ianburrell in your browser, and you'll have a hypermedia document modeled after the file system. Take that concept to its ultimate conclusion, and you'll see that a dead-simple hypermedia document can be built without resorting to any new technology or infrastructure.
I would have to think a bit to come up with something that could reasonably match the mature centralized architecture. First thought would be that if I like your comment I could chose to seed your most recent GB of comments. They would load faster for everyone for each who seeds them.
On a higher level it's fun to notice that HN isn't suitable for developing anything. People are just talking for talking sake, small nuggets of information are tossed around that may be useful to the reader any place but here. It is quite preposterous actually that this kind of architecture is even a thing. I have higher expectations from scribblings on a napkin among drunks in a loud bar.
When I first got on irc I had the audacity to ask people to do things for me. I even used private messages for this. Everyone got really angry but I disagreed, I still do. irl if I need a drill ill go ring the door 5 houses down the street and expect them to lend it to me. If some stranger asks me to dig a swimming pool for them ill show up with 10 friends and go do it. People who don't normally function that way should be offered assistance until they've so abundantly made use of it that they too will lend a hand. Online the opposite is happening. People have culturally rendered themselves to be quite useless.
Imagine for laughs trying to architect and build a house on Twitter. Even GIT is pretty useless. If it cant get an 8 year old to contribute in 5 minutes a tool has failed. Sure, all parts of a project might be extremely complicated or dangerous but that is just looking for excuses.
Therefore, I would argue, everything that doesn't directly contribute towards setting or accomplishing real world goals should live behind some imaginary line in the sand and be treated like useless content. Educative or inspiring does count as useful but only barely.
Compare that with print media. Want to make a memo? Fire whatever word processor, write it down, export to PDF, share anywhere. Done. If hypermedia was easy like this, do you have any doubt that people would put it to good use? Do you realize how PDF might be overused simply because of the durability, simplicity and fidelity traits it has?
It's not a law of nature that hypermedia should exist only in the context of browsers. Neither that it has to use HTML, CSS and JS.
Your example could be done with directory of media files and metadata in JSON. JavaScript generates pages from the JSON file. Browsers can serve from the filesystem. It would need to be static since updating the JSON safely is hard since filesystem don't provide atomic guarantees.
If you wanted it writable or larger, you would need web server running API for accessing something like SQLite for database. Running a web server is super easy these days, single command.
Instead of thinking about how would remake hypermedia, come up with a JSON metadata schema that can be dropped in directory of multimedia. Then have JavaScript to render it. There are lots of people, me included, that would like that for our music and image libraries.
Well, you're damn right! That's almost exactly what I did with my research! Perhaps to your displeasure, it's not JSON, it's something else, I needed something easy to write by hand. Plus I wanted to be have composability and a lot of other traits. Basically it's a novel data structure.
You can check a sample here https://jon.work/og/index.fifo.txt
P.S. Just to be clear, if a given information model has stable constructive and reproductive characters it can be considered media. And if it's multidimensional and multimedia it can be considered hypermedia (by definition), so doing what you suggested and guaranteeing the mentioned properties is precisely what "inventing hypermedia" entails. There's not only one possible hypermedia, it's an open concept, category. You can have as much renditions of it as you're capable to invent.
It is sort of a law of hypermedia that different resources are different files, though, isn't it? I guess you could make a file format that embeds images and fonts and HTML in it, but why bother when we have PDF?
My students at university (Gen Z) have no concept of the “file system”.
We grew up in a time when understanding file systems in terms of “a system for organizing your files” was not optional. Gen Z has grown up in a time when their data was a Google Drive search away from their fingers.
In the end, I actually agree with this! I have also been thinking about filesystems, which are trees of of dictionaries (directories) and blobs (files), and that there are many other examples of tree-like data. Data structures in our programs are tree shaped, perhaps with references/pointers to other parts of the tree. JSON is a tree of dictionaries (objects), arrays, and data (the primitive string/number/null). The arrays are ordered, much like the content inside a HTML/XML block.
I agree that adding an "array" style of directory to our file systems would be really cool. I've been toying with the idea of writing a FUSE driver that holds some structured data (possibly including arrays) and just converts the (integer) index into a string. The idea is that you could e.g. view and edit some JSON tree with the file explorer. And not just JSON - basically any piece of structured data that we have in our programs can be "viewed" as a FS this way (e.g. just convert structs into directories of fields). It could even be a pretty cool and universal debugger - a breakpoint could make the program pause and serve a FUSE driver and later continue when it is unmounted :)
And yes, exactly what follows from this is some program could be written to open some "directory" and render a "document" based on the contents. The filesystem supports links, so we have the "web" like experience.
The "document" angle does require adding a kind of directory with ordered array semantics rather than dictionary/map semantics. It's the first missing ingredient listed in the article. Though some filesystems use sorted dictionaries (b-trees or whatever) for directory maps so you could maybe hack this ordered semantics in that way.
The second missing ingredient listed in the article is the hypermedia part. I mean my computer is actually OK at inferring if a file is a movie or photo or text document, so we kind of have a way of dealing with that, too. The "blob of bytes is a narrow waste" thing is quite powerful. That said, sum types could be useful to demark different kinds of "stuff", and there's no reason an implementation of this idea couldn't support sum types as part of its data model.
You're just a great understandeour
> I agree that adding an "array" style of directory to our file systems would be really cool.
I think this is sort of a low-hanging fruit people have slept on. We've proved list-based systems are extremely versatile for data structures (s-expr) and programming (lisps). What about for media in general? Everywhere I look I just see lists, with very minor stylistic distinctions between them. Of course there're abysmal infrastructural differences between chats, feeds and what not, but it does not invalidate a universal list-based frontend, similar to what you developed in your comment.
I didn't really think about (linked) lists much, only (flat) arrays, but maybe that's possible already with files laid out like:
/head /next/head /next/next/head ... etc
or whatever.
I'm not saying its _ergonomic_, mind you. I'd like my file viewer to lay out the list flat, for example. There might be technical limitains with the length of the file path. Etc.
I agree that putting a playlist-like concept into, say, the filesystem would be an extremely interesting idea but I think a big danger is running into the same problem as hardlinks and symlinks. This problem is that if a file is "present" in multiple places (or playlists) deleting/modifying/moving it can have unforeseen consequences and it's hard to reason about (and if you copy the file now you get to invent a way to track different versions too!). I think this is also holding tagging filesystems back.
I'm currently writing a non-hierarchical FUSE filesystem and have been thinking about this list-directory concept but I'm still not completely sure how it would work, especially since I need to remain backwards compatible with the POSIX interfaces. Will probably have to just try it out (xattrs to the rescue?) and see what sticks I suppose...
A linkdump of interesting somewhat related stuff:
- https://newsletter.squishy.computer/p/knowledge-structures
- https://newsletter.squishy.computer/p/all-you-need-is-links
- https://thesephist.com/posts/search-vs-nav/
- https://karl-voit.at/2017/02/10/evolution-of-systems/ Especially "Information-Centric Systems"
- https://www.theatlantic.com/magazine/archive/1945/07/as-we-m... A classic 1945 article cited as inspiration by Ted Nelson, Doug Engelbart and Tim Berners-Lee.
- https://en.wikipedia.org/wiki/Content-addressable_storage
- https://www.nayuki.io/page/designing-better-file-organizatio...
Surely we can discuss (computer programs can interoperate on) something ordered other than raw bytes (ie the contents of a file).
xattrs I suppose would be the best way to do something backward compatible yet descriptive enough to be useful? I agree it’s a steep hill to climb.
To the author's take of using a file system as an interconnected 'web', we have networked file systems today, typically clustered though.
We've also had the _concept_ of a media-rich file system, like WinFS [as an overlay to NTFS], which was dead before it was alive due to the WWW.
Networking file systems is _complex_. All vendors would need to agree to a common export model on top of their preferred file systems. Or users would need a specialized partition/overlay developed just for this purpose.
FSes are great, but they're not fit for WWW. Without a control plane, they lack any tooling that makes the WWW better -- redirection, access control, programmatic execution of content (ASP.NET, PHP, CGI ...), etc.
Ultimately this would be a complex solution. Just like many don't simply "open up" their web server to any and all traffic to any and all content, a file system would need to be carefully partitioned the same.
The time for file systems as the vehicle for WWW content is long since past. We have better ways to do things, better caching mechanisms, better performance [through CDNs], better security mechanisms, and so on.
...not to mention, I certainly don't want to open my personal computer's file system up to the Internet.
There would have to be a big leap in evolution of file systems across all major operating systems for the author's dream to come true. I would certainly be excited to see it, but we're talking about allocating _talented_ developers to create a new file system and certainly an open source file system. Like many file systems, it would take years to become a trusted file system to host any content of value.
In the mean time, the author can always investigate WebDAV. Slower than dog shit, but it's available with every major web server.
I cannot disagree with you, you’re on point on everything when considering the file system as an OS component.
But if we entertain the thought of file system as a document model, or as a transactional data structure, it should come naturally that we can piggyback on the modern infrastructure, at the application level, to achieve the desired qualities.
This very website is an experiment on how this could be done. The main takeaway with my research is that we have much to gain if we leave presentation and layout concerns out of hypermedia documents, letting the client software decide on it, like we do with our editors and IDEs, choosing the theme and font we like, the information is the same no matter. To abandon the fetishism inherited from print media and to transact pure data is to make the web democratic. That’s precisely the recipe used by all social networks: standardized, systemic presentation of schematic payloads following a given ontological model. We need only to copy them with an open model
Never stop experimenting.
> file system as a document model, or as a transactional data structure, it should come naturally that we can piggyback on the modern infrastructure, at the application level, to achieve the desired qualities.
I'm having a little difficulty groking what you're trying to say, here. Can you explain it like I'm stupid? My mind immediately bounced to SQL storing binary data or a document DB of some sort.
> if we leave presentation and layout concerns out of hypermedia documents, letting the client software decide on it
Every browser engine does this today. Every engine developer has their own idea of how a standard should be implemented. Granted, this isn't end-user choice, which is probably what you're after. Well, sorta, you have some customization of fonts and [link] colors.
> To abandon the fetishism inherited from print media and to transact pure data is to make the web democratic.
It sounds like you would like some simple scheme that provided you the text and binary data (images) of the website which allowed you to manipulate them into the "newspaper" layout of your choice. Am I getting that correct?
What I’m proposing with the research is a data structure transmitted as text. This text payload is the equivalent of a file system volume or tree. Each line in this payload is the equivalent of a file. It’s only a semiotic analogy to file systems.
Given this is such a trivial structure, it’s easy to develop multiple rendering approaches that maintain an isomorphism between the text representation and graphical representation. It does not transmit layout information, the syntax itself is the higher-order layout.
You might wanna checkout the source of this article by appending index.fifo.txt to its URL.
It also is simply a blank page when browsed with UBlockOrigin blocking all the java script from executing.
You lost me there. WebDAV is nothing more than HTTP calls with some XML data with a slightly different syntax than the S3 API. There is no fundamental stuff in the spec that command the protocol to be "slower than dog shit" as a file transfer protocol. Please prove me wrong with another argument than: "the particular server implementation I tried was dog shit"
The web doesn't require "advanced computer skills". (Unless you use non-flexbox CSS alignments ;) It is fairly trivial to create basic HTML files. SSG + MD have removed a lot of the remaining obstacles. Most web sites are structured files, just with a "compiler" and possibly a database to store the files.
But what they still do require is the ability to reason about structured data and its best configuration. And that is the truly hard problem, ever since Ted Nelson first talked about it.
It also requires us to reason about how to best make that data consumable for humans. It doesn't just magically "arise from the structure", as much as I wish it did. The web site is a clear example - the lack of understanding how humans consume info, and what helps/hinders, leads to odd boxes around each paragraph.
I still agree with the fundamental idea. The more structure we can encode in an easily graspable way, the easier it becomes to impose structure.
But even then, the fundamental advantage of the web over hierarchical file systems is the non-linearity. And yes, correct, hierarchies matter, but the fundamental point the article misses is that there isn't just _one_ hierarchy. Wikipedia is a great example here - it fundamentally cannot be expressed in a meaningful way as a tree, even though it has many hierarchies.
And hierarchies alone are insufficient. We've now learned, thoroughly I think, that hierarchical taxonomies always break down. If we're given to snark, Linnaeus took a good stab, he failed. In more practical terms, the emergence of "tags" has shown that we need a way to have non-hierarchical cross-cutting data.
I think for a discussion of the subject, there's value in separating a few topics:
* Presentation. The author is right, HTML made a grave mistake including that
* Local representation. Again, agreement here, giving a file system structure that allows to infer meaning for later presentation is super helpful. (See point about SSG/MD)
* Organization/Navigation: Any sufficiently complex set of data requires several separate overlaid structures to help humans navigate.
* Human psychology: We're bad thinking about relation schemes beyond trees & grids. That means our organization schemes need to mirror them at least partially so we don't break our head. Corollary is that any sufficiently complex set of data needs searchability.
There's probably more. It's a topic that's been brewing in my head for a while, you're getting a very rough first draft, sorry :)
These are advanced computer skills IMO
> It doesn't just magically "arise from the structure", as much as I wish it did. The web site is a clear example - the lack of understanding how humans consume info, and what helps/hinders, leads to odd boxes around each paragraph
I appreciate the criticism. It's impossible to please everyone in terms of design, and I think your antipathy towards this particular style agrees with the general premises.
Regarding if disposition arising from structure is desirable or not, I think it's a matter of culture and habit. The time and complexity savings for authoring and publishing afforded by this model, for me, satisfactorily offsets whatever could be said that it misses in the aesthetic or funcional department, which can always be patched and improved. The positive feedback I had from interested users, all of them tech-illiterate, is what gave me the confidence to pursue investing in the research, and also made me realize that my insecurities towards its acceptability, which stemmed from sentiments quite similar to what you put forth as criticism, were mere whims. As far as experience and perceptions can be trusted, I believe serial multimedia has been proved as a viable format.
> And hierarchies alone are insufficient.
Not disputing that. The fact the document model is hierarchical does not mean the document system has to be. In fact it was never planned to be. There are many mechanisms in place affording hypernavigation, down to the design of the in-memory representation. Just haven't been implemented for lack of resources.
> you're getting a very rough first draft, sorry :)
I'd love to hear more. Feel free to ping me anytime if this is a subject you find exciting to discuss!
To set up? Yes. Absolutely. But that's equivalent to asking users to install a filesystem before using the machine :) Past installation, it's "write text. It shall appear"
Nobody needs to hand-edit HTML any more. It has become, for better or worse, the assembly language of information design ;)
> The time and complexity savings for authoring and publishing afforded by this model, for me, satisfactorily offsets whatever could be said that it misses in the aesthetic or funcional department
I'm curious. Are you saying the boxes are helping you author/publish (in which case, please say more!), or am I wildly misunderstanding?
> As far as experience and perceptions can be trusted, I believe serial multimedia has been proved as a viable format.
Absolutely. But it doesn't create meaning by itself. It's merely a well-understood and simple way of organizing information. (I should've added "linear" to "tree" and "grid")
And just having a pre-defined structure doesn't give meaning in general. You'll need to conform to it, and you need to deal with the parts that just won't conform. (soft/hard links exist to satisfy a need, if we want to go back to file systems)
> I'd love to hear more. Feel free to ping me anytime if this is a subject you find exciting to discuss!
I just might take you up on that ;) The topic's near and dear to my heart. (Alas, it is not my main occupation, so... feel free to ping as well. I might fall off the face of the earth from time to time ;)
Doesn't that prove it absolutely failed? I mean, you have to abstract it so much that, like you said, it's a compiler target at this point! That's precisely my reasoning when I said about needing a second system in the article. And, mind you, not only a second system, most of the time also a third or more! Might as well write a game engine with the budget needed for an HTML WYSIWYG editor.
> I'm curious. Are you saying the boxes are helping you author/publish (in which case, please say more!), or am I wildly misunderstanding?
Not the boxes, per se. I'm refering to the general approach of having independent elements that can be composed with no rigour or planning. This UI came out this way cause I ported the code for the data structure editor, which, for accessibility reasons, had a full-blown cursor around each element, that's why it retained the borders. It's obious that there should be a read mode with less clutter. But honestly, It does not worry me too much, for a proof of concept it was not worth to sweat every detail.
> And just having a pre-defined structure doesn't give meaning in general. You'll need to conform to it
Beautifully said. It's for precisely this fact (if you accept Peirce semiotics) that it becomes a cultural factor. You are compelled to think the way the medium demands you to think. And this is completely dystopian but also absolutely real, and serious! That's why design decisions in authoring interfaces are undeniably political. The abysm indeed looks back. One of the main reasons I started this research was to find a way to express myself in a more forgiving way, that better reflected my thought structure: loose. I can't bear the typewriter model, it drains all my energy and creativity, it smells of formalism and grandeur. Now, when I chat with people, my mind is free, I type at the speed of thought. So that influenced the whole design. I trust you can see it.
I think they meant for absolute novices. HTML is still very very easy to learn if you need/want to write it by hand.