The data model behind Notion's flexibility
notion.so
notion.so
> We came up with a frame-based approach. Most of the functionality particular to the various application types was packaged up into "frames": word processing frames, graphics frames, etc. These frames were then used as building blocks to make documents of the appropriate types, in a unified programming framework. E.g., a word processing document was essentially a bunch of text frames, one per page, linked together. (Doing this neatly was a big challenge - many subsequent efforts at building a component-based architecture (e.g. OpenDoc) have failed to take into account the top-level user interface requirements.) The result was that not only was most of the code shared across the document types, but the application was also truly integrated - the frames could be embedded in each other. E.g., you could plop a spreadsheet frame right into your word processing document. Text objects in a graphics document had a full-featured word processing engine behind them. The database form editor used the built-in graphics environment. Etc.
> One related cool thing we had was a "shared graphical context" mechanism: sometimes, stuff would wind up being displayed in multiple frames at once. E.g., maybe you're dragging an object across a page break in a document with multiple pages (like this). We developed a general architecture for displaying actions live in multiple contexts. Of course, a lot of this kind of stuff is old hat today, but it was new and exciting in 1989. Some creative programming was required to do these things efficiently on the hardware of the time.
> There were some cool features that didn't make it into the shipping product. For example, originally spreadsheet formulas were much more powerful: you could relate, e.g., graphical object positions and document properties to spreadsheet cells. So you could have the result of a calculation move objects around graphically, or vice-versa. (Further work in this direction led to a novel constraint-based programming paradigm called MOOSE, which I may resurrect some day...)
(From http://groups.csail.mit.edu/mac/users/bob/clarisworks.php#on... )
I've been kicking around the idea of writing a CRDT-based editor using this model.
FWIW I use Notion.
I got around to creating a data layer (p2p, browser-based, CRDT-backed) for something like this:
https://github.com/hyperhyperspace/hyperhyperspace-core
I'd be interested in collaborating on your editor
One thing I've been working on is rich structural editors for the scripting language used to build these local-first applications. So the environment becomes something like squeak or a lisp machine, where all application data is automatically synchronized over the network.
First off, let's discuss more general Wasm programs. Currently, there are three primary modes of execution for Wasm modules - browser, node, or native runtime. The issue here is the dependence on JS - If you want to do any high-level interop between the language you're compiling to Wasm and some external library, you're most likely going to have to go through JS, pulling in a tool like emscripten to enforce the 'ABI' (so to speak) at the boundary between Wasm and JS. There are proposals, such as interface types, which may alleviate this in the future, but for now it's either go through JS or roll your own.
I'm not a huge fan of having to pull in JS for something like this, which is why I've instead been working on an alternative 'ABI' for Wasm, which supports algebraic datatypes. The basic idea is pretty simple: we hand the Wasm module a large hunk of Wasm linear memory that is encoded in accordance with the ABI. Wasm modules must expose an `update` function that takes this memory and updates it in-place. The runtime then 'deserializes' the shared memory, extracts useful data, and calls out to external libraries, modifying the shared memory (once again in-place) with any results. We toss this `update` function in a loop, run it in a background thread, and watch it to ensure it doesn't crash, write out bad data, or hang. That's the rough idea, but the actual format itself is pretty straightforward (similar to messagepack); all we need to implement is a library to serialize/deserialize this shared memory representation for each language that compiles to Wasm. All programs operate on these algebraic datatypes, so to save program state, we can just serialize the shared memory and write it directly to disk. By making a module stateless, we can easily synchronize computation and memory.
I'm currently using a hybrid operational transform / CRDT model that operates on the algebraic datatype model expressed by the ABI. Tombstones for text are largely inefficient, so we use a diff-based patch model for text inside blocks, but CRDTs for individual block-level transformations. (Wasm programs also have the ability to choose how to resolve conflicts.) Blocks are synchronized with a protocol similar to DAT, but, as stated, the protocol has built-in support for multiple writers and conflict resolution.
So, with respect to Erlang/OTP-style process migration - it's certainly a possibility. If we exposed an actor-style library to Wasm programs through the aforementioned ABI, implementing message passing would be as simple as sharing an append-only log between two computers on a network - the data synchronization would happen automatically. Spawning a new process is as simple as creating a new block in the algebraic datatype tree, and giving a Wasm module control over it. It's all pretty interesting to think about, and I'm just skimming here, but that's the basic idea.
To render out the user interface, we do something pretty simple - in addition to providing an `update` function, modules may provide multiple `translate` functions that take an ADT and translate it into another. Each node in the ADT is like a notion-style block, so all we need to do is define a `translate` function that takes the program-specific shared linear memory, and translates it to an ADT that represents a DOM (not a browser DOM, just a DOM). There are a number of optimizations that can be easily made here.
TL;DR: It's easy to synchronize Wasm state if modules are stateless, program state can be encoded as a block-based ADT, we can synchronize this state through a mixture of Operational Transforms and CRDTs, GUIs are a translation of program data.
(Woah, that's a lot of text...)
\o/
By coincidence (with your question) Marvin later became Bob's PhD advisor. And at the time Claris works was being written I was working on a frame system with Marvin.
Have you investigated OT at all? I'm not sure CRDTs are a great fit for complicated datatypes like an editor, and their strong convergence properties don't get you much in something as free form as text.
Having built my own private-ish collaborative editor/note/todo/sharing/pm tool back in 2012 (that uses indent level permissions) I don't see how you could have a non "block" implementation?
I also don't see how they are having so much performance difficulty moving up the inheritance tree at the client level?
Does anyone else who has worked on these kinds of problems take a different approach? This being HN and a space I develop in I am actively curious if any other architecture are in use generally?
Are there approaches outside of blocks for tools with this level of granularity?
Really cool of Notion to share these kinds of "inside ball" deep dives.
What notion has done that is impressive IMO is made the editing experience enough like the MS-Word “inline” editing that people seem to understand it and use it without minding that it’s a block editor. The project I worked on back in the day wasn’t quite as slick, and there were a lot of users who got frustrated when things like copying text across multiple paragraphs didn’t always do what they had in mind (imagine a text selection from the middle of one paragraph (block) to the middle of another - that’s complicated to deal with).
Nice to see Notion take this model mainstream with such a nice UX.
I think I would need to redo the browser-based editor, because even if I solved e.g. cross-block selection across blocks of the same type, it's unclear how it should work across, say, a text block and a video-embed block.
I mean, documentation is mostly text, image and code. You can easily be document-based instead of block-based, like Confluence, Google docs ... you don't always need to be able to insert everything into everything.
I'm actually building a Confluence/Notion competitor, and it is document based. Being block-based makes a lot of things much more complex
You will never need to give a permission to a line of text
Some examples: context dependent rendering, having multiple "types" at once, etc...
> We later decided this wasn't useful or necessary and collapsed them into one thing. WTF was I thinking??
I haven't given extensive thought to where we diverge from Notion as the comment below mentions them moving away from it, but just wanted to say that your armchair analysis fits with our view.
And after reading this article, I realize it’s a lot like blocks... I guess the way I’d explain it is that I add a template element to the blocks so that untemplated versions with live data are like a black box to other ‘pages’, only when a user says ‘I want this visible again elsewhere’ aka templates do I extract out the non-sensitive bits (user identifies sensitive info) to be used again in other pages.
I'm just worried that so much engineering complexity is going to make things slow to move. We will see in a few years if that was a good idea or not
The only people that really want them to move and change things are investors, not customers.
Same with Dropbox. A folder that syncs is all I need from them as a customer. Everything else they try to do is just getting in my way.
> The only people that really want them to move and change things are investors, not customers.
I'm glad the Notion works for you, but I think we have a long way to go to reach perfection. To pick an example from the post, even indentation needs work. There's also always more work to do to make the product faster and easier to use.
I raised a bug for this, they said it was intended behavior. Like yea, ofcourse.
https://www.notion.so/dheeranet/Foo-a4b0f734fdab4ec4949a9b69...
What are you looking for that is not this?
Notifications in the system are still pretty hard to work with, and email notifications are no replacement. Comments on pages are unwieldy and mess with cursor focus in annoying ways. I've definitely submitted this feedback to support before and I can understand that not everything can be worked on at once. However, I still have faith that the team will work the kinks out eventually.
EDIT: To be less negative, I should also note that Notion still has probably _the_ best LaTeX / mathematics support out of any documenting system I've ever used. I vastly appreciate this, as it is often easiest to share complex mathematics via a shared Notion doc since I work remotely.
It's text editor is very weak but I have stronger than average preferences so I'll refrain from commenting but more importantly the desktop clients are just down-right awful with no rebindable keys (even the keyboard shorcuts "help" needs to go online to a webpage), no plugins or any sort configurations at all. It's fine if you use notion once a week but if you work with it every day you really feel constricted and I can't imagine what people with disabilities even do.
This is an interesting question. I actually think Notion's data model is much more conventional than the data stores behind document editors like Google Docs or Figma. Blocks are "just" (these quotes are doing a lot of work) rows in Postgres.
We use JSON for properties for flexibility and for user-defined property schemas. We could use an entity-attribute-value (https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80...) table as you describe for that, but such a table would complicate our caching techniques. It would also be enormous, but, it might be time for another think about that since we finished sharding.
I've tried using it for personal notes but it's speed and lack of offline prevented me from migrating to it. But I think it's the best choice for any organization that needs a collaborative docuement/wiki/Confluence-esque tool.
Not to mention that I think it has the best default fonts, spacing, and general aesthetic of any tool I can think of.
I need a system that syncs, works offline, helps capture structured-ish data, is quick+flexible, and has fabulous linking.
I discovered and am currently using Obsidian.Md. I use Ulysses to access and edit on iPhone+iPad.
I love that it is markdown-based, linking is super easy, and everything is fast.
Cons: doesn’t work super well with native macOS features (e.g. services), native app for iOS/Android is still in private beta, and I don’t like the separation between edit and view modes. Also it could probably use a stronger UI for database-like features (e.g. tags). I wish there were better options for inking on mobile, but that’s not really their thing.
Previously was a OneNote devotee, but as I get into more complicated community engagement work, I needed something that could cross-link and find patterns.
Mobile beta is pretty similar to the desktop app. Like even most of the plugins I use work fine on my phone. The only real complaint I have is that it's kinda slow to start on a phone. Not as slow as Notion, but still takes a few seconds.
> and I don’t like the separation between edit and view modes.
That's next on the roadmap after apps become polished enough to exit the beta. Public roadmap: https://trello.com/b/Psqfqp7I/obsidian-roadmap
I decided on DevonThink since it closely resembled what I used Evernote for but ditched it after just a month due to to bad iOS apps, weird default formatting issues, hostile developers (license covers max 2 computers, and refusal to support a few basic things).
Stumbled upon Ulysses and I've been using it since then. Excellent plain text/Markdown support (you can turn off the markdown formats you don't want to use), sensible default fonts and such, fast and native Mac app, great iOS apps, etc.
Google docs
Trello
Each page is some content (optional) with a set of arbitrary attributes of different types, e.g. date, string, number etc. When you create a table and populate it, you are creating a set of pages all of whom have the same attributes and the table is just a view of those pages. You can then create a calendar that's a different way of presenting those same pages. You can also deal with those pages directly and put content in, and reference them from elsewhere.
It's a really simple data model, and it's pretty flexible. We built all our processes on Notion initially and then moved them off when they got to an appropriate scale. I'm glad the API exist now, because then that approach can be incremental rather than a wholesale move. All the ways of displaying things - tables, calendars etc. are just displaying collections of pages.
The design decision to feel "default-text" is awesome - text is a great foundation for a "custom workspace" product. But, it's only when the bulk of content is in Tables that things get extremely powerful.
Once Notion has in-line text references, ala Coda, the sky is really the limit. I think Notion nailed the UX and speed / performance.
Even if it's slow compared to Apple Notes, it's lightning fast compared to Coda, where "Documents" are extremely heavy, and relatively isolated.
Notion has a clean route towards eating Coda's most powerful functionality ([an incredible charting and formula](https://coda.io/formulas)). That's the kind of thing that a company as well-financed as Notion can dedicate a team to, and make incredible progress on.
Similarly, They can eat Coda's in-line text references, so your text can live-link to information in tables. A matter of resource allocation.
Coda, on the other hand, started from an "easier application builder" approach, like Retool for Excel experts. That's brilliant and powerful, but the UX creates a bigger dividing line between consumers and creators.
Notion feels enough like a normal word processor that consumers start doing crazy things without realizing it.
This is the problem with Notion, in my experience at least. The docs are so limited and the maneuverability so unusual that it's very easy to do something like completely change the type of document accidentally.
This seem to be a really good use case for a NoSQL database. Am I wrong ?
http://patshaughnessy.net/2017/12/13/saving-a-tree-in-postgr...
You can break out the "block" model into several tables and represent it in a relational database that way.
NoSQL = NO JOIN?
Hope that helps.
We do lean very heavily on the TypeScript type system and try to make invalid states unrepresentable.
Which is pretty much the ideal scenario for a document store. The article describes Notion as being very strictly hierarchal
The underlying persisted data doesn't necessarily have to be a bag of KV pairs.
A block is related to its parent and descendant blocks.
These relations are suitably represented in a relational database, not a document store.
EDIT: In graph theory, a tree is an undirected, connected and acyclic graph.
When comparing a document store versus a RDBMS, in terms of suitability and appropriateness, the distinction is primarily along the lines of a tree, versus an arbitrary graph (by which I mean that an RDBMS is more powerful, and more general, but not inherently as optimal in either performance, “scalability”, or UX in the places where a document store makes sense.
More specifically, the way the article describes it, you’re not interested in “give me every block of type X” — you’re only interested in “given block Y, what type is it?”.
That is, the question is one-way, and fits cleanly in a hierarchal format of a document store.
The only question posed that operates in the reverse direction is permissions, though even that’s a little odd, since it seems to me it should only go “downwards” as well — a block’s permission scope is the sum of all of its parents, and you can store it there upon iteration.
> The underlying persisted data doesn't necessarily have to be a bag of KV pairs.
It doesn’t have to be... but it can be, and appears to be.
> A block is related to its parent and descendant blocks.
Right; the singular parent, and the multiple children. A tree.
> In graph theory, a tree is an undirected, connected and acyclic graph.
When discussing trees and graphs, I think it’s obvious a distinction is being made between a graph forming a tree, and graph forming a not-tree (more complex than a tree). When I say that a square is easier to encode than a rectangle, I do not mean that a square is not a rectangle, but that a rectangle is not a square — that a square’s more specific properties give us opportunity to simplify/optimize (I only need to store one length to represent it).
A database can encode a tree just fine, but that doesn’t mean it’s the best tool to do so.
There are other properties to a document store I don’t care for, and I don’t like them in general (like the implicit schema, and total lack of data consistency validation by the data store, and the fact that you often don’t truly have a tree), but representing a tree is what’s been described, and it’s exactly what they’re specialized for.
If you want to argue against it, you need to specify why you think this isn’t a tree, because I feel it’s quite obvious it is.
https://www.notion.so/Tree-breaking-3a90e2bcd2154f4fab06a3c7...
Breaks the tree model, as 'Complete Task' would need to have both 'Subtasks' and the page itself as its direct parent.
That said… it's mostly a tree, and there may be merit to optimising for that access pattern.
From there we started looking for a narrative. We extracted out the sections you see in the final post, and removed a lot of the superfluous technical detail so we didn't end up with technology buzzword soup; for example we cut discussion of Postgres, Memcached, etc etc, how we host the web servers; the kind of details that don't actually matter to the narrative.
The illustrations were in the post from the beginning as Mermaid diagrams (https://mermaid-js.github.io/mermaid-live-editor/). As we got close to publication we polished them up in Figma.
This is really the first engineering blog post we've put out, there was a fair amount of figuring-out-how-to-do-it going on. Now that we've had the experience, we're starting to write up our playbook internally.
Also, are you adding presentational tables any time soon? :)
Most of our queries are "pointer chasing" - we follow a reference from one record in memory to fetch another record from the data store. To optimize recursive pointer-chasing queries, we cache the set of visited pointers in Memcached.
We use Elasticsearch for search features like QuickFind.
> Also, are you adding presentational tables any time soon? :)
Sorry, can't talk about future plans like that :)
Do you first query the root to see it's content blocks, then make additional queries to load the root's children block, then make additional queries to get those blocks children blocks (ie. recursively) until there are no more children?
Does that result in too many database queries? Or do you have other ways to optimize it?
[1] https://en.wikipedia.org/wiki/Hierarchical_and_recursive_que...
The major issue with the document model is storage. How big can a document grow? How do you persist it, and make small changes inside it? How do you handle "hot" documents that are very popular? Moving data between documents or having part of a document reference part of another document are complex. Complexity comes from building derived data that looks at slices of a document.
With a block-oriented model, your records are much more manageably sized. It's easier to reference or move data between documents, but in turn, you need to do these kinds of recursive shenanigans because your individual records are smaller in scope. Complexity comes from derived data that composes blocks together into a document.
Would be significantly easier for others to work with, could easily query it with SPARQL.
I have a requirement to allow users to create adhoc tables within a web application. I was wohoe beat to represent this on database. Any thoughts?
Here's a rough progression of (not entirely) imaginary user requirements for an imaginary, general purpose table rendering software.
"I just have a bunch of values, can we make this into a table?"
"That looks good. You know what would make it great ? Header rows. Just make them stay there while the table scrolls."
"Hey! can you give me some icons on the header row for each column that I can click and sort the data ?"
"Please give me some pagination!"
"Can you fix the sorting logic? A row with id 10 shows up before the row with 2?"
"Can you support merging cells ? Please !"
"Can you support re-ordering of columns ?"
"I want to adjust the width of my columns but the first column should be fixed width"
... A year later...
"This is great but you know its kinda slow. My users prefer to do this in excel and paste it here. Hey, here's a great idea, can you just parse data pasted from excel and just make a table from it ?"
I thought the reference to field pioneers was a bit much, I wouldn’t have the blocks to do that with a straight face. I wish these app makers (see also Figma, AirTable) would refrain from making their self-aggrandizing “we’re bringing computing to the masses”; what a load of blocks. What you’re doing is creating a product (another silo in waiting) serving a market that presently is interested. This block model is also not unique (see WordPress and Drupal): Notion may be refining some of that in a more seamless experience, but this is a different kind of statement. Perhaps “we added collaboration to blocks” would be a possible claim for uniqueness.
Aside from this, I found the technical decisions interesting and worth a read.
Those apps you mention - Drupal, Wordpress, etc - have write modes that exist in backends with alien experiences to the read modes.
By unifying read/write into a single continuous UI, with built-in relational databases usable by non-technical users, Notion really does let people create damned powerful custom apps.
If you squint, you can see how Notion looks a lot like the next version of the Internet. By hyper-focusing on company intranets they miss out on the power of connected workspaces ("domains"), but the thinking is there.
I'll be seriously surprised if Notion isn't looked back on as a breakthrough paradigm shift.
To that end, I'd be surprised if they're even remembered much at all. I cancelled my subscription after being seriously disappointed in the feature-set.
So it's a bit of a surprise to see them bragging about their technology today.
You can’t have simple tables in Notion (you know, like a comparison table without structured data).
You always have to create what they call a “database”