The Fall of Roam
every.to
every.to
At a certain point I realised I was spending more time figuring out how to tag my mp3 colletion than I was listening to music. I've been observant for similar realisations in other fields ever since.
Which is why there’s a massive industry catered towards productivity tools.
I use Zim Desktop for my notes because it's simple and (precisely) because it stays the hell out of my way and lets me focus on my own writing. Over several years of doing this I've tried quite a few other tools, but none manages to hit the sweet-spot of simplicity and affordance I find in Zim. Roam, Notion and other web-based tools are just a fucking annoyance to me, being clunky to use and offering little since I have no need to access my notes across multiple devices, nor any use for publishing/"sharing" my personal notebook or pages with others. If I do want to do that I can (and do) just copy the relevant material over to my (static) blog, `git push` and the job's done.
(Edit to add: the "Backlinks" plugin is vital for using Zim for this purpose.)
Music/video is much more challenging in that regard: our computers' capabilities for searching those is pitiful, but then... keeping bookmarks to such stuff is hardly the use-case for a personal-note tool. It's for keeping your own thoughts and observations, not for bookmarking random blatherings.
I can see how someone who checks out blogs/youtube about Roam-like tools might get the wrong impression but those are usually a business so they have to output content.
It also comes down to personality, I suspect some people have a tendency to optimize tooling as a form of procrastination.
I then came to conclusion MP3 tags are harmful and removed them altogether. Because you can never tell a song genres reliably and many things can be written many different ways.
It's the same with hyper-optimized keybinding editor setups. I'd need to be able to think and type about four times faster before some of the things I've seen would make any meaningful difference.
I've fond memories of hanging out in my friend's room, listening to punk while expanding our extensive Clarion MS DOS database, and then reprinting it to admire it.
During undergrad, I found I rarely reviewed my class notes, and when I did they weren’t very helpful. I simply stopped taking them, was happier, and giving my full focus to lectures improved my comprehension. Notes that don’t get read again are a distraction in my opinion.
And, really, there's no substitute for coming to class prepared (by reading ahead or whatever). It makes the lecture process much more useful. In that sense, a list of notes of things you don't think you understand from preparation is probably helpful.
Review as in look at every single one of them from time to time? No.
But I do review them as in it's my first stop in finding something. It's my personal DDG for repetitive queries (vaguely remembering a quote, needing a reminder of some code syntax, or what happened in a book I've read 3 years ago and kinda liked). All my notes are oriented towards what my future self might actually want to find: most important things at the top, going in more details if I scroll down, sometimes adding unnecessary words just because future self might search for them in hopes of reaching that note, etc.
I'm firmly of an opinion that if the search bar in your notes app isn't the most crucial part of your workflow, you're doing something wrong and wasting your time.
The older way of organization may seem outdated and clunky, but it indeed works.
Yes. I have a bunch of notes which embodies larger subjects or procedures in a more compact and fit-for-purpose format.
These notes are regularly visited, updated and read. Both my personal and professional note troves are actively expanded, reviewed and pruned. However, this is not a deliberate effort, but a part of my normal routine.
Discussing the word "notes" is not easy because it encompasses a bunch of different uses.
- (1) notes as transcription of an external source or memorization aid: e.g. a professor might say something and a student writes that down. Or one comes across a fact and makes a note of it.
- (2) notes to organize internal thought: a canvas for generating/synthesizing/connecting/etc ideas. Things like personal commentary on topics, weighing pros/cons of different future actions, drafts of song lyrics, research ideas, etc.
I think your comment is mostly based on thinking of notes as (1). A lot of people that constantly review notes are talking about category (2).
Sometimes when I spend an hour trying to figure out the right ffmpeg command line options or Linux "find" syntax that works correctly, I'll copy paste those as "notes" in my "tipsandtricks.txt" file. Those feel like a combination of (1) and (2).
Also, if the mental model of "notes" is dominated by (1), it would seem like having a hypothetical universal recording tool combined with a more complete search engine would make note-taking obsolete. I mentioned in a previous comment that such a search engine still wouldn't handle type (2) notes: https://news.ycombinator.com/item?id=30153479
With auto linking, I think the main issue for me is that I want the connection made in my head rather than in a tool. So searching and finding related notes works rather well and the deliberate act helps gradually internalize the links for me.
That's interesting, because I almost wanted to make the exact opposite point. I'm actually right now in the process of revisiting my notes, and I think it helps me a lot. Not because they are super useful in and of itself (After 1-3 months have passed now, I understand the stuff deeper now, obviously) but because they help me piece the stuff together.
Anyways, the reason why they then get useful is because I have an extrinsic motivation to read them! (Exams are coming up)
That does in no way make it easier to read them, but it helps in actually reading them.
Then, reading notes and really understanding something will always be hard. I think there is also quite some research ouf there that finds just this: If you really want to understand stuff, you need to put in the effort. It kinda needs to hurt...
This is why I don't think these personal wikis / note -taking tools help too much, especially if one is, like me (and probably most people), more a person who in their downtime maybe likes to read something new and interesting, but better keep that on the light side - it's supposed to be downtime anyways, no?
That leads me then back to the extrinsic motivation: If for example you're a researcher and use Roam (or whatever) exclusively for that (that means restraining also from putting notes in there that have little to do with your actual research), I think the benefits would be much more clear. To be honest, though, that is just a guess. I'm not a researcher myself and I have n't been using note-taking systems for a while now, because of the points explained above.
The limit of my notetaking is daily journaling because if I link out to the pages I created or updated with some contextual information (e.g. "read chapter 2 of book X, went off and read a blog post at https://X that was mentioned") then I can retrace the backlinks to find out when/how I learned something. I wish I could do that with things that I haven't put into Roam (or Notion, which to be honest I use more often).
I'm now happy with Logseq, which is basically a clone of Roam, with an open-source business model. I really hope the business side works out for them, because for all practical purposes, they've out-engineered Roam to the point of embarrassment. It allows local storage, and will have an iOS app soon (I was able to register for the beta, it's already very solid).
Now if you can name the relations, that allows you to (1) create shorter notes and push more information into the edges and (2) it makes the graph into a true "mind map".
When you can't name relations, you're forced to make "nodes" for Topics or Tags. And this is where the difficulty comes in with "where do I put this?" Not only that but you're forced to write notes for basic concepts like "programming language" and then you find yourself re-making wikipedia altogether. It becomes absurd.
Mek works at the Internet Archive, and it clearly follows the same spirit of "we'll operate the service, feel free to bring your own frontend if you don't like ours". I wasn't happy with the way that one at graph.global tries to subvert/duplicate native browser features, so I put up a minimal "client" for browsing existing nodes that feels similar to the default one, sans annoyances on those specific axes:
<https://graph.5apps.com/LP/streamline>
I never got around to allowing editing, unfortunately. You'll have to use the default frontend for that (annoying, since it's buggy) or write a client of your own.
The key issue I see with the graph.global model is that you have to use triples. I've found that this results in big hurdles for throughput—i.e., the opposite of notational velocity. The ideal thing would probably be to allow a Roam-like system where you can start out by simply linking two related nodes, and then fill the edge details after the fact. You could sort of approximate this with w2g as it stands by just using a generic is-related-to connector and then reify the relation. This does mean you would lose the ability to query by relation unless you add further attributes or went back and edited the original connector to replace it with something more appropriate before reification. Stopping in your tracks to find the appropriate connector is something I found to have lots of overhead.
- Personal therapist helping me find patterns in my thought
- A personal journal helping me determine my life value and goals
- A productivity coach through an OKR system with daily, weekly and monthly checkups
- A system for learning and focusing on what to learn
- A system for planning and what to read (a reading list ++)
- Zettlekasten system for developing ideas in my research and startups
- A place for random notes about everything (a short meeting from last year, a startup I stumbled upon two years ago, a recipe someone told me about)
Roam is not the only system I use. I find inadequate for organized thoughts. If you need an engineering wiki for your project, I would not recommend Roam, notion would be the better option. Similarly if you need project planning, go with Clickup. But, if you need a second brain, somewhere chaotic yet very powerful, Roam is your best option.
But as always, the solution to every problem in automation is some more automation, and it seems like we could do better than forcing users to aversively write titles. A neural net like GPT-J or T5 is good at abstractive summarization, so there's no reason a note-taking program couldn't make titles strictly optional: use the internal ID as the placeholder, and update the title based on the contents until such time as the user wishes to manually edit or write it themself.
This is the sort of gradual automation which works best in practice: I write my note as I please, the NN does the best it can, and when I read it, I realize it missed the important thing (and I realize the important thing in the first place!) and I edit it. Or I don't because it's fine, and I move on to the next thing.
(And making links/backlinks likewise. We don't need to make the user keep the whole note taking graph in their head and curate it themselves: just use more ML. I do something similar with text embeddings on notes on gwern.net: I write a note, and then it gets embedded and a list of suggested links is inserted for me to edit. I find it relieving to write while knowing that it'll suggest links I've forgotten. A poor man's remembrance agent https://www.aaai.org/Papers/Symposia/Spring/1996/SS-96-02/SS... Is it perfect? Far from it. But when it's not, I just delete the suggestions and move on.)
I was not aware that such a system already exists. Can you point me to some references? Is it a private system you developed? Are you willing to share more details? Thanks!
The particularity of Roam, is that it was designed for developers, product managers and execs in the tech industry. So its market is pretty small
Otherwise, the it is is super powerful. In a few cases, too powerful and having cases when one doesn't use all of its powers.
For what it's worth, we've cut p50, p90, p95 search latencies in half compared to December. There are more improvements in reliability and accuracy coming soon too.
- Almost all solutions that attempt to take seriously the concept of knowledge graphs are Software-as-a-Service. Ouch. They say you should “build a second brain”. I certainly don’t fancy the fate of my second brain (whatever that may be) to depend on the commercial success of some company.
- All existing editors, even the web-based ones, have some hangover from file-based content management. Even with ability to cross-link to nodes inside a tree structure within files, the rigidity of file boundaries is still over-constraining, especially when one is trying to grapple with unfamiliar concepts during which the structure and links are constantly shifting.
- The maths-authoring abilities (i.e. facilities to edit LaTeX formulae) at best look like an afterthought. Writing anything more complicated than Maxwell’s equations is painful.
Out of desperation I decided to roll my own. I set out the following desiderata:
- It should take the concept of knowledge graph as far as possible. The foremost thing is to do away with files, and promote vertices (nodes) and links (relationships) to first-class status.
- While upholding the previous principle, it should make editing as natural as possible, preferably confirming to existing text-editing habits. The experience should definitely not be like creating nodes and linking them together inside graph databases: slow and confusing.
- It must work without an Internet connection. Or better: it should not make any attempt to connect to the Internet at all.
- It must not be too heavy-weight, and data should be easy to back up. This precludes any JVM-related stuff, docker images, or spinning up Postgres/Neo4j/Datomic/what-have-you just to write some notes on a local machine.
- It should cater for the needs of heavy maths editing.
Now I have been using my own app, powered by sqlite for storage engine and ProseMirror for frontend, for quite some time, and I am really satisfied (hard not to be satisfied by your own work though). It is becoming increasingly polished (full-text search anywhere, very fancy query, loops in the graph, etc.) but I hesitate to open-source it, because as far as I know maintaining open-source projects can be really time-consuming, especially when people actually start using them.
how do you handle graph queries and looping/traversal?
I am also wondering how i will sync across devices (in my case im thinking of a browser based ui with absurd-sql), probably will just start with the new filesystem api and syncthing or similar alternatives.
For syncing, what I did was to implement a simple automatic backup logic. Then if I want to move to a new device, I just copy the backup I want, which is just a sqlite db file, and restore it. It is not ideal if you are moving from device to device constantly, but it has the benefit that you can compare backups to see what exactly you have changed (I even wrote a tool to build graph visualizations from the output of sqldiff).
Me too. I see no problem in querying it and manipulating data. I don't really care about syncing. What I consider the most interesting and problematic are rendering the actual graph and the data, especially an anesthetic and interactive way. Because what I do NOT want to do is another browser/electron-based tool. I am only interested in desktop-native development but I have never seen components relevant to this subject. Apparently they are yet to be developed.
Open sourcing something and maintaining an open source project are two things that should not be conflated. You can have one without the other.
The idea is to center everything around a data-interchange format and then build a ton of tools. I'm sorry the presentation is still very disorderly and that paper is obsolete and so is my website rishi.is and the original declaration of intent at datalisp.is
However! things are moving rather fast lately and I have more or less completely figured out some of the things that have been blocking me so far and I hope to have working prototypes of the most basic tools by the end of the year.
I don't think Roam invented this. I remember using Notational Velocity on my mac ~10 years ago which is centred around this concept, also using those double brackets in the exact same way.
Edit: Okay, I was curious, so I searched the Google Groups page, and found it mentioned as an existing feature in 2006: https://groups.google.com/g/tiddlywiki/c/KQRS-_sKukQ/m/DXZk9...
It is referenced as a feature on WikiMatrix, so it was clearly an established feature for wikis by that point.
Systems following were cloning Wikipedia I assume.
I am glad I am still causing endless chittering about syntax choices twenty years on. I am sure my gravestone will be “[[RIP]]”.
Single brackets were used for numbered or named “footnotes” by the way.
Here is the design discussion.
All other tools i‘ve seen fall flat on this because they implement backlinks as links, not block transclusion portals.
However, when you want to search or reorganize, most tools only give you minimal support. Searching by keyword ? Only if you tagged it yourself or there's an exact match in the text (forget about synonyms). Notes in domain A but not domain B ? Filter yourself. Split or aggregate notes ? Copy-paste. No housekeeping alerts/tips for reviewing old content. No sharing and user permissions. Taking notes and organizing them on the go is still very second-class citizen.
I'd still use these tools over everything else, the paradigm is too powerful and enabling to leave it aside. I just feel the complete experience is not as smooth as taking a note.
[0] I use Org-roam but the other systems are functionally the same.
I don't think that is true, because solutions today are not much better than 20 years ago. It's just more flavors and whistles now, but the fundamental flaws are still there.
> However, when you want to search or reorganize, most tools only give you minimal support. Searching by keyword ? Only if you tagged it yourself or there's an exact match in the text (forget about synonyms).
There is fulltext-search. But true, synonyms would improve this. It might even be not that hard to implement. All you need is a list of synonyms for your preferred language, and maybe a spelling check for the input. And it might be better to be just an option, people don't really like oversmart systems which do things they didn't ask for. Google is doing this too, and often enough people are annoyed by it as it leads to wrong solutions.
> Notes in domain A but not domain B ?
Each domain it's own folders/vault. But this is something for a mature search engine on googles level. Not the kind of technology we will see on crappy apps yet. Maybe google could build their own note-app, boundled with their web-search. But I fear it would suck on details and closed after 18 months.
> No sharing and user permissions.
Such systems are called wiki or CMS. Note-Taking is not limited to better text-editors. There are mature enterprise-systems in this space. Though, a simple wiki is already enough for most of this.
> Taking notes and organizing them on the go is still very second-class citizen.
Not really. The Web-Solutions are not worse or better on mobile than they are on desktop. The major problem are small smartphones which are generally bad for working well on the go. Though, you could use a tablet, chromebook or notebook computer. The other problem is, web-service generally suck if you have custom demand or just don't wanna pay for them.
But then again, mobile note-handling is one of the most improved areas in note-taking-space in the last decades. But mostly because mobile is now possible for cheap and on relative high level, not because note-taking itself has improved much.
I would agree with the authors assertion that Roam is well suited to write-once/read-maybe. I certainly spend far less time reviewing notes than I ever imagined I would (that might be related to Roam's poor search/explore experience). Where I diverge from the author is that I find the Daily Notes page answers the question of where do I put this?
I have a daily notes template that includes headings:
TODO
Talking With
Found This
Thinking About
Working On
Completed
Every day a new slate, but ordered around the same core principles. At each heading I use tags, for example under [[Talking With]] might come a heading [[Some one]] & [[Someone else]] from [[Foo Corp]] and then proceed to take notes under that. If I can I will [[topic]] things but if the notes are contemporaneous that can be difficult. Under [[Thinking About]] are any of a set of headings I am current musing on, along with a fragmentary thought I just had. If I ever do review I most likely jump off from the [[Thinking About]] tag.
This allows me not to worry about where to put things but, for example, I can usually find notes of sessions with clients quite easily and branch out from there if I want to follow a train of thought.
As time has gone by I've found the product remains useful although my hope for it and the company is not high.
I really enjoy sometime stopping in front of the Chromecast tv and looking at a picture from a family trip. I think a similar experience with notes will make them a lot more useful for the effort you put in to them.
I just clicked through a few until I got to "automate testing". Did a fulltext search for "automate" and insert a new link to "only automate deterministic processes" because it should be related for me. One more (bidirectional) link in my notes.
For one job, where I managed a few deep learning engineers, I had my work life, including evaluations and promotion materials in EMacs and orgmode. When I left, no one wanted those files because, I think, no one wanted to learn orgmode.
If you like Google, then their Keep notes are pretty good. I am mostly on Apple so I bought an app to export Apple Notes to pdf files automatically for backups, and just use notes. On Linux, I keep a tab open to iCloud.com
My worst such shelved toy turned an org-mode file in your project repo into Kanban board, locally editable in the browser using a websocket to running Emacs server. Spent weeks getting it to sort of work. Never used it once
This sounds interesting. Have you posted the source/repository for this somewhere?
Strangely, I've never had a problem with it at work. Right now I use boostnote, but I could use pretty much anything else with a search box, however I think the reason there is that I make notes of solutions. If I ever run into the same problem again, I can search that problem and find a solution.
At home meanwhile, I take notes about ideas. Obviously ideas (at least not in my case) aren't triggered by a problem, so theres never a need to search for them unless I force myself to, and yet theres much more of a desire not to forget them.
Add to that the desire to have a "system" thats simple and reliant on no proprietary software features and low maintenance, ect... as well as all the possibilities that present themselves (each with their own flaws) and I end up in this terrible loop of perfecting a system I don't particularly want to perfect.
I use Roam differently:
- Most of what I write goes into daily notes in a flat structure, using bi-directional links for tagging.
- If a bi-directional link is used often, only then I clean up my linked notes into the page and starring the page if necessary for quick access.
Source: https://github.com/asemic-horizon/sursis
Working demo (used by like two close friends besides me; please don't vandalize it):
http://legibet.casino-rhizome.com:8501/
[There's lots of fun stuff where nodes have "weight" so we color them by the Gaussian gravity field. But the idea of notes without body is already novelty enough for one sitting, I guess]
I can manage tags for particular projects.
Most importantly, as far as I know, our brains not only make connections between concepts, but also grade them and mostly work on the strongest edges. It is not clear if we expect the same from the bidirectional note systems, or do we want to expose all the connections to "fabricate serendipity".
Jots go on paper, and are dealt with quickly.
Dev notes are for my own reference, and I provide them with a certain formality and completeness, as if I were writing documentation.
Years later, when questions inevitably come up about that legacy code, I have a plain text file I can send to whomever, with reference to the Jira ticket. Often I attach these notes to the ticket.
So a whole back-linked network of little informal notes and jots isn't in /my/ flow. It has no useful place. Instead, notes are where I expect: in local files related to the project.
My other major use for notes is the novel I'm working on. There, I can relate to the OP's "where do I put this?" dilemma. I have a scratch section, but it's more like a graveyard of ideas. Otherwise, my notes already have so much structure that I have to make decisions. But, that's the job, isn't it?
This is not a problem specific to bi directional linked note taking systems. Anyone who takes a lot of notes knows that majority of their note database is never touched again. Most notes have a very limited time span after which they are just garbage. Note taking systems are not optimized for this. They try to be a "second brain" excluding the forgetting part. It can't be done no matter how you organize or connect your notes.
After a time you'll realize organization of notes is an unnecessary burden. If you really want to find a note, you'll almost always use search.
That's not really true. Backlinks exist since the first Wiki-Generation and has found their way into other note-apps I've seen over time. The problem is that it's just one feature of many, and rather poorly communicated. Roam had a stronger focus on this, and delivered a modern Wiki-Interface, making it more successful then the old boring wikis which are around for decades and already forgotten by most people.
It seems Roam is now walking the same road? After the initial hype, it must survive the contenst of the daily grind and prove it's worth beyond one dominating feature.
Having things written down is a good idea. The big pile of notes you no longer care about doesn't matter. The value is in the small fraction of things you do end up wanting to come back to. Good luck figuring out which is which ahead-of-time. Writing down things that you think might be useful is so easy, and the cost is negligible.
The notion that writing is mostly useful as a memory aid seems like a totally misguided "both-sides" synthesis. Having things written down is actually just a good idea.
1. It is worth writing them even though you never revisit them. You are gaining new information, you are processing them, and then you are passing along your version of the information to something. This process is very helpful in learning.
2. If you are quite sure that you are not going to revisit your written notes, or it is the likely scenario, then, you should write on paper, with pen. This is much better for the brain.
There is a very short book on writing notes which I am currently reading, called How To Take Smart Notes by Sönke Ahrens. I am liking what it has to say. This book asks you not to overcomplicate your system of maintaining notes. Then, it becomes a chore in itself.
This is also covered in a book I read recently, written by Andy Hunt called Pragmatic Thinking and Learning. I recommend this.
Where it really failed for me was the mobile experience. All I can say is that I kept getting lost in it. This was occasionally a problem on the desktop as well, but on mobile I too often simply could not make my way up to an overview level. This may very well be a problem with me, not the app, but it was enough to kill my interest in it.
I'm back to my ever-growing "brain.md" file at work and (horribly organized) Apple Notes for personal use.
I started off having a single page for all my tasks but it soon got very cluttered so I started making use of the daily notes instead. I found that the fresh page in daily notes each morning is much more inviting and it is helped make this process a part of my daily routine.
If a task comes up I'll write it down in the daily notes and put a `/todo` and `#Project-{some-project-i'm-working-on}`. I'll check off the tasks as I work, and if I feel stuck I will add more bullet points to break the task down into smaller chunks to help me get through it.
For any tasks with a due date or that I think I could use a reminder later on I will add a date using the date picker on that todo.
To see all my todos in one place I can go to the TODO page. This is better than scrolling through my daily notes and I can prune my tasks from the TODO page from time to time.
For anything I see that I won't do I can just delete it or I will put a `#someday-maybe` for anything that I can put off or won't get around to soon. I have a filter enabled ignoring anything with `#someday-maybe` to makes the TODO page more focused.
Sometimes I will go to the `#Project-{some-project-i'm-working-on}` page to review what's been done recently, this is good for quarterly reviews and at appraisal time.
That's about it all there is to it. I'm looking into making use of the query syntax to improve reviews but I found ordering by date is not working.
Then, if a user decides that a particular topic of interest looks promising the person can go to the [[]] page, review linked and unlinked references and expand on that topic. Focusing on bidirectional links makes no sense in the long run. The point here is to keep writing notes and use the Roam provided serendipity
For me the right mix was essentially replacing my physical Bullet Journal with Roam. The big value add for me is that I can see my todos in the context of their creation. I often create a task directly within my notes for a meeting. I can tag it inline to link to a specific day/month it, and then later on find the context of the task (the meeting notes). That’s a big help because I’m not always great at writing down details on the “why” of a task - so seeing the back link to the context that created the task is awesome.
Happy to answer questions.
That has been my journey. Each one improves on the other.
The big downfall of roam, I suspect, will be that the thing they've created is reproducible by others.
It's a great tool though. I just don't think there is mote around it. The next thing will incorporate its advances.
Already has. Logseq [1] is essentially an open-source Roam that stores its data locally, in markdown files. It's not quite as polished yet, because it's still in beta, but I'm quite happy with it so far.
[1]: https://logseq.com/
Thought this was a great write-up of the missing link with roam and tools like it. They make it incredibly easy to add notes but after a while, make it incredibly difficult to find those notes again when needed
Disclaimer: I'm the founder of https://dendron.so/, a YC backed PKM tool focused on hierarchies and structured note-taking
Wrote a tweet thread about my takeaways from the article here: https://twitter.com/kevins8/status/1493020520815546368?s=20&...
There's some sleight of hand here, suggesting that bi-directional links were proposed as one's "primary" entry point to notes. Every bi-directional link aware app I've used has a global search feature, and most have a daily journal. Bi-directional links is a strictly additive concept, not mutually exclusive with simple search, daily journals, or even hierarchies.
This wasn't my experience, though. Maybe because I don't put everything there like braindumps, I can find what I need very quickly, it is enough to remember some tag, or just use Search. (Note: I use Roam free alternative called Logseq, but it works almost identically).
I crossed someone in the street a few months ago who handed me some cute little trinket. If you had the reflex to open your hand in response, you'd be informed you have to pay for it. It's for charity, you see.
This whole category of stringing people along into a bait-and-switch so you can extract something out of them, I wish I had a word to describe the precise kind of way it makes me feel.