The Sad State of Personal Knowledgebases
marcusvorwaller.com
marcusvorwaller.com
I've tried a few and at least twice tried to write my own (and gave up due to scope creep). Right now I just have a bunch of Markdown files and a giant messy DropBox folder and an almost-as-messy set of private GitHub repos. And photos. And videos. And e-mail. And backups. (What does one do with a Zip disk these days?)
I bet a lot of people my age (mid-40's) or older have the same problem: choosing or building a system we think would work long-term, and then gathering and organizing all the "legacy" stuff (which you definitely want!) is already an enormous undertaking, and so instead we live with a bunch of ad-hoc systems that aren't in any way cross-referenced.
Nobody wants to be the sysadmin of their own junk drawer, but I for one don't want years' worth of my stuff locked into somebody's Cloud du Jour.
(Then again, I have this same problem with actual paper documents, so maybe it's just me.)
Apparently started in TheBrain in 1997 and now has more than 250K "thoughts". That's breathtaking for both the longevity and the scale. I'm guessing that most mind mapping programs wouldn't handle that well.
Gathering the legacy stuff is not as important as getting started with something, I think. The legacy will make it into a new system as it's needed.
I've been using Evernote for a number of years. I don't particularly like it (cumbersome UI without great navigation), but the search has worked well enough for me to find things and its ubiquity has been great.
That said, I do use it still, and will continue doing so until there is a solid alternative that I don't think will go poof in 3 years.
There were a few interesting things I've noticed by randomly viewing old bookmarks. The first is that linkrot is a very real thing and the web is decaying at a measurable rate. Second, it's amazing how limited my memory is. Half the time I don't remember something I bookmarked from years or even months ago at all. It's new to me, all over again. I'm going to bookmark this discussion and then completely forget it existed within a few months at most. Regularly reviewing old web pages and notes makes me feel like my mind is a sieve, as well as highlighting how my interest and thought processes have changed over the years.
I'm really interested in this topic, feel free to contact me to discuss.
I did my best to group the text of the urls, using first kmeans in php (which is a very stupid idea) and finally settling on LDA. I'd like to use hierarchical LDA to automatically discover topics within my bookmarks, but that's "Future Work" for now.
I described the system in a previous discussion: https://news.ycombinator.com/item?id=10301205
What you say about review resonates with me. Several years ago I got excited about Spaced repetition software (like Anki, discussed recently on HN) and tried writing an Evernote app that would periodically show me interesting things. Their API didn't work very well for this, so I stuck to doing this manually, and it's some of my most creative and intellectually productive time.
I am contemplating moving to Emacs org-mode for future things, with some sort of conversion process for the accumulated things. That should make a more intelligent system possible.
Standard bookmark with optional quote via highlighting:
%T %N %B %U %N %L %N %?[S{"%S"}]
via %U%N%L%N%N%?[S{"%S"}]
#note via %U%N%L%N%N%?[S{"%S"}]%N----%N
The Sad State of Personal Knowledgebases | Hacker News
https://news.ycombinator.com/item?id=10739227
12/18/2015, 10:04:11 AM
"A friend showed me http://gethibou.com/ yesterday. Basically the idea seems to be that you bookmark a page, highlight what you want to remember and they use spaced-repetition to help you remember it. It might go along well with the PK that you created yourself."
That gives me backups, and access from other computers / laptops.
Search is via ack or grep.
It theoretically also gives me access from my phone using MobileOrg[1], but I have never actually set that up.
C-c C-j. You're missing out :)
If you set
#+OPTIONS: inlineimages
I'm a Vim user, but thanks to SpaceMacs/evil-mode I feel right at home. Actually, it works so well that I've come to prefer evil-mode over the real Vim!
Thanks for the reminder!
org-mode and evil-mode alone, without Spacemacs, work pretty well already. It's just additional glue to make it more seamless.
So, how do you suggest I acquaint myself with Org-mode? Vanilla Emacs + evil-mode?
Take Emacs as a lisp machine, a working environment, a framework. Then start building it up to your liking. A bit of Emacs lisp knowledge is essential, and it is an easy language, though with rough edges. With org, be experimental and incremental, don't try to create a perfect system from day one, start small and tame it to your habits and tendencies. But get used to the capture+refile flow (see info).
I can't comment about evil, but I do not really feed the need for vi-style editing. Typing speed is not the bottleneck for me. When I type prose, I use Emacs standard bindings + visual line mode. When I program, and nowadays I program exclusively Emacs lisp, and exclusively for my needs, I use paredit. I feed good with these, and I do not feel a need for more speed. But everybody is a bit unique in their needs, so if you'll feel better with evil, go for it.
My last suggestion will be that of avoiding things like Prelude and SpaceMacs. Steal from them, but build your own config yourself. Apart from this, I always use a stable release of Emacs and always check in to git the packages I use, so that I will not have to depend on 3rd parties for my setup. My .emacs.d is simply vital to me.
Edit: about controls, mapping caps lock to ctrl is a life saver, though I do not know well the mac keyboard.
What kind of scripts would you add to your personal knowledgebase?
I have written a script in D (my programming language of choice). I have a web interface to all of the notes I've jotted down in the past. I can click a topic and see all notes related to that topic. I click the new note button, and it opens Emacs with the file that will hold the note. I type in a note in markdown, and as soon as I close Emacs, the note is converted to html. If I want to change it, I hit the edit button, Emacs opens up, and I make my edits. Upon closing Emacs, the html is rebuilt. Anytime I create a new note or edit an existing note, there is a commit to the Git repo.
That's what I mean by automated. I write scripts that handle everything except the content of the notes. I have written or am writing scripts for just about everything I do.
"Camlistore is a set of open source formats, protocols, and software for modeling, storing, searching, sharing and synchronizing data in the post-PC era. Data may be files or objects, tweets or 5TB videos, and you can access it via a phone, browser or FUSE filesystem."
My plan is to land PDFs of research papers, web bookmarks (and archives of those pages), ebooks, book scans, textfiles/markdown documents, imports of social networks (favorites from Twitter, mostly) and scans of written notes I make to myself in Camlistore. From there, sync to devices and publish to the web as appropriate.
I don't expect that I'll ever find the time to build this all myself, so I'm hopeful that either my data will be in a format that other Camlistore apps can read, or if I get the format wrong I'll be able to write a translator to whatever the Camlistore community decides to model for that specific data type.
This was five years ago at least, mind - I've no idea what the problem might've been, and even if it wasn't user error, I suppose it might've evaporated by now.
(note to self: try tiddlywiki again soon)
Over time I've built up some scripts to help with general tasks, eg list TODOs etc.. It's perfectly moulded to my needs. For example, when I find a new utility I run mk_notes.sh [thing], which sets up three files:
- links (for relevant web links)
- thing.txt (which has basic info about it)
- cheat_sheet.txt (which has quick useful things I want to keep)
I'm not sure I could buy such a thing, and if I did, I'd get irritated with some aspect of it sooner or later.
Search is easy (grep et al), git means it's available everywhere (for me).
It's literally changed my life, since I now don't 'lose' any research, and can pick up threads later and build.
Also, I'm a vi user :)
Most people use it for planning, but it's also a very nice personal wiki. The great thing is that you can define your tasks inline, if you want.
That's a computer, full stop. Or, in software terms, an operating system. In storage terms, a filesystem. These high level tools are designed precisely for the tasks he outlined: Handle data of ever increasing size reliably; make it easy to navigate and search; be simple to use, convenient and fast, and structured.
Where operating systems and filesystems fall down is in searching heterogenous data in a powerful way, and that's where people turn to PKs. Evernote is/was good at searching text and even pictures of text and for tagging; "the Brain" is apparently good at cataloging structured to-do and project type data; Kumu seems to do social graphs; inforapid seems to specialize in process descriptions.
There is no reason operating systems can't get better at processing this data natively. Apple has slouched in that direction in fits and starts with projects like Spotlight and Quick Look. Microsoft once fantasized about building a filesystem from a relational database and has developed an interesting system of "search contracts" for Windows.
It's too slow or too inconvenient for most things you'd want in a PK. Creating files isn't fast or easy. Usually you need to open some other app to do it, even if it's just a terminal. Linking one file to another and showing the connection with metadata like the direction of the link is possible, but not easy, especially if you want multiple links to and from a file. You can sync with things like Dropbox, but search is painfully slow and viewing/editing files means opening a different app. It's just cumbersome.
The reason I like TheBrain is actually because it's good for much more than todos and project data. I have one "file" that has over 10 thousand notes in it. It's organized arbitrarily by my preference, but definitely structured, searchable and "infinitely" expandable. The best part though is that it's really easy to visualize related data in TheBrain. In fact, as I've read comments today, I've come to realize that that is the one crucial thing most PK's are missing. There aren't good visualizations for Evernote, OneNote, most filesystems, org mode, etc.
I"m with you, there's no reason the filesystem can't become the best PK, but it's far from that now.
If I were to evolve the product category I'd use Confluence as a basis.
In fact I'd want to toy with using Confluence (+ Postgres backend) as the basis for any I.T.-related product -- write plugins for user input, generate all reports to Confluence. I'm not sure what licensing restrictions might be but Confluence provides a lot of useful UI for $10/year (structured + unstructured) for any customer, and building on top of that seems like a no-brainer. Perhaps this is what others do with Drupal and other CMS products, though ... I'm not familiar with that model.
Like most Atlassian products, its only apparent notable quality to me is that it integrates with other Atlassian products, as well as being a semi-functional platform on which to pile plugins and customization. It's possible that there's a good Confluence install out there, probably administered by the same people who run that really nice JIRA instance that people claim to have used once.
So, while Confluence itself as a piece of software makes me grumble more often than not, I'm more than happy to trade that off against the easier sell to others.
Some of the "answers" are just a single line. Then all I have to remember is either something in the title or the gist of the thing I need to know and google docs search provides me with a list of related stuff.
http://freemind.sourceforge.net/wiki/index.php/Main_Page
(The one thing of note is that there is a bug in the default version of OS X Java, so it frequently crashes unless you update to Java 8 and edit the config to use that version of Java.)
A mindmap starts at a central node and moves outward. This makes it limited in the practical number of nodes you can have--anything over a couple hundred is hard to manage. It also makes it difficult for one node to appear in multiple places within the mindmap.
I've never understood this preference; would you be willing to explain it? It seems almost exactly backward to me: I model tasks as trees of sub-tasks, and knowledge as a general graph of connected concepts.
The idea of forcing yourself to put knowledge into hierarchies is that it makes your thinking more clear, makes it easier to remember where things are, and perhaps paradoxically makes it easier to see how things are related. (Or at least highlights certain types of relationships that are only visible in the context of hierarchies.)
I'll admit it's pretty weird at first to only get one root node, I think it took me a couple years to get used to that. But once I embraced it the benefits quickly became clear.
Gmail nails most of the features on his list.
Only thing it doesn't really cover is linking between notes. Though I wonder exactly how valuable that is anyway, when you have a powerful search available.
I can't figure out how to do that in gmail.
That is a pretty show-stopping limitation for a "knowledge base"... have they fixed that?
I stopped developing it once I realized it was a security nightmare. I built a version that encrypted the data before storing it but that ballooned the DB size and I also didn't have a user-friendly way for users to obtain a key set.
Early on they hadn't quite nailed "findability", but with the new (more generic) folders ported from the desktop metaphor and "starred" items this is largely solved. And the search is very fast.
Not only that, but it makes it incredibly easy to share with friends and even have them comment or contribute to individual files, without having to expose everything you write. Learning curve is near-zero.
All in all, super impressed.
Let me know if anyone has any question: hello@faqt.co
Edit to add: Some other interesting features. You can sync to Simplenote automatically, and notes can be saved as text files for easy import/export. There is some markdown support (though I see a number of bugs in the preview). The main downsides appear to be: limited color schemes, no syntax highlighting for markdown, development appears to have stalled.
This lets my data remain portable and still gives me (almost) all the niceties of proprietary formats/editors.
https://youtu.be/VZQoAKJPbh8?t=2818
https://news.ycombinator.com/item?id=10726536
Eve, the new "programming" tool developed by Chris Granger of Light Table fame, looks absolutely amazing for this use case. I personally can't wait to try it out.
Org-mode is probably the closest contender, if only for its sheer flexibility - but the fact that it's wed to something (i.e. emacs) that the gen-pop would immediately recoil from is only ever going to limit its adoption to a tiny subset.
- markdown
- + folders (synced to my mobile)
- + a flat-file cms to wrap it up in the browser (in my case: yellow cms, but there are a few based on markdown and I could switch any time)
Still problematic:
- fast and easy linking
- suggested tags (no body likes tagging manually)
- rich media (especially as you'd ideally want a local mirror, like some research tools do it)
and the general problem:
- text is linear
- folders mostly hierarchic
- information is ribosomal
- ribosomal connections are rather associative and therefore manifold
- manifold structures are a pain in the ass to do manually by linking and tagging
An even better google would be the solution:
- you just dump information (+ meta data to give some more context and spare you explaining the current event surrounding the note)
- and it will be found on demand by AI
maybe in a few years I can feed something my wiki and it will make sense of it
I don't use it nearly as much as I should... it's one of those things that's 90% what I want, and just missing the last 10%.
Any "advanced" text editor that can search directories should work: Notepad++,Vim,grep,... anything really.
I've been using a bunch of synced markdown files. Works for me, I just have a Markdown editor and a "grep-like" app for android. I'm not sure what you can do for iPhone though. When I've talked to people they couldn't seem to "browse for files/folders" which I thought was strange.
The one thing I'm missing is I'd like to "tag" my notes. I haven't decided how/if I'm going to do that yet.
Basically the core problem isn't saving and searching for data, it's having great ways to structure and visualize it.
I've been thinking of switching to some kind of wiki format, like maybe creole. Then finding an app that can understand links between files or something.
I’ve tried Quiver, Quip, & Workflowy and cannot find the perfect app to suit this space for my usage patterns. I’m almost 100% back to text files & sublime text.
Now that it's been revived with Tiddlywiki5 and development's picked back up, I'm somewhat tempted to get back into it. Main advantage: Better crosslinking, rich content, etc. Main disadvantage: Less simple to edit, some tedium to export.
I'd love to have a Tiddlywiki-like frontend to a set of Markdown text documents backing it. (This is something that's been gnawing at the back of my mind long enough that I might have to buckle down and make it myself) Maybe Markdown + JSON for rich content data...
Partly because I hate Markdown due to its inconsistent syntax (that's my personal opinion), but also because it feels like a fad of recent years, much like a successor of the MediaWiki fad and the BBcode fad. I feel like in another 2 or 3 years someone will invent Markleft or Markright or Visual Markdown#.NET 2.0 or Dustindown that's 10 times better than Markdown, and then I'll have to switch everything over yet again.
HTML and LaTeX on the other hand, aren't dying any time soon due to their extensive deployment, and I'm confident that my syntax will be renderable for at least another 10 years or more.
orgmode is really a superset of markdown with PIM markup added.
It exports to markdown and a ton of other formats.
Cheers.
- Zim (http://zim-wiki.org/, Open Source)
- WikidPad (http://wikidpad.sourceforge.net/, Open Source)
- ConnectedText (http://www.connectedtext.com/, Windows-only, $$$)
Take a look at the last 15 minutes or so of this video for a demo:
https://www.youtube.com/watch?v=VZQoAKJPbh8
(the whole video is interesting for the evolution of it)
A personal knowledge base is far more valuable than what it might seem at first glance. Very happy to see you have articulated clearly and made people realize it. A PKB has great potential for knowledge transfer within organization or even at research labs for new people to familiarize with literature. Many people want to capture their thoughts for writing a book someday, but current tools aren't good enough for this. Sadly, a person's social graph is well captured today rather than their knowledge graph.
With these thoughts, I started implementing Hyperbook - www.getbook.co few months back. Currently it is in beta and really working hard to improve it every day. We got good feedback and reviews from thought leaders in content curation and knowledge management industry. Thanks for bringing the problem for discussion. It gives me motivation that is very much a worthy problem to solve.
1. Does it work under less-used OSs? Linux/BSD..?
2. Is it a proprietary format? Will you be able to open the file in a decade? My text files will be able to be opened, I'm not sure anyone can still open their MSWorks files.
Maybe I'm crazy to worry about opening these files after a decade since using them probably isn't worth while unless you keep up with it. However, I've got files from the 90's that I've been transitioning from machine-to-machine since storage costs keeps getting cheaper.
https://itunes.apple.com/us/app/microsoft-onenote/id78480155...
https://msdn.microsoft.com/en-us/library/dd924743(v=office.1...
it is only about 5% of the size of the Adobe PDF specification and I am sure there are corner cases, but it is not too bad.
I've been using Pocket for now and it seems to work fine.
sigh