Ask HN: Why do you keep a personal knowledge base?
Why do you keep a personal knowledge base? What are you trying to accomplish by saving content? What insights do you want to discover? What tasks do you want to achieve?
Although high general intelligence is common among hackers, it is not the sine qua non one might expect. Another trait is probably even more important: the ability to mentally absorb, retain, and reference large amounts of ‘meaningless’ detail, trusting to later experience to give it context and meaning. A person of merely average analytical intelligence who has this trait can become an effective hacker, but a creative genius who lacks it will swiftly find himself outdistanced by people who routinely upload the contents of thick reference manuals into their brains.
I'm a researcher. Absorbing data in a "blue sky" kind of way ("no idea if i'll need this, this doesn't serve any explicit purpose, but hey it seems interesting") is a survival trait.
i cache data in a lab notebook, a mendeley collection of papers, and pinboard.
It really is like bread crumbs when you have deal with things like information overload. There's that whole idea of a 'memory palace' that I like to think about before I go to sleep, because it gives me a structure to meditate on at the end of the day, the thoughts I want to retain and arrange in the ways I'm curious about retaining and arranging them.
A lot of the information we collect can appear ordered, but unless it's a rigorous order proven mathematically, there's no real way to prove anything about the efficiency of our memory. Some thoughts may be useful tomorrow and others may be useful 10 years from now.
So I think it's fun to give it a structure I remember and can travel into, because I chose how to organize it over a very long period of time.
But ultimately I think it depends on ways you are most comfortable thinking about information. My imagination tends to be very visual.
- Recording and crystallizing ideas: most deep thoughts and informed opinions about things often don't materialize on first try. They are the result of many iterations, pivots and revisions. Writing them down helps one regain context quickly after leaving them for a while, which allows one to make incremental progress without being susceptible to the fallibility of memory searches. Writing is also frequently the best way to teach oneself something and to push oneself to identify gaps in knowledge and thinking.
- Sleep aid: committing ideas to (virtual) paper helps unburden them from one's active memory. This has inadvertent therapeutic effects as well. I'm one of those people who has tons of ideas racing in my head all the time, so this helps me sleep at night. On a related note, Cognitive Behavioral Therapy or CBT (for managing emotions) can also be done through written therapy.
- Business ideas/hobby ideas: list of stuff to do when the right combination of opportunities arises. You can have ideas but not all ideas are ripe for execution. Having them in a cache as one is constantly monitoring the environment helps one time their deployment better.
- Latent solutions to problems: Richard Feynman had a trick where he'd be constantly thinking about 6 problems at any given time. When an attack presented itself due to sheer chance, he was able to execute on it quickly, and make himself look like a genius when in fact he had been thinking about about the problem for a long time and was merely lying in wait for the right piece of the puzzle to come along.
- Jokes/good turns of phrases: not for the purpose of plagiarism, but sometimes one needs a bon mot or an apt phrase for a presentation or piece of writing. Having a store of such phrases to synthesize from is much more reliable than drawing from memory alone. Many good writers/presenters aren't Mozart-like geniuses who can produce polished work on-demand: many maintain disciplines like this to aid in producing quality work even when their memory fails them.
- Travel destinations: sometimes we read about an interesting place and tell ourselves "we'll visit some day" but then never do because when the next long weekend comes up, we'd have forgotten all about it. Keeping a list helps one to quickly converge on a destination when a vacation opportunity arises.
My one idea for recovering the lost/wasted time reading and saving this information is curating and publishing it online as blog posts for others to learn from. I’ve spent so much time categorizing an filtering through information out there, only to have it sitting in a private file, might as well make it public for humanity’s benefit (and my own).
So I started doing this:
- Create 1 file per book (I keep it all in Google Drive)
- The problem is my notes are huge, so at the top of the file I include a TL;DR
- Include in the TL;DR a list of actions: things I can put to use straight away. I copy stuff onto a personal Trello too, so I don't forget
- Also when I'm working on something relevant I can go back to the TL;DR (instead of searching through all the notes)
On top of this, I try to read books that are relevant to what I'm currently working on. I have a huge backlog of books I'd like to read on all sorts of things, but I'll hold back if I think it'll make more sense to read them later on.
I feel like I read, remember what I need to google later, and I'm onto more.
Tbh, I pick topics and have goals in mind, even hobbyist goals are more fun than surfing the web.
Don't use the online services to store your personal information!
That is stupidest thing that can be.
First, sooner or later, your information may be stolen and distributed online just like apples: https://raidforums.com/Announcement-Database-Index-CLICK-ME
Second, find yourself a computer, device, where you can store your data offline, and not online.
I recommend reading the online article on SASS or Service As Software Substitute: https://www.gnu.org/philosophy/who-does-that-server-really-s...
and using free software, free as in freedom to do what you want with it to keep your personal knowledge.
Recommended software:
Cherrytree: https://www.giuspen.com/cherrytree/ It works just fine for taking notes.
GNU Emacs: http://www.gnu.org/s/emacs It has notes, reminders, calenders, you name it. it has Org mode and note taking features that blow your mind.
Vym mind mapping tool http://www.insilmaril.de/vym/
and plethora of notes and ming mapping tools available that free software is offering. Search for packages: https://www.hyperbola.info/
I used to use Evernote for this but it wasn't quite working because of constant context switching and lack of spaced repetition.
So I wrote open source program called Dnote [0] and have been use it every day for the last year to store my learning. To minimize distraction, it works as browser extensions, cli, and IDE plugin. And I automated the spaced repetition by writing a cron job to send me digests every Friday.
Lately I've been using screen recording to create video notes. I developed an app called Outclip (https://checkoutclip.com) with my buddy. Although the app is meant for a different purpose (bug reporting) I use it to screen record as I'm doing things (like configuring an AWS service for instance) in case I have to repeat the steps later.
I get my mind back in the groove from sometimes years ago by just opening my orgmode section or file.
The brain finds the connections again and the thought train picks up and off I go.
As a side note, I use orgmode and I wouldn't consider any other program (that I know of) for a specific reason -
What other setup could I be GUARANTEED of being able to pickup in 5, 10 ,30 years and have it work just as it did?
A paper notebook for sure but orgmode gives me much more (and less in some areas).
orgmode and emacs lets me INVEST my time and effort and know I won't lose the data and time because some webservice thing went down/changed business models/lost interest.
It's taken a while but it seems to have built some community and happy users.
But, yeah, I don't think open standards or longevity probably factored too much into the storage implementation or features.
I would think something built on top of annontated pdfs would get some of the features and flavor of onenote but I haven't seen anything attempt it.
By asking yourself this question explicitly and then actually writing down the key points you practice your own judgment on what is worth remembering and what is not.
Now the added benefit of a PKB is that you can actually keep these key points in a central location and reference them in other contexts or share them with others. You are basically doing prework for later (research) questions you might have. However I would say the biggest benefit is the process; explicitly writing down what is important and discarding what is not.
For more on this subject I recommend following @fortelabs on Twitter. He does a lot of interesting writing on the subject.
There are a few basic principles that can help keep the system organized and useful.
Personally I use Google Keep and OneNote: every weekend I funnel the unstructured thoughts, ideas and realizations into a structured format organized in OneNote. It takes a bit of time, but I can look back on any of my notes on programming or real estate or marketing or psychology and instantly find them useful.
Doable, but much more effort than a first attempt on bookmarking.
When trying to decide where to move next, I kept a lot of information on various places that were under consideration. This included not only US states and cities, but at least one other country as well.
I did a lot of therapy in my youth. I find that having stuff I wrote that I can refer back to is enormously helpful for my state of mind, for keeping my goals on track, for tracking progress, etc.
I see the world differently from most people. If I don't keep track of links to articles, comments and research, I get dismissed a lot as crazy, not knowing what the hell I am talking about, etc. and other people typically do not back me up, even if I am saying it in the same forum where I originally read the information.
So I try to keep track of certain kinds of info so that the next time I make a comment on that subject, I can back it up without having to put up with so much disrespectful crap off of people. They don't have to agree with me or see the world the way I do, but I don't care to wallow in their contempt either. Supporting links goes a long way towards preventing the worst of such behavior.
(There are probably other reasons. This answer not guaranteed to be comprehensive.)
It's kinda pricey, and I'm taking the course right now. But I find it's pretty effective overall. It uses evernote as the default platform for your PKM.
It's basically an organization system that serves as an adjunct to "getting things done". GTD's system relies on "filing non-actionable items", but it doesn't really explain how to do that. Traditionally, people would rely on filing cabinets or whatever. BASB relies on evernote to keep basically anything information in there that can quickly be retrieved (becuase you're going to forget years later). And when you later need to the information, you can criss-cross the data for more creative output.
If you are a fan of emacs org-mode (as I am), here's an article that shows how to use it with BASB: https://medium.com/@mwfogleman/implementing-a-second-brain-i...
it goes into the process a little bit, so it can help you decide if the course is worth it for you or not.
Previously I would save notes and never use them. Now my notes are set up in a way that I get more projects done. Having reference material progressively summarized makes collecting ideas and shaping it into a deliverable way easier. Loading context for a project is way easier with a system like this, it makes an interruption much less painful.
The other benefit of taking the course is the forum, plenty of smart people to meet, discuss pkm and other ideas.
I started to use the wiki as a notebook for problems that were too complicated to solve in one day. Plus it was a good way to keep these notes together, searchable, from home and work. I started to use it for installation and configuration manuals for applications that I had to maintain. I did keep work related stuff in it, but only general things like how to setup a webserver.
The task file is the one always opened, there are my pressing issues and there I'll note anything that later I'll pass to the other files. The most important though is the tactics/strategy one, where I write details, thoughts of my life, business strategies and general stuff that I like to read at least every week, the projects I want to do next, hobbies I want to try, advice that I like to read, and even if I have some of this etched on my mind sometimes it's a great north on a confusing day. It's great to read about this idea I had a week ago and now completely forgot about because I was focused at the current issue. Writing is also great to organize the thoughts, I had so many breakthroughs just by reading and writing on these files. It's an extension of my memory and my process.
I don't know how well it works but the zettelkasten method is a way to organise your notes contextually, with only a little manual overhead. You can also add in what they call "structural notes" (i.e. tables of content) to let you quickly find your way. This would aid you in building up a mental map if you will of what notes you have, and to let you quickly find whatever you can't remember.
Then you can do things like SRS flashcards to continually refresh your memory about topics. They're originally meant for just learning things in the first place but they could easily be adapted to keep throwing back your ideas at you at irregular intervals. That keeps the ideas floating around in your mind without you having to manually go and read them.
I read the other 20 or so files monthly or when looking for something specific.
- learning what I need to know about my specific job (institutional knowledge). Even if my title is “Senior Developer”, in reality I usually have a fair amount of architectural level responsibilities and meetings with C level of people working at small companies. For that, I ask a lot of questions and take notes with Evernote or if I take notes on paper, I take a picture. Evernote is surprisingly good translating my writing to text for searching.
I keep a personal knowledge base so I can prepare for a meeting and be ready to answer questions.
- If I am introducing a new to the company technology, process or framework, I have a list of links that fellow developers or my manager (who is technical) to review.
- getting “interview ready”. About three months before I start seriously looking for a job,I freshen up on architectural subjects and make sure I can talk the talk. I keep a list of bookmarks and PDFs. Again, I am at a point in my career where no one asks me to do a whiteboard coding session but they do want to talk architecture. As an in the weeds developer, I know sound architecture but I don’t talk about it every day.
- I have a list of topic areas that I need to study to feel in the gaps to really consider myself a “full stack developer/architect”. But I’m usually only focused on one thing at the time. But if I find an interesting “getting started” walk through about another topic that’s on my radar, I bookmark under a folder “Things to Learn”. Right now, that’s getting deeper into AWS, Docker, NodeJS, and React.
Basically my goal is to decrease the chance of going through the same process of learning something, multiple times.
I also think of it as a cache for Google. I sometimes search my knowledge base before searching Google.
For example: how to delete duplicate lines in emacs? I just found it in my knowledge base in under 3 seconds but would have taken more time googling (also more distracting)
It's also very convenient to have a reference on various devices that you own. Specs, links to manuals, common troubleshooting steps and an inbox of future work and enhancements.
I've found with several groups of material that reconnecting your brain's old steps in logic is way easier. The same material in a different way can feel interestingly very foreign.
Since it's txt file it opens quickly, is easily scannable and searchable. I will sometimes tag entries with phrases that I know I'm more likely to use later when I'm looking for something. The txt file is stored in dropbox and is typically open all day on whatever device I happen to be on.
I've found that tools that organize by things by folder or tag (Notational Velocity, Evernote) don't work for me - I lose track of where something went. Organizing by calendar/time allows me to remember "oh, the thing I'm looking for occurred before this thing".
I have been saving most of mine as browser bookmarks, if I see something interesting or a solution on stack overflow I bookmark it. Then I export those bookmarks occasionally for safe keeping.
I usually just re-search google for a solution or item I'm looking for. But sometimes I search my bookmarks just to find exactly what I used before.
I also have a few snippets text file where I add interesting code related snippets. And some google docs files that are super easy to search too. Gmail is also a great tool, with boomerang so I can have something fly back in my inbox someday or a certain time of year.
All in all I rarely have a hard time finding anything I'm looking for by using google, bookmarks, code repos and snippets.
* Take notes throughout the day. I do this in a notebook.
* At the end of the day, collate those notes into long term storage with a link to the source material.
* When an idea or theme begins to arise, create a document to begin building on that idea, linking to the notes you've taken.
There's a little more regarding organization of notes and linking and so forth. I like the idea and in the short time that I've been practicing this approach I feel like I am getting a much deeper understanding of the texts that I read. Again, time will tell.
The best personal knowledge base is my blog.
The act of writing, of pasting work and thoughts and reflections and associations and tagging and organizing, allow me to really embed it in my own memory.
I really gotta process and write it in order for me to really get the most out of the knowledge base I’m saving.
Simply archiving and tagging is not enough.
That said when I saw ‘personal knowledge base’ my first thought was ‘personal knowledge graph.’ An advantage of knowledge graphs is the ability to start with public data sources and combine in your own information. Like, customizing DBPedia with your own scheme and data.
For organizations, ontology development, defined vocabularies, etc. make sense, but not for most individuals.
Low-friction access is important.
The trick is organisation. This is the hard part, but it's also the hard part if you keep paper notes. I have some catch alls, '_inbox.md', '_work.md', '_til.md' (where I keep track of things I've learned today, I try my best to go back and re-read the previous weeks every once in a while) to keep track of random/non-catagorisable notes; but other things exist in 'administation/' (ie. 'administration/linux', 'administration/windows'), 'intersting-articles/', 'programming/*', 'work/meetings/<project>_<date>_<topic>.md'.
If you use sublimetext and the above plugins, cmd/ctrl-r (navigate open doc by symbols) lets you move/search through headings quite easily. My '_work.md' file has a list of dates at the top for when things are due and uses '- [ ]' as checkboxes (which MarkdownPreview outputs as html checkboxes).
I find that as long as I write it down and watch what I'm writing (don't just copy without following your output) it helps to make it stick—or at least gives me a spatial memory of where it might be.
Flexibility is key. But what works for me, probably doesn't work for you. The important part is just to start doing it, write down all the things. Once you find something you'd like to be better about it, improve it. The nice thing about text files is that they are easy to parse and transform into something else.
[0]: To keep allow you to paste images into your markdown files. I've modified it so that it stores the files in a '.images' directory in the same location as the .md file. ImagePaste also has an image preview mode that lets you expand images directly inline in sublime-text.
For many years, I maintained my technical KB as a local WordPress blog (installed using XAMPP. XAMPP is a portable WordPress + MySQL easy to use installer). The main reason was that WP had plugins for code syntax highlighting, searching and organizing content in a hierarchy. But drafting and formatting articles with code and math was (still is, IMO) kind of a pain in WP's editor, which is why I stopped using WP.
I had also tried out some of the wiki software, including PmWiki and MediaWiki. There's nothing bad I can point to, but somehow I just didn't like using them.
Later I switched to plain old Libreoffice Writer. Drafting, formatting and organizing content are a breeze. It doesn't have code highlighting, but I use an online formatter (hilite.me) and just paste the snippets as HTML. Works great for me, exports to PDFs if I want that content in my tablets. Since searching for documents is inconvenient, I keep all the information cleanly organized in directories.
If you like markdown, I suggest RStudio. It's actually an IDE for the R language, but its usability, markdown and export to HTML support are so good that I use it a bit like a WYSIWYG HTML editor when writing content related to data analysis and visualizations. However, it's a desktop application, and some people don't like those.
If you like markdown and prefer web applications, try out any of the static site generators, like Pelican. These usually require a little knowledge of the programming language they're written in and some command line usage.
That being said, I strongly suggest a personal wiki. I write about Dokuwiki on my blog (https://blog.ivansmirnov.name/how-to-make-a-personal-wiki/).
Key takeaways: - Make sure the underlying storage is as simple as possible. Markdown in text files is as safe as it gets - Friction has to be close to zero, otherwise you won't use it - Put information in a deterministic location. Store any information that you might come back to, so that you don't have to repeat your research.
Inside is mostly documentation, to-dos, recommendations of friends, notes, poems etc...
For technical notes, I use the blog[1].
---
https://github.com/synchrony/smsn
The explenation and thought process behind it seems a bit esoteric though.
TL;DR - I was thrown into a relatively complex environment with little help other than asking the two guys who had done this before me, both of which were rather busy (they had done the administration a side-business of sorts). (I still wonder how they managed to keep the IT infrastructure running, but in my first few months, many of our users told me how happy they were that I responded to their calls for help right away rather than "sometime next week".)
I used emacs' org-mode. I kept a file where I wrote down every little bit of information I could gather. After 12, maybe 15, months, I had memorized everything I needed to know and stopped taking notes. So my example might not fit your question all that well. But for the time I was getting to know this company's infrastructure, its servers, networks, and people, that .org file was a lifesaver.
The blog entries let me save time if I return to a problem and tell me at what point I stopped.
Yes, and the recall effect is much stronger if you use handwriting vs typing[0]
[0] see for example https://redbooth.com/blog/handwriting-and-memory/amp
That's a good thing, because I'd have to rely on my memory rather than trying to read my handwriting.
Hand writing forces you to rephrase in your own words because you can't write fast enough to keep up—but there's no reason you can't apply this same method to typing, AND get all the benefits of legible, searchable, organisable, easily expandable text documents.
Why assume everyone here keeps a knowledge base? I personally don't.