Ask HN: How did you build up your personal knowledge base?
I'm looking for a good solution to build a personal knowledge Base. What are your experiences? Which Tool you are using? How long is it in use?
All files: https://github.com/nikitavoloboev/knowledge/blob/master/SUMM...
The way I did it was by treating the wiki as extension of my brain and having a perfect workflow for editing it at a speed of thought. In my case it's Sublime Text + Alfred + some macros.
https://wiki.nikitavoloboev.xyz/other/wiki-workflow
In fact I recently started to use my wiki to host article drafts I am writing.
https://wiki.nikitavoloboev.xyz/fragments
The best thing about having a wiki is the 'in progress' nature of it. Soon I plan to extend my Alfred workflow to access any link inside any of the files in seconds too.
I like the way you've got yours set up. I think we have similar mindsets to reasoning about personal wikis. In case you are interested in hunting around, I've run across some personal wikis larger than yours, but not many. Even my own wiki-epeen is about three times the size of yours in ~11k files (wrapped into one) and ~470k lines.
Drafting what will eventually be static content from within the 'in progress' nature of the wiki is hard to beat. Including, this post, I draft many of my communications openly in the wiki, and that has been extremely useful. Lots of odd things come out of the practice.
Or is there another kind of application you'd like to be able to use (that may or may not exist yet) in place of Alfred?
Would any search system work for that matter? Could a command line system be used (ie - some form of grep perhaps)? What about something like a "google appliance" style search system (a couple of these exist - both open source and otherwise)?
On Windows Keypirinha[4] is pretty good. There's also Wox[5].
I don't think any of the alternatives are as well fleshed out as Alfred is but there are a lot of 'good enough' if you're using a different OS. Any launcher is better than the default search on windows 10, for example.
[1] https://github.com/albertlauncher/albert [2] https://github.com/DaveDavenport/rofi [3] https://ulauncher.io
currently building a service that can index and query across text based knowledge bases. you can find demo here: http://demo.alphacortex.io
would love to hear your thoughts and talk further about organizing knowledge :)
Human memory is squishy, and that's great. You retain facts + feelings that help you make future decisions and everything else is integrated into smaller and smaller summaries until it fades away entirely. I spent years clinging to every experience wanting not to forget anything, and being horrified when I couldn't remember. I documented everything. I wanted my memory to work more like infinite dropbox storage and less like a tool, evolved over millions of years, to keep me safe and making good choices.
These days, my personal knowledge base is whatever I've bothered to remember. Usually I don't know I've kept something tucked away until I'm in the middle of a conversation and realize it's still there. My memory of events shifts and degrades over time, and I'm fine with that.
If you don't use a tool to document every page you visit and every thing you read or learn, that's completely fine too. It all fades away eventually, and if it remains relevant enough you'll hold on to the important bits.
The real power-up isn't keeping it all, it's being able to change and grow, with just enough focus on the past to make good choices.
"Mind palace", forgetting, "relax!": https://news.ycombinator.com/item?id=11809676
"Oxbridge method": https://news.ycombinator.com/item?id=11805264
Not taking notes: https://news.ycombinator.com/item?id=11853258
I love learning and will regularly do courses to pick up something completely different, or something applicable to my career.
However, I do think it's okay to allow some of the things I've learnt to fade away. Technology or previous interests become less relevant, and trying to retain everything keeps me too focused on the past, making me feel like I get worse over time, rather than keep a positive learning attitude towards the future.
Also, relearning things I've used a long time ago, because it has now become relevant again, is a lot of fun. The second time I've looked at trigonometry, I actually really enjoyed it.
But knowledge value is only positive if it helps you achieve your goal, when recalled. Otherwise the time taken to recall and discard knowledge is a negative. So really optimal knowledge is where you only know what you need to achieve your goals.
Love how you framed it!
https://news.ycombinator.com/item?id=8270759
https://news.ycombinator.com/item?id=17892731
https://news.ycombinator.com/item?id=19095849
https://news.ycombinator.com/item?id=19294799
https://news.ycombinator.com/item?id=19847258
I used to write down information in note-taking apps but doing so didn't help me retain more knowledge. The reasons were: (a) no spaced repetition, because I never went back to my old notes once I wrote them. (b) environment switch due to having to launch external apps every time to write a note. Environment switch was especially painful when I was immersed in a complex coding problem.
I solved those two pain points by sending myself an automated digest of my notes every week (spaced repetition on autopilot) and building CLI, browser extensions, and IDE plug-ins to talk to my personal knowledge base so that I don't have to switch environment to put stuff in the knowledge base.
My experience is positive. It has helped me advance my foreign language skills, learn new vocabulary, and retain various technical micro-lessons that I come across during programming.
Oh, this is fantastic, thanks. I'm going to have to look into how I can do this for my own knowledge base. Especially the spaced repetition part.
> https://www.getdnote.com/blog/how-i-built-personal-knowledge...
Btw, this is a great article.
1. Stopped trying to categorize cards and just create them all in a single deck (with tags in lieu of separate decks).
2. Whenever I find myself googling something or looking it up on StackOverflow (that's not wildly specific or related to domains I don't really care about), I add a new card for it.
3. Review cards for a few minutes in the morning.
The above has worked rather well. I mostly focus on topics I'm interested in diving deeper into (python, for example) and has helped fill various lacunae in my knowledge.
For broader sorts of knowledge acquisition, I use Boostnote to keep track of interesting notes/articles, but I have to confess I'm not that great at keeping it updated/relevant.
I shared the Java portion of my deck at https://ankiweb.net/shared/info/647806244
I have noticed that many of the solutions here do not really translate well into cell phone use. Most of my life is not spent in front of my own computer. Even though I spend a lot of time at a work computer, it is heavily restricted (defense industry) and thus can't be used for personal projects. Anything I use has to work with my cellphone as a 1st class capture interface.
In my case, I recently decided that I since I am a heavy user of wikipedia, I should just use mediawiki. I'm used to it and it is well supported. The syntax isn't great but it has great API support. They have a decent cell phone interface, though I find that it isn't fast enough for taking notes. So I take quick notes in Evernote and move them into the wiki when I'm at my home PC.
It's working okay so far. I need to put more effort into my workflow. I want to have an automated infrastructure for capturing my various data from social networks, email, calendars, things like that.
Side note: I can't believe we are coasting into the year 2020 and copy/paste is still so friggin bad on my cell phone. Android has had copy/paste from text entry fields for a long time, but I still can't copy text out of most apps. WHY??? Case in point: YouTube descriptions and comments.
the other benefit of an outliner is that you don't have to worry about things getting unorganised. everything is a bullet point but you can also nest points under each other and then collapse the list or zoom in to focus on a certain point.
with a single page like in Evernote there is only so much you can write before things gets messy. if you want to reorder something it means you can to cut and paste whereas with dynalist you just drag and drop the bulletpoint
heres the full feature list https://dynalist.io/features/full or check out the showcase section in the forum to see what other people are using it for
https://digraph.app/wiki/topics/39b38b69-1c22-4e98-bb79-661b...
https://blog.digraph.app/2019-07-06-ten-thousands-links.html
This is a new app, although it's an iteration on one that I made more than ten years ago.
Backed up as part of my usual backup procedures, but only (easily) accessible with the GUI on my laptop. It would be trivial to sync it between devices and use it from multiple locations, but I don't need to.
Reading books and taking notes.
Watching videos and taking notes.
Attending meetups and taking notes.
Doing. Specifically, by applying what I learned to small projects.
I collect all the notes and projects in GitHub. It's simple (markdown), searchable, versioned and all in one place.
Oh, and free, portable and highly available while hosted by someone else.
I benefited immensely from taking the course. I have no personal benefit or stake in the program.
The author/program does use evernote primarily as the tool but the technique/process/workflow is independent of tool ( I think they do address notion and onenote as well if that's what works for you ).
A brief high-level summary: https://www.youtube.com/watch?v=SjZSy8s2VEE
You can see it at https://maxmasnick.com/kb
Essentially it is a folder of Markdown files with some YAML metadata at the top to define categories. Hugo takes care of rendering the HTML, which is automatically deployed every time I push updates to the ‘master’ branch on GitHub via the GitHub Actions beta. It works pretty well, and allows me to edit on iOS via Working Copy.
About 3 years, and I use it every day. The experiences with the tool are documented in the tool.
It's a little different than a traditional wiki in that each "page" (called a "tiddler") is not really designed as a "page" that you view, but rather a snippet, which can be of any length. You can link between them and whatnot as in any wiki.
The killer features for me are support for
* code syntax highlighting
* LaTeX math with KaTeX
* adding in media (pictures, screenshots, etc. beyond just plain text)
* creating dynamic/aggregate tiddlers based on tags. For example, the code to create a list of links to all my tiddlers tagged with both `blog` and `ideas` is:
`<<list-links "[tag[blog]tag[ideas]]">>`
"Banks have three alternatives for determining operational risk regulatory capital under Basel II
1 - Basic Indicator Approach 2 - Standardised Approach 3 - Advanced Measurement Approach (AMA)"
I make sure that each Tiddler has a link off to at least one other Tiddler. This one in particular has links off to my entries on the three particular approaches, which in turn link off to mathematical formulas and other related thoughts. I find by doing it this way I naturally tend to dive more and more in-depth to a particular topic until my Tiddlers are getting extremely specific but then one may link back to a very broad topic and the cycle repeats itself.
By doing this, when I read over my notes, I find it almost feels like I'm having a conversation with them. Just like a normal conversation that you and your partner are deeply interested in, you'll dive deeper and deeper into specifics until another random, broader thought pops up and you repeat the process. I'm continuously surprised about the thoughts I run into as I randomly walk through my notes.
But by far the most important thing for me is that this process has actually made note taking fun! I love it now. Every other system I've ever tried has felt like a chore. I enjoy it so much I'm usually adding several hundred new Tiddlers a day.
* It works in any standard browser.
* You own your data, and it's very hackable.
* Its learning curve is better than comparable tools (e.g. emacs).
* It lowers the friction between you and your machine, enabling you to upload a semblance of your mind into the machine with minimal effort.
* Search and organization are mind-bendingly powerful.
* It is amazing at allowing you to reuse and reconstruct your work. It's designed to evolve.
To me the main choice you make for a knowledge base is the linking structure: do you want just an unorganized pile, a web, something hierarchical, or something linear? In general CS/programming people prefer less structured forms like webs (as can be seen in these comments), while physics/math people prefer the opposite. For example, the Stacks and Kerodon projects are huge collaborative knowledge bases for certain subfields, but they're organized like a standard textbook.
This is partly because physics/math people prefer more stable software, since they don't have time to keep up with or build the latest thing. LaTeX has been stable for decades. Furthermore, the material in physics and math really is more ordered and hierarchical, while CS seems to have a broader, more organic structure from its youth.
I love the linear format, and I'd encourage you to try it too. It requires no special software. It forces you to think about what is most foundational, and how certain insights lead directionally to others -- issues which you can ignore if you give yourself the freedom of a web. And as you move forward to more advanced things, spaced repetition of the more foundational things will happen automatically.
For my needs text files were that solution. Indexed entries, with or without HTML, that I can custom-search and display in a web browser. Adding links to great online resources takes little time. Adding 'features' is the other dangerous fun.
I just make a new file with a YYYY-MM format and dump everything in there, and on the next month I start a new file. I keep them all in 1 folder.
It's really simple to use and very searchable with grep. I like it because it's all offline and doesn't get in the way.
It becomes a brain dump of interesting links, notes about certain things I'm doing or whatever else I feel is worth jotting down at the time.
1. I use Omni Notes (simple note-taking app) and Bear [1] to write down thoughts and longer texts. And a simple voice recording app [2] for bigger ideas that need more words.
2. After this, I evaluate them and add them to my personal wiki. I’m using BookStack [3] which is the best open-source wiki I’ve found. Every aspect of my life gets its own book with chapters and pages. It’s a pretty straight-forward structure.
I'm using this process for a year and it served me well (understanding life goals, writing down business ideas, structuring knowledge). Next step is bookmark management, I'll probably try out Shiori [4] (was once featured in a Show HN).
[1]: https://bear.app
[2]: https://github.com/dkim0419/SoundRecorder
It is by far the most straightforward wiki I have used. The two features I rely on most are (i) creating new pages that link back to a piece of text simply by highlighting the text and entering a shortcut and (ii) adding aliases to pages, so that each time either the page name or its aliases are mentioned anywhere in the wiki, a link is automatically created back to the page.
Using these two workflows, pages can fulfil a similar function tags do: If a concept appears again and again across many different pages, I simply create a page which is named after the concept or has the concept as one of its aliases. When I open this new page, I can see all pages that link to it in a panel that lists all back-links. I then go on to edit the page, going into more detail.
This has turned out to be the most organic way for me to write a wiki where editing, linking, and exploring works almost frictionlessly.
It got picked back up a couple of years ago, but when I checked it out recently it didn't even have dark mode implemented. I'm sure I could do it in a style configuration or something, but the bigger point is that Mojave has been around for quite awhile now, and lack of that update points to it being abandoned again.
So for me, non-starter. On Mac, being abandoned means you only last to the next major "all the old software is screwed" change they seem to do every couple/few years now, so it's kiss of death.
Ok - well, other than that it's a hodgepodge of text files on my phone and main workstation, some of which get moved to my small home NAS.
And a few notebooks. And some stickies. And scraps of paper.
Not a great system - well, let's face it, it stinks.
At one time I had a website/blog I had built, but my vhosting provider updated PHP and it broke my stuff; not so much I couldn't fix it, but I got lazy and never fixed it, and now I've been mulling over for a while how to do everything as a github supported static page website instead, since I am pretty tired of my crufty PHP junk.
I'm going to eagerly read everyone's thoughts here, though, as I think maybe they could help me become more organized in what I am doing and such.
I've got 16,399 thoughts in mine and add more just about every day. Anything I think I may want to know later goes there. More ephemeral notes go in Apple Notes (I used Evernote for a long time but became annoyed with their business model and haven't missed it). Everything else goes in The Brain.
Search great, the ability to link notes as parent/children/siblings is extremely powerful once you get the hang of it.
It's not free, but it is, in my opinion, easily worth the cost.
I've been through
* evernote - slow and buggy UX, properietary format
* emacs org mode - good but you need buy in to org format, user experience across devices e.g. mobile not so good)
* quiver - pretty good, markdown format, but lacking android support
* git + markdown files - ok, but a bit finicky, again annoying across devices
* joplin - pretty good, has mobile support, has markdown etc
* nVault - lightning fast, no mobile support
I saw a thread on HN recently talking about a similar topic and someone mentioned TiddlyWiki which seems promising, I'll investigate it soon, although I suspect I'll probably jump when the next thing comes along.
Luckily most of my 'wiki' is in markdown format so it's fairly portable, I just want something that
* is easily accessible across devices
* supports markdown (+ rendering)
* supports inline LaTeX (e.g. $$x = 5 + y$$)
* renderer should support inline images
* good search capability
you can add images but at the moment you need to hover over the link to view them unfortunately. (although I think somebody made a browser extension that makes the images load in view, but then that won't work on the mobile app version. their web app is fairly useable in a mobile browser though)
I just wrote a spiel about Dynalist in another comment if you want to word search for that
I use it myself, and file is "overwritten" when saved. (as expected)
I have local file that I open in firefox, and edit. Every so often I commit it to git.
If you open from non-local URL - it can not be "saved" - which is what you may have encountered ?
In short:
- Emacs OrgMode with a specific setup as the core of everything.
- I store interesting web-pages as PDFs on my local drive, currently synced using Dropbox.
- Academic articles go in Zotero, with PDFs on local drive, synced using Dropbox.
- On mobile, I use the Dropbox app to create and edit markdown files (I wish they would just treat .org files as normal text files!), and to save any web page to PDF.
- I sometimes draw flow charts, architecture diagrams and UI using an Apple Pencil and the Notability app on a 2018 entry-level iPad, which syncs these sketches as searchable PDFs to ... Dropbox.
[0] - https://www.notion.so/
It's future proof and require no special tool. Another philosophy that drive this way of managing my data : The Lo-Fi Manifesto http://kairos.technorhetoric.net/20.2/inventio/stolley/
I use almost exclusively plain text files. If I need formatting to produce things like HTML, PDF, ... I use Markdown or LaTeX.
For images/audio, I use lossless file formats (ex: TIFF, FLAC, WAV, SVG). If I need smaller formats for a web site, for sharing or to bring it with me on my phone, I convert it to smaller formats like MP3, JPG, ... Those files are disposable when not required anymore.
During my research if I found a web page I want to refer to later on, I save it in PDF format. Why? Because PDF is fairly future-proof (ISO standard) and it's so widely used that it will not disappear anytime soon. I don't save it in HTML format because I will probably not be able to reopen it in the future because HTML spec changed too much. I don't want to have a folder full of images/css/js for each page. Also, there is no garantee I will be able to read it in the future to way I read it initially. PDF have this garantee.
For anything else (like speadsheets), I use LibreOffice the most I can. LibreOffice is based on OpenDocument open format. Open format fit in my definition of future-proof formats. And no, Microsoft OOXML format doesn't fit in my definition of future-proof format because it's driven by Microsoft (same for any big business like this). They can decide tomorrow to stop keeping the standard updated and start adding proprietary "features" in the "OOXML-NG" format (which will make Microsoft Office files unreadable in other Office suites).
For future-proof information/knowledge management, stay away from closed/proprietary softwares as much you can do.
Not exclusive to personal knowledge base, but I keep a lot of notes in there because it's my default catch-all, including on the phone since I have Termux installed. It's easily greppable.
Tried and failed :
* emacs(org-mode)
* plain text files
* markdown files
* tiddlywiki/ mediawiki/bunch of other wikis
* google keep
* one note
* notational velocity
* anki
* evernote
What "works for me (TM)":
I use a separate email just for this. Every new thought, todo, bookmarks, quotes, images of doodles, event, information about people becomes a new email.
Updates are email replies on last email on that subject. Todos are right there in my inbox reminding me each time I open up my email (many times a day). Once/twice a month I go though my email history. Any email with enough content or insight to a "WIP" wiki page in my git repo.
Whenever I find time or feel like the need to research some concept that I left mid way, I go through my queue (my WIP page) of pending items.
If there’s enough content/files/code, it gets its own markdown file and/or directory. Common patterns, get linked via their respective wiki pages. Related topics are back linked and cross referenced.
This setup has worked well for me for almost a year now.
I do not worry about email / git becoming obsolete. Or how I would move my email/git data when I need to switch vendors. I can access it via any device I own or use someone else’s with just a browser. There are backups of my git repo in almost every laptop/desktop I have access to. There's no shortage of client applications for any of these to choose from. And above all... like I said, this system "works for me (TM)".
Something that was not obvious at the start but I do now, is to keep my wiki relatively small. Because the purpose of the wiki is to supplement and complement my brain and its finite resources. I try to keep the gist in mind and offload details and references to this non-degradable store for when I need it. I've deleted/merged more content than I have added. I routinely see stuff added to my queue and deleted because its a rehash of a concept camouflaged as something new. No other system I used did this for me.
I've posted this comment before [1] reposting as I still find it relevant.
www.notion.so
I've used Tomboy for 8 years now to:
- brief note usage - blog drafts - turning those moments of inspiration and excitement into lasting documents
In the past year I've adopted Anki for committing knowledge to memory. Topics include:
- family birthdates - UNIX core utilities - python builtin functions - multiplication up to 20x20 (still a WIP) - US bill of rights - AI/ML terminology - Puppet concepts / idioms
I’ve gone down the special note taking software and formatting route before (OneNote first, OpenOffice later). What a mess.
Tempted to try a self hosted wiki though. Definitely never touching a commercial solution again though. Evernote and co. Might replace filestream with nextcloud though
Data is stored in firebase and authentication is handled by firebase as well. Just a site to stash notes, links. Purely javascript.
I am planning on upgrading the site to support markdown. So notes can look like article. Some can be private, some can be public.
heres is a basic example Propædia in Dynalist: https://talk.dynalist.io/t/propaedia-an-outline-of-all-possi...
if youve never used dynalist before, hover over the bullet points, then click to expand or collapse the list. click the magnifying glass icon to the left of the bullet point to zoom in and focus on a certain point, then use the breadcrumb trail at the top to go back. since every bullet point has its own url you can also use the browser back button
Each technology gets its own folder. I add a new markdown file whenever I learn something new. Then I push to git and it is published live shortly after.
I use Markdown + KaTeX to render math and code. I keep everything in files.
I moved to Bookmark OS which has been great https://bookmarkos.com
TiddlyWiki (https://https://tiddlywiki.com/) is a very interesting and useful tool that matches my two criteria above very well.
Its usable immediately. Just click to create a "Tiddler" and type. Later you can manipulate/present the "Tiddlers" you created in numerous ways.
The basic version is a single HTML file that self-modifies (for interested programmers its a rare example of a pure "Quine"). It works in any browser.
Browser extensions are available for it so that it can overwrite save, rather than download save.
It will work with the cloud if you need that. Interfaces with Google Docs. Free publishing from local TiddlyWiki to online publishing is built-in for http://tiddlyspot.com. And more I can't list. See main site for info.
It has its own scripting language that includes string matching, maths and complex filters. In particular its "filter" mechanisms, that are largely list oreientated, allow complex interrogation and presentation of content.
It supports JavaScript modules allowing full integration of good existing tools.
There are libraries of plugins--both official (https://tiddlywiki.com/#%24%3A%2Fcore%2Fui%2FControlPanel%2F...) and unofficial (https://dynalist.io/d/zUP-nIWu2FFoXH-oM7L7d9DM -- hundreds) that extend function in many ways.
There are also other versions of TW that are node based. Of the more interesting branches are "Bob" (https://github.com/OokTech/TW5-Bob) that provides all of TW's base functionality plus interaction with the OS on all major desktop platforms. Another is TiddlyServer (https://github.com/Arlen22/TiddlyServer).
IMO its a very sophisticated and very effective tool for recording unstructured data in any form.
The real plus in it is that you can then, if you need, begin to structure that data in innumerable ways.
An upside is you can learn the basics very quickly. A downside is that it can take some time to fully understand all you can do in it and how to do it.
For developers its Github gives insight into how it works and current developoments (https://github.com/Jermolene/TiddlyWiki5).
It has a useful, friendly, discussion group (https://groups.google.com/forum/#!forum/tiddlywiki).
Its a kind of "Swiss Army Knife" of information tools.
In Tiddlywiki it's an outstanding central, user documented, feature of huge utility ... some basics ...
https://tiddlywiki.com/#Transclusion