My Hacker News knowledge assimilation stack
abishekmuthian.com
abishekmuthian.com
You still have to remember those keywords you added so you haven't gotten rid of the memory problem. If even one moving part fails to function properly, the whole system will break so you haven't solved the maintainability problem. And clearly, this doesn't solve the time problem either.
When I bookmark something on HN, it is usually because I don't have enough time to read it or I want to read it again later on. I don't bookmark comments or save them. Not because they aren't interesting but because informal content like comments are really easy to understand. They are short and to the point.
So I take what information I can and move on. If I forget...well, maybe it wasn't that important to remember in the first place. If I forget but want to remember, which is rare, I can just go through the bookmarks, open the relevant link, and go through the comments again or Ctrl + F if I remember something. This is rare. So rare that I can afford to do it manually.
The problem with "knowledge assimilation" aka note taking is that after a certain point, you have more "knowledge" than you even want. The extra becomes noise. At first, the idea to clip and save anything interesting seems so fascinating but soon you realize that you have saved so much that it's become a problem.
That's why using the right tool for the right job is important. Bookmarks are perfect for things you want to save without them getting in the way of more important things. They are just a list. Nothing overkill. No complex organization. No complicated processes. Just press a button and viola!
I do, however, commend the ingenuity of the OP in making such a system. I wonder how long it took to set this all up.
>The problem with "knowledge assimilation" aka note taking is that after a certain point, you have more "knowledge" than you even want. The extra becomes noise. At first, the idea to clip and save anything interesting seems so fascinating but soon you realize that you have saved so much that it's become a problem.
This doesn't matter at all. The thing with Obsidian and other bidirectional linking tools is that if you never see a note again, if you never link to it again then it doesn't matter. There is no cluttering the system.
It’s important to look a second time to remove the noise. For me, I prune every other week and usually takes 10 mins tops.
If nothing else, my brain gets a second chance to add the idea to memory.
> It seems simple in words but consider how many steps, tools, plugins, devices, human intervention are required to achieve it. Is it reliable? Nope. Is it better? Nope.
I had to create this setup because 'I had the need', Browser bookmarks alone just didn't cut it, I have at least a decade of bookmarks, For old bookmarks I have to check every page with a particular tag before I can get some value out of them.
In spite of trying numerous browser bookmark-organizing tools[1], My need wasn't addressed completely.
> You still have to remember those keywords you added so you haven't gotten rid of the memory problem.
Knowledge management suites specifically address this with 'knowledge-graphs'. All I have to do this put some thought into naming the tags when saving(highlighting in my case), link to relevant notes as and when I write them .
Although I wish I could address the memory problem itself[2], I can now also use spaced-repetition for the content I'd like to remember(Even Kindle apps have built-in flash cards feature).
> The problem with "knowledge assimilation" aka note taking is that after a certain point, you have more "knowledge" than you even want. The extra becomes noise.
I don't think I'll ever have more knowledge than I ever need, Intellectual curiosity is what makes life less painful to me; literally and metaphorically.
> I do, however, commend the ingenuity of the OP in making such a system.
Thanks, I like to reiterate that 'I don't claim my-setup to be the best setup for the task', It's evident from this thread that many others use what fits them and 'HN -> Kindle -> Obsidian' happens to be mine.
> I wonder how long it took to set this all up.
As a developer,
HN to Kindle: ~14 hours for the first release, Further development/bug fixes took few minutes to an hour every now and then(Check change log). Not counting the go based web framework I re-use for web application (or) using existing ML models for classification of HN stories.
Kindle to Obsidian: Couple of minutes to find the plugin, Few more minutes to find what I need from Nunjucks template, A week to work with the developer to fix the feature parity with Nunjucks on the plugin(Effort here completely is from the developer).
Obisidan: Since I already use Markdown/ YAML for all my writing, There's was not much of a learning curve here.
I carry on incremental development as I have finite time(due to health reasons) to work on something each day. Someone with lesser constraints could do all the above development within a day.
As for an user, This complete stack could be setup in 10-15 minutes.
Disclosure: The links below are from my problem validation platform, I've linked them to show that I've explored some of the problems stated by the parent.
[1] https://needgap.com/problems/57-i-forget-my-web-bookmarks-qu...
[2] https://needgap.com/problems/41-human-memory-lack-of-thereof...
> The critical issue with this so-called stack is the effort to set this up. It seems simple in words but consider how many steps, tools, plugins, devices, human intervention are required to achieve it.
This makes it pretty clear that you don't actually understand the system very well, because it is not very complex. Let's count the tools: HN to Kindle, Kindle, obsidian-kindle-plugin, Obsidian. That's four pieces of software (five, if you count HN), and you only need to set-up/manage three (two? Obsidian works really well out-of-the-box) of them. This could be done in, what, two hours?
> Is it reliable? Nope.
Incorrect. Software doesn't just break - people change it. The only piece in this stack that the user can't control the updates of (which is the only way they could reasonably break) is HN - and HN has a very stable "API" (I assume he's scraping HTML, and HN HTML rarely changes).
> Is it better? Nope.
Also incorrect, and this is the thing that gives away the fact that you have no experience with knowledge management - or even serious reading, for that matter. Almost anyone who reads a lot (or has studied human learning) can tell you that retention is lower when reading from a screen than from paper (and a Kindle is closer to a piece of paper than to a screen, from a cognitive perspective), and anyone who's done knowledge management can tell you that tags and highlights are incredibly valuable tools for learning and KM.
> You still have to remember those keywords you added so you haven't gotten rid of the memory problem.
Again - a naive statement that wouldn't be uttered by anyone who does KM. You learn frequently-used tags very quickly, and can always refer to the other ones (or just adopt a "create a new tag if it doesn't exist, and then rewrite later" strategy) if you don't remember a name. I've been using a KM system that is tag-like for several years now, and I learned all of my frequent tags within a few hours of continuous use. Memory of the tags just isn't a problem at all - and they make it significantly easier to find content, so they do solve the memory problem.
> the whole system will break so you haven't solved the maintainability problem.
See previous comment about software stability - it won't break unless you want it to, or unless the HN API changes.
> And clearly, this doesn't solve the time problem either.
Yes, it does. After a fixed up-front cost of setting up the system, it makes it easier to capture and retain knowledge, and learn/take notes, for the rest of the life of the system. You should look up big-O notation - this system has a setup cost of O(1).
> When I bookmark something on HN, it is usually because I don't have enough time to read it or I want to read it again later on. I don't bookmark comments or save them. Not because they aren't interesting but because informal content like comments are really easy to understand. They are short and to the point.
You must not read some of the more interesting comments, then, because I have dozens that are multiple long paragraphs long, many of which are difficult to find again without explicit management (e.g. https://news.ycombinator.com/item?id=27017072, which I've had to look up on five separate occasions now, and is difficult every time because it's not tagged properly).
> So I take what information I can and move on. If I forget...well, maybe it wasn't that important to remember in the first place.
Try applying that strategy to getting a degree in college and see how far it gets you. (there's a tiny chance that it worked for you - but I can assure you that the vast majority of college students have to study in order to pass their classes)
> If I forget but want to remember, which is rare, I can just go through the bookmarks, open the relevant link, and go through the comments again or Ctrl + F if I remember something. This is rare. So rare that I can afford to do it manually.
Again, it seems like you're not reading the more interesting comments. Or, perhaps, you're not reading very much - I have over 6800 bookmarked HN comments and comment threads (derived from reading dozens of comments a day), which also happens to be impossible to "just go through the bookmarks" bookmarks are simply inadequate for that scale of items.
> The problem with "knowledge assimilation" aka note taking is that after a certain point, you have more "knowledge" than you even want. The extra becomes noise. At first, the idea to clip and save anything interesting seems so fascinating but soon you realize that you have saved so much that it's become a problem.
The "extra" only "becomes noise" if you don't have a good system to store it (which, as we've established, you don't). Folders aren't adequate. Tags are slightly better. A Gmail-label-based system could probably handle hundreds of thousands of items, at least - and I've never found a piece of information in my system that would have taken longer than 20 seconds to find if I had labeled it properly.
> That's why using the right tool for the right job is important. Bookmarks are perfect for things you want to save without them getting in the way of more important things.
From an organizational point of view, bookmarks are just bookmarks. They're not an organizational tool. However, if you meant "bookmarks plus folders and tags" then you're a little more on-track - such a system seems to scale into the tens of thousands of items, at least based on my own collection of that size - but all existing browser bookmarking systems are still woefully inadequate for someone serious about learning on the Web.
From a note-taking and implementation point of view, bookmarks are even worse - browsers are not good note-taking tools, and running a full browser on an e-paper device like a Kindle is a bad idea (if it's possible at all).
> They are just a list. Nothing overkill. No complex organization. No complicated processes. Just press a button and viola!
The listed process is only "overkill" if you either (a) have something approximating a perfect memory (which means that you don't need to do knowledge management in the first place - lucky you!) or (b) aren't really interested in reading or learning that much from the internet. As someone who is interested in learning gobs of stuff from the internet, I can assure you that not only is this system not "overkill", but it's inadequate - I would need better tools than it (although mad props to the author for taking many steps in the right direction relative to other currently available tools). Neither is it "complex organization" - it's not powerful enough. And it's definitely not "complicated" by any stretch of the word, certainly not compared to the complexity of the platforms its running on, nor is its difficulty of use comparable to that of using those platforms in the first place.
> I wonder how long it took to set this all up.
Your subtle jab against the author putting in quite a reasonable amount of work to satisfy a real need isn't lost on me.
This seems like a perfect Chesterton's Fence moment - you don't understand why people want to do knowledge management, so you don't get the point of the system. There is a need for it, though - a need that's shared by the author, me, half of my friends, and all of my co-workers at my day job. You seem to be an outlier in that you don't understand that need (especially as a HN user), so I suggest you learn why people do it before making comments about it.
Moreover, you seem to have the mindset that it's a good trade to accept simple, inefficient tools that don't provide much value to your workflow as long as they're easy to learn. It's a popular attitude, but almost always misguided - I strongly encourage you to re-think it. The definition of a tool is a device that allows you to either accomplish a task that you couldn't in the first place, or makes it easier to accomplish a task that you could already do. Tools are meant to increase efficiency, and computers are very good at automation. Throwing away useful tools that can make you more productive for the rest of your life just because they require more upfront work is the epitome of foolishness.
I use Org Capture to bookmark links using a Chrome extension on my desktop. Org Capture also allows me to clip comments too from a thread. I highly recommend it for a process like this.
As for Emacs learning curve, it's small when compared to the amount of time I would have had to spend learning other systems (over and over and over . . . ) over the past two decades. Emacs lasts, and so does the learning you spend on it.
However in this context, I wanted to ask, why not use tools like https://hypothes.is/ for annotation, notetaking on websites? Ideally it is a disconnection of HN notes and Kindle book highlights, but I've derived a lot of value from the Hypothesis tool (it's self-hostable as well!).
I've heard good things about Hypothesis but I haven't used it yet. This thread has various tools for knowledge management with each sharing how and what they use it for; Exactly what I wanted from this discussion.
I've had Devonthink recommended to me before, Unfortunately I don't use macOS anymore but I'm glad it exists for those who do.
I do also live in my note taking app, Bear, and I tried a similar system with bookmarks: I was using Pinboard and set up automation to ingest bookmarks into Bear directly. It didn't work out because 1) too clunky, too many moving parts 2) automation was removing ME from the process, I was not able to "curate".
What I do now is just dropping relevant links in my notes, creating new ones when needed. It's dead simple and ugly and yet works way better than any other bookmarking system I tried.
Creating notes from ebooks is definitely more complicated, because we are constrained by the Kindle interface. I have yet to try connecting the highlights to my notes, thanks for the inspiration!
Using highlights, notes on Kindle and exporting `My Clippings.txt` opens a wide range of possibilities and should address your problem too.
It's too bad that Amazon doesn't store the highlights, notes in the .txt file on its mobile apps and the cloud sync works only with the books bought from Amazon; Though individual highlights/notes can be exported as .html from each book regardless of where the books is from.
Then I created tinygem.org [1] to automate this process with a browser bookmarklet. Now I had a place to send stories I want to read later.
Finally I added auto-discovery based on my saved links, because a lot of interesting content gets lost in HN New, Reddit and other places that I simply do not have time to check daily.
I would use your tool, if you had a super simple mobile app that would pop-up as one of the "share to" options in other apps. For example, I am on Android and whenever I come across a useful link or story, e.g., in a HN app (I use Hews), I tap "share" and select Pushbullet from a list, which saves the links in my Pushbullet account. If Tinygem was one of the options, I can absolutely see myself using it, because Pushbullet is pretty clunky for "knowledge assimilation" (no search, tags, etc.).
I couldn't check to see the feasibility of doing the same with tinygem because the registration page is broken.
You can't add it directly, but you can take an existing bookmark and change its URL to the javascript: link.
Do you have any stats on how many re-visit their saved bookmarks and in what ratio i.e. saved:retrieved?
I ask this because an acquaintance of mine wanted to build a reddit saved posts manager and I was like 'no-one visits the reddit saved posts again' (or) at-least to an extent where building a platform for it matters; I went on to create a poll on reddit meta subreddit to prove my point.
I was proven wrong as a need-gap for 'Search and organize saved reddit posts/comments'[1] was posted on my problem validation platform with solutions to address the same. Around 2500 people have come to that thread by searching for a solution that problem.
Let’s me query: “what was that comment I read about X last month?” Pretty well. I haven’t touched it in over a year but I think it’s still scanning each night…
If it's interesting, bookmark, if you need something, Ctrl-F in your bookmarks.
If it's immediately related to your work in a way that can be applied right now, don't bookmark it or make a note or anything.
Just... apply that to your work.
I applaud your minimalistic workflow, however, ive always found bookmarks next to useless (i still use them). Of course thats my own problem, just pointing out what benefits I get using other tools.
Yes you can. Use sqlite on the underlying datasource and some minor scripting.
> ive always found bookmarks next to useless
You need to change your view on bookmarks: they are mostly a data source.
Nobody maintains a carefully crafted tree of bookmarks, if only because interests and personal ontologies change with time.
Instead of expecting yourself to click on your bookmarks, see them as providing extra data to create your own JIT approach, that will depend on algorithms and automation instead of requiring you to do many things by hand (including reading and summarizing)
If you want to write something, here's a simple candidate heuristic for such a score:
- level 0 is if you've visited the page at least once (if you haven't, you can't include it in your search)
- level 1 is if you've visited the page more than once, then and how frequently / recently
- level 2 is if you've bookmarked the page
- level 3 is if you still have a tab or a collection (in Edge) featuring this page
The Ctrl-F approach is the simplest way to test the efficiency of a JIT approach, and to let you compare it to the complex proactive approaches often suggested (which require effort, and will waste a lot of time for sure on the 99.99% cases where YAGNI)
Notice how without even touching sqlite, you can Ctrl-F in different places to hit level 0 (history), level 2 (bookmarks), level 3 (this one depends on your OS and how you can search active windows)
You'll get over 80% of the benefits for far less time spent on it.
If you really insist on making some DIY solution, using the time gained to improve the way you can use these datasources by putting them under one umbrella is likely to give you more gains than any approach requiring so many manual steps and tweaks.
It's like new years resolution: you'll give up after a while because of the effort/commitment required.
Except it's worse, because you'll have wasted time and money creating a complicated knowledge assimilation stack, while you could have use the same amount of effort to do something you're more likely to stick to, as it requires less effort.
The approach suggested here still requires highlighting and writing a note. I have no time for that: I read, and if something is interesting, I bookmark and hope I will remember the keywords.
And if I didn't even bookmark, as long as I more or less remember some keywords, I will be able to find that on DDG in a few minutes.
> You need to change your view on bookmarks: they are mostly a data source.
ill try to integrate this mental model.
But still this seems far less flexible for me then what is currently set up. I spend very little time maintaining my setup, actually to me its conducive to how i read articles in general. Highlighting for example, if i see something important ill reread the sentence until im sure i understand what of its point/implications. i nearly always have time to highlight something..
> I bookmark and hope I will remember the keywords.
i dont see this as a reliable system. My memory is not perfect.
> Except it's worse, because you'll have wasted time and money creating a complicated knowledge assimilation stack
its no different then your 4 level process just outlined, which even relies on sql, a deal breaker for ~95% of the population. you still have a mental model for your system, same as i do. You're correct on the money aspect. But to me the value far exceeds the cost. The setup is frankly very simple, save article for now or later, read, highlight anything interesting and add my thoughts, those automatically appear anywhere i need them device wise, and spaced reptition is automatically sent to me each morning. I wouldnt switch to manually writing SQL queries of bookmarks and relying on my memory if you paid me.
> It's like new years resolution: you'll give up after a while because of the effort/commitment required
On the contrary.. i rely on simple short habits and system building to be efficient long term. My setup fits both those criteria.
--
But again, this stuff ultimately is a preference, your tone almost seems to suggest everyone should only set bookmarks and search their contents using SQL queries, which is mind boggling to me. Your perspective is appreciated however.
It totally is. It's also far less costly in time.
> i nearly always have time to highlight something..
I don't. I guess we have different usecase
> My memory is not perfect
Exactly! And it's a feature!! See that as a mechanism to purge an entry from the cache if it hasn't been refreshed for a while.
The problem with bookmarks (and TODO, and everything) is that they tend to accumulate, since there's no natural pruning mechanism. So you need to only use them as a data source, or augment them with some algorithm to extract what remains relevant.
> you still have a mental model for your system, same as i do
My model runs in the background, and doesn't depend on any action.
If at any point in time I need something, I'll fire up a query that will hit all datasources to give me a list of possibilities sorted by this rough score, so something in edge collections (level 4) will be above something in level 3 etc.
Someone here gave stats: in 4 years, for 7000 entries, they used their system 20 times. That's like a 0.28% usefulness. It's not worth spending much time on.
> I wouldnt switch to manually writing SQL queries of bookmarks and relying on my memory if you paid me.
Well, if your system makes you happy, stick to it.
Personally, I prefer to optimize my workflow for 99.72% of the usecases than the 0.28%
> this stuff ultimately is a preference
Totally
> your tone almost seems to suggest everyone should only set bookmarks and search their contents using SQL queries
If you got that impression, sorry. I think your system would work for people who are say researchers and need to carefully read and categorize everything.
My personal belief is that isn't the case for the modal HN reader, and that they'll be best served by being lazy and avoiding premature optimization - and if they really want to optimize, maybe they can start tapping into the data that's already collected (bookmarks, timestamped visit) some of it automatically instead of creating a parallel workflow that's highly dependent on their ability to do some manual tasks (highlighting, reading, annotating) that will require a lot of effort considering it will be useless about 99.72% of the time
I also want to do as little work up front as possible, but I have spent so much time trying to find something I remember that I read that I try to optimise the process.
I do that (on the Mac) by automatically indexing all the websites that I visit via the app HistoryHound. That way I am pretty sure that I can find a website again by searching by words (that I think) I remember from the content.
I have also tried a lot of external bookmark managers and have now settled for an app called BookMacster. Reading a webpage I then just press a key combination and get up a small window where I type in tags (with autocompletions from earlier tags) and if I want, copy and paste an interesting part into the note section of the window. This way I get tagging that is non existent in Safari and can automatically push bookmarks back to all the browser that I am using. (Not affiliated in any way with these apps by the way).
Finally, sometimes I wonder when I read a page for the first time. I can't see that directly in Safari as the browser only shows the last visit in its history function. So then I access the Safari History via sqlite using something like this:
[$] sqlite3 /Users/myuser/Library/Safari/History.db
sqlite> select datetime(history_visits.visit_time + 978307200, 'unixepoch', 'localtime') as dantime, history_visits.title, history_items.url, history_items.domain_expansion, history_items.visit_count FROM history_visits INNER JOIN history_items ON history_visits.history_item=history_items.id WHERE history_visits.title LIKE "%My Hacker News knowledge assimilation stack%" ORDER by dantime;
I didn't remember where I picked up the idea for this script, but using HistoryHound searching for 'dantime' it looks like I got it at 2020/05/11 19:03 from: http://2016.padjo.org/tutorials/sqlite-your-browser-history/...
So there: Ate my own dog food :)
It's one of the reasons I added classifier to 'HN to Kindle', Instead of manually adding tags to stories now I receive stories according to the categories I want.
If plain bookmarks works then it's great, but I use Markdown for majority of non-code content anyways(From reminders to this very blog) and referencing to a content already on Obsidian saves me time.
So true. Eventually you'll give up, given enough evidence that it's useless.
I mean, 20/7000 is like 0.28% and optimizing for a fraction of a percent is what I define as "overkill"
I do confess I am a digital hoarder and it is complicating my life.
Edit: and of course there are no backups and the installation is so old that the sqlite DB can't be migrated to mysql :D
You Aren't Going To Need It.
EDIT: You Ain't Gonna Need It. Thank you for the correction!
> You Aren't Going To Need It.
There is no T in YAGNI, the phrase is 'Gonna' rather than 'Going To'.
The original also used Ain't rather than Aren't, but that doesn't affect the acronym.
Especially in HN comments or Twitter treads (careful who you follow!)
- I use Pinboard - I tag interesting comments with my own set of tags e.g. "culture", "persuasion" etc
That's it!
my Pinboard has all of my favourite pages, tweets, comments etc so fits in nicely.
instapaper, readwise and Notion are not perfect (standalone or in conjuction). But they do work pretty well for now. I eagerly await a future where these tools become more robust.
go to readwise -> dashboard -> export -> Notion.
be sure to test its working by doing the flow over a week or so, i remember the initial setup was not seamless, but i just went to look now and its been doing its job well for a while now. Youll want to ensure it exports when you take new notes, so you dont have to manually export. Cheers.
Before Obsidian, I tried 'Kindle 2 Notion'[1] to sync My clippings.txt from Kindle to notion, But Notion is too heavy for my liking and I didn't use it for any other purpose.
Obsidian is built upon self-hosting principles, The obsidian-kindle-plugin referenced in the post is useful for anyone to self-host a read-wise type setup.
[1] https://github.com/paperboi/kindle2notion (Warning: Project is likely abandoned).
ill be looking into obsidian in the future, for now Notion works well for a lot of different things i do
OP seems to be reading HN on the Kindle with a pay for subscription kindle app, that he sells. Via a free kindle to obsidian plug-in for obsidian, these are transferred into obsidian, which is a freemium knowledge management app.
Edit: I have added a sentence about the subscription details now.
Replicating the obsidian plug-in functionality to suit your own note/knowledge management app would be probably even easier.
What I don't like is the tagging part - kindles are terrible to type in. And if you want to avoid it by using your phone/tablet, you'd be better to avoid kindle app altogether and do the note taking using browser extension or something.
Did you get a chance to try the recent Kindle SW update(> v.5.13.7)? Performance has improved a lot, Tagging (Notes) is smooth due to a surprisingly responsive keyboard.
The browser is out of experminetal too.
I wrote this comment from a Kindle Oasis 9th gen.