My Knowledge Lakehouse
tabokie.github.io
tabokie.github.io
From a content point of view I personally find having structure but not too much structure is the way to go. A bit of yaml if needed but just a gentle “daily notes = meeting notes” (on my work obsidian) and “daily notes = journal” (on my personal one), a few tags but nothing too rigid or cumbersome.
Omnisearch is the dogs nuts plugin too, and a couple of others but again, keeping it quite minimal has really worked for me.
Just so nice knowing that everything is under my control, using just markdown but with this fantastically powerful framework underneath it. Huge fan.
There's a bunch of cool looking stuff though
Luckily for both of us there is a solution https://github.com/vrtmrz/obsidian-livesync
Works well on ios.
If you just want to support the project, they have the Catalyst once-off payment that gives you beta access and VIP status on their Discord.
Aside from not being easily accessable by phone, this works and the .obsidian folder shares the environment + plugins ( minus one for open tab/window layout ).
If anyone has a solution to the lack of git android ssh auth problem I'm all ears though.
FWIW I ended up going all in on Obsidian and paying yearly, but that outcome was far from a forgone conclusion, even a month in. I went through a full cycle of honeymoon phase, then discovering warts, and then workarounds, etc. It does take time.
I also pay for sync because I genuinely love the product (though I pay $5/month early bird pricing), but thanks to their file-first philosophy, I'm not locked in.
Totally understand if you're not employed, or barely making ends meet, but given the forum I'm going to take a shot in the dark here.
I recommend reading the article, it was pretty interseting.
Finally - it’s just a fast and solid bit of engineering. Rarely / (never, actually?) crashes, shortcuts are really powerful, constantly being updated… all manner of great :-)
The daily node is so useful because when I look at an old resource I can see from the back links exactly when I had taken notes on it before and go to that daily note to see any additional context of what I was doing that day.
I use Trello for task management and jotting down things to investigate later.
Can you explain this one? I use it and the daily note thing bugs me since I would rather separate work and personal in the same value, but I did not know I could fix it.
> 1.1222 Sometimes log from different tools or locations need to be “merged”, “persisted”.
These two are the main reasons I've stuck with Google Docs: It's available everywhere, and everything's always in-sync. Google already has all my info of value, so the incremental trust necessary to the threshold of log visibility is minimal.
I'll add one more requirement: I don't want to leave an unencrypted on-disk footprint containing my notes. This means I can access the entirety of my notes on any machine and the only thing I have to worry about is keylogging/screenscraping.
Google Doc is lacking in many respects, though. Linkability is nonexistent. Docs longer than 100 pages really struggle with latency on mobile. Searchability is bad (!) because you have to open the doc first to see the matches from within.
Really wish there was a self-hosted alternative with sync and encrypted storage that didn't result in sync errors. I've tried DEVONthink, Obsidian, LogSeq, Google Keep, Notion, NotesNook, GoodNotes, Samsung Notes, Loop, OneNote, Apple Notes, Org Mode, plain text files, and probably a dozen others... I'd say NotesNook is the best so far, with DEVONthink a close second, but, nothing beats the reliability, omnipresence and privacy of Google Docs.
Searching multiple files, editing and organising files is quick and simple on desktop, and you can fallback to Google Doc on mobile when needed (Google Drive app lets you view .md files, there's no way to edit?). You can install extensions as well (like for doing inline maths) and if you already use Visual Studio Code for coding it's one less thing to learn.
I'm not familiar with Obsidian, which gets mention a lot. What does it improve on compared to the above if you don't need a mobile app or complex linking between Markdown files?
Unfortunately, as other have mentioned, Google Docs omits some seriously impactful features.
All it would've taken, at the time, was collapsible/foldable headings for me to stick with Docs.
But since then, I've grown to appreciate the millions other things Obsidian has to offer, like the ease of developing plugins which, to me, make Obsidian feel like it's an OS within my OS.
Eventually I switched to Obsidian for mobile support (syncing with free 'Remotely Save' plugin using S3). There are 2 other features of Obsidian that I came to appreciate over time:
1. Daily Notes
2. Calendar via Full Calendar plugin that uses your daily notes as one of event sources. So I can freely mix my Google Calendar 'official' meetings with my personal timeslot allocations for current day.
It is possible to have both of this features in IDE too, but with Obsidian they come for free.
Unless you are offline
The infinite document length paradigm also made it impossible to print and review notes; text would slice mid-height across pages.
I know people who swear by the desktop version of OneNote, though...
https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F2...
Personally I’ve abandoned all of my complex attempts at productivity and knowledge systems and have recently moved to the ever reliable txt file. You can format it however you want, use fun ascii art, or just plain paragraphs.
I’ve got a daily.txt where I prepend a new days section every day and put in whatever I want over the course of the day. I also create other files for courses I’m going through or projects as needed. The nice thing is using my editor I can make a file reference and hit gf and it’ll go right to a referenced file. But this “system” will keep on working regardless of what tools or system I happen to be using.
Been doing this for years, haven’t lost anything yet. It works so well that time organizing it would have been wasted.
Idk how long these tools will exist but at least Notion allows exporting to markdown or PDF.
[edit] in the latest version it’ll even organize your grocery lists by category, for easier shopping.
For things that are mostly lists I prefer using Microsoft To Do though (I use it ever since it was called Wanderlist)
[edit] “why screenshots?”
1) To record gui workflows, walkthrough-style.
2) to record whole screens of values from guis while preserving formatting perfectly (think: cloud dashboard vital stats screens for various resources)
3) to record short message exchanges from ephemeral messaging with all the formatting intact with zero extra effort. (Think: feature discussion in a periodically-cleaned chat channel; I can always turn it into text later if I need to, recording with screenshot is fast)
4) plus now that it’s almost as easy and reliable to copy-paste from images as from regular text, on macOS and iOS, why not?
But since it is all in one place I just do search queries using my editor to find what I need. I don’t use tags, but it would be very simple to just append some #tags in my content and search for those.
Haven't set it up yet, but it's on my TODO list.
The howm-mode shows you a summary view, comprised of information (just the title/heading) from the (configurable) last 20 notes you have made. Everything goes under ~/howm/YYYY/MM e.g. ~/howm/2024/01 for notes I am making this month.
It has a simple way to enter schedule, todo, deadline, etc. which it shows just below the header that has single-key commands (s for case-insensitive searching, c to create a new note, etc.) for searching, navigation, new notes.
The genius of howm is that you can write and have very simple back and forward links to any file, URL, or tag, any kind of text etc., and you write "fragments" that is, whatever the smallest unit of text you want - it just needs a title plus anything else you want to add to it.
When you save the file, you hit Ctrl-C plus ",," and are back in the summary view.
So you can have very small sized units/fragments, which are very fast to create and save, but due to the forward and back links and the ease of searching (which will collate in a temporary list, everything found), you can "create fragmentarily but view/search collectively" as the documentation puts it.
My next step is to author HOWM notes in AsciiDoc or Markdown format which can easily be fed (since each note is a separate text file) into a converter for fancier output.
Syncthing easily keeps things in sync between my desktop and laptop.
In the very best of FOSS tradition, someone else wrote up a full PDF manual, in English. https://github.com/Emacs101/howm-manual/blob/main/Howm_tutor...
Hitting Command+N in Firefox on my Mac opens up my planner issue for the day, or creates it if one doesn't exist yet. I wrote up how that system works (a tiny bit of GitHub Actions + Pages magic) here: https://til.simonwillison.net/github-actions/daily-planner
So I completely stopped taking notes. A to-do list plus a Google Drive for my son and our stuffs are good enough. The only problem is that we are not comfortable to put sensitive information on GD so probably build a NAS for that.
I use obsidian.md for notes, which I have on phone and desktop. Nothing fancy. You just need something that's always with you, has great search, and ideally allows you to link notes together.
For work we do use Confluence but again the quality of our notes are not great. From my experience the most important notes are the ones that tag a PoC if issue X flares up.
* Some information that I don't trust myself to remember
* A canvas to refine my own ideas
* Summary of others' works.
Now it's easier for me to connect dots across works, themes and disciplines. That works wonder for compression of knowledge inside my brain.
I don't know when ideas without execution become thought of as so valuable. Ideas are a dime a dozen, the hard part is testing those ideas to see if they actually work.
But I don't use linking and all the other complexity. But if I ever needed it I could.
Then it's synced with next cloud and also available on my phone.
I like this the most because just as explained, the time line is the most accurate representation of reality. And I can very quickly scroll back to see what I did last week or do full text search.
Simple and useful!
[0]: https://github.com/Quorafind/Obsidian-Daily-Notes-Editor
(pc, circa 2010) notepad > Evernote > notepad > (mac) > vim > Notational Velocity > NVUltra (still in) beta > "The Archive" (Zettelkasten) > (n)vim
- notes synced with Dropbox folder (could just as well use iCloud) - with vim fzf/telescope plugins for search
What finally flipped me back to vim was using a setup script to give me a useful set of plugins that did nice markdown highlighting out of the box, and then working out how to do things like auto-indenting, opening URLs in browser, renaming files more easily etc.
Slight annoyance: yet to find a way to search both filenames and file contents at the same time
Do systems that link from one note to another help me? Apparently not much. Quite a lot of effort to maintain the links, better to express hierarchy in filenames.
(still using Omnifocus for todo lists, but it can import/export Taskpaper text format)
The only rule I have is to avoid unlinked notes.
function note { mg +-1 "/home/user/notes/$(date "+%Y-%m-%d").txt"; }
I like this setup because it's plaintext (or markdown if you prefer), easily sorts on filename, and easily greppable. I have a couple years of notes saved this way and it's fun to explore old notes.
1. Formatting easily becomes a distraction, but a little formatting is vital, so I use markdown to write my plans and notes.
2. Mixing streams from different projects is confusing, so each project has its own workspace - either a folder or a distinct prefix on the file name.
3. Work or ideas often become irrelevant for a while as plans change, but I hate feeling like I might lose work that might be valuable later. So I have a "dump" file or folder where I can dump such things. Entries are typically dated so they can be referenced by other files (see below). I typically use a level 1 or 2 markdown section header to delimit entries.
4. I often have a "log" markdown file where I append descriptions of indisputably important project developments (as opposed to maybe-important-later stuff that goes in the dump file). Entries here are dated and delimited similarly to the dump file.
5. There is a "plan" file that I try to keep clean and concise. This is where I go to remember what I have resolved to do next.
6. A common issue with plan files is that they get bloated with discussion of current or past context, as you try to figure out what to do and how to do it. When this happens, the discussion is moved to the "log" or "dump" files and replaced with a "link" describing how to find it again (e.g. "see log date 2024-01-12 for more on this").
7. There is a master plan file that coordinates all the projects. Bloat in this file is moved down to individual project files and replaced with a link.
To recap: markdown; project workspaces; log and dump with dated entries; concise plan file; move bloat out of plan into log or dump and replace with a link; master plan file with similar bloat management strategy.
I have used this system for five years. It has developed over time, but all elements have been in use for at least two years.
I am a staff data scientist leading a team of a half dozen people that is overhauling the core analytics pipelines and models at a Fortune 100, and 80% of my day is IC work. I have this job not because anyone asked me to do it, but because I proved I could do it on my own time. I work normal hours, parent of two kids, working spouse. I don't have time to putter around. This system is a big part of how I make it work.
Every time I see that other people are contemplating these subjects, I notice that they hit on many of my own considerations and miss many of the big picture things I consider essential.
I think this will evolve into tribalism of mental models for organizing information. More vi vs emacs energy to go around in the decades to come.
Not only that but all the people I connect with for notes/calendars/reminders are also in the Apple ecosystem and they aren't going to switch for me. This means that a lot of the extra features that Notion, Obsidian, etc. offer just aren't valuable for me.
It makes me annoyed that Apple's Notes app is so basic.
[0]: https://noteplan.co/
I find the app experience to be terrible tho.
It helped to pin down my own approach, which is a very similar workflow, with these differences:
- The log entries are flashcards with ID
- The structure / tree follows the scientific method, aka the glue is not ad-hoc.
- So far I need no lists, because I only learn things and I'm not tracking any activity
This post helped to give me some additional ideas on the processing of those events.
I have a few sources built out right now and some fun utilities thanks to the browser extension, but in the end, I’m expecting to have 15-20 different data sources that are sent by the browser extension, collected by the daemon, and polled via crons in NestJS.
I’m taking ideas i gleaned from the AI-Town paper about taking “memories” and then generating “reflections” and then doing some other stuff on top of that and also building out a knowledge graph of everything i do, read, listen to, etc..
I’m currently figuring out the standardization of the disparate events into memories to then transform into reflections and knowledge graphs.
I’d love to hear more about what you’ve been thinking of and the issues you’ve had.
Edit: One cool thing you can do with the browser extension for this stuff is to send your cookies and headers to the database and then use in the cron jobs to get the data—think pandora.
The approach taken here is essentially event sourcing, btw. It's old and it works.