If anyone would like to say what they use, and why they chose it, I would appreciate hearing about it.
Thanks in advance.
https://datasette.io as the interface for running queries against (and visualizing) my data.
https://sqlite-utils.datasette.io/ as a set of tools for creating and modifying my databases (inserting JSON or CSV data, enabling full text search text)
https://dogsheep.github.io as a suite of tools for importing my personal data - see also this talk I gave about that project: https://simonwillison.net/2020/Nov/14/personal-data-warehous...
I'm also using GitHub issues and the new GitHub projects for a bunch of personal data - my notes and TODO items specifically - much of which I can then suck into my own SQLite databases using https://datasette.io/tools/github-to-sqlite
I use this simple tool for most my notes now:
https://github.com/codazoda/nolific
For temporary notes I use my Ponder web app (it just uses the browsers DB) and is partially inspired by old AlphaSmart word processing machines:
But even in software it's not common that I need a DB, since I do a lot of embedded. Most software I write uses text, with an eye to version control and SyncThing sync.
For personal information management I use Obsidian/Syncthing to take relatively unstructured notes, combined with Google assistant for reminders. Various bits of non critical data lives in assorted apps and clouds, like Garmin connect and Google location history.
On the very rare occasion I have any personal use for managing structured data, I use Google Docs or LibreOffice Calc, but that doesn't come up a lot.
More than what tools people use to manage data, I'm curious about the data itself. I always hear about programmers using Awk and sed, and all kinds of talk about processing data, but I'm not sure what people actually do with structured personal data.
Grist [1] is an "augmented" but still "classic" relational DB builder. It is (kind of) open-source, backed by SQLite, and you can self-host it. I like it because it is user-friendly while also being highly programmable (Python formulas, widget grid) and "data-friendly" (nice filters, Merge imports...)
Tana [2] is a less classic kind of DB. The main view is an "advanced" outliner (like Roam). But every line of text could also act as an "universal Form" and a "node" in a semantic graph DB. Wherever and whenever you type, you can turn your unstructured, freeform blocks of text into "as you go" structured data. This works well for Evergreen-Notes [3], Tasks, Contacts...
A bonus : for any kind of references (and not just citations), Zotero is very flexible. As local/opensource personal DBs go, it is a solid bookmark manager and pdfs-annotations store.
[1] https://www.getgrist.com/ [2] https://tana.inc/ [3] https://maggieappleton.com/evergreens
In DB terms IF I have to write some personal software outside Emacs/org-mode I'll choose the simplest and lightest tools I can found, case by case, however so far I have encountered no personal cases...
TW is relatively future-proof, as it's just a single HTML file that encapsulates all of the content and app in a single file that can be moved anywhere or used in any browser, but it gives you powerful structuring, storage, and linking capabilities inside it. (I almost never use flat text files for reference any more except for the simplest things, or if a machine needs to read it...) It gives you the best of what your filesystem can do for you (and filesystems are DBs, especially these days) combined with browser/wiki capabilities, and you can even hack the JS if you're into that sort of thing. My perfect solution would munge together the capabilities of TiddlyWiki and Jupyter notebooks into one awesome live, flexible, portable, programmable metacontainer.
(I started on this with the first version of TW 20 years ago, combining test programs, data, and analysis/viz tools for sensor data collection into what I called "testacles" - test data receptacles capturing everything about a real-world test with IoT data, setup, docs, notes, code, data, etc. (I was building the world's first embedded IoT sensors at the time...))
BTW, thinking of adding BOOX's new Tab Ultra into my workflow - I've used pen on various Surface Pros for the last eight years (how OneNote became my go-to), but like the idea of not worrying about battery life. The Tab Ultra can run Android apps and a browser, so I should be able to use TW even if it's own pen note capture is too proprietary to be useful outside the device... we'll see.
it stores:
- My TODO list (items -> entries -> checkboxes)
- My credit card transactions (scraped from alert emails with gmail api)
- A running log with miles/times/routes
i've implemented misc features to organize everything like content tagging, custom markdown rendering, etc.
Whichever seems appropriate for the given task I'm accomplishing.
I have some that are in PostgreSQL. Others in SQLite. A few that are just files organized in a filesystem directory hierarchy. There is really no "one size fits all" solution.
Outside of software dev I use a bit of Facebook to store social contacts, iOS contacts for phone numbers, my gmail inbox as the only source of email addresses… I use Google Keep to share a grocery shopping list.
I avoid like the plague anything that’s a file, script, database etc. I don’t take notes, don’t use web browser bookmarks or in any other way try to organize/persist/index data around me. I’m not 100% successful because I still self-host photos (hey Lightroom is SQLite!) but I’m getting there. Spotify is a godsend compared to organizing music files.
A couple of my older scripts still use JSON files on disk that are loaded into memory, which are actually perfectly fast for the purpose (fast enough that I see no perceptible difference between them and SQLite).
It also runs pretty well on smallish machine instances.
Why MySQL? I know it in and out, since I was a MySQL DBA for a few years. It doesn't surprise me anymore, and I can quickly get it to a point where it requires effectively no intervention on my side.
In any case, for files I use iCloud and for notes I use Obsidian.
I personally would like to toy around with some graph database for knowledge management but haven’t found the time to get into it, yet.
Other than that, it’s dendron (and git).
PostgreSQL handles my finances, my database of quotes & collection of notes.
I can send any file up to 2GB, any number of times. Can even set reminder messages. It's great because I don't have to go anywhere else to put things to remember.
It's slow as molasses to write and read though.
Apple notes for things I type quickly.
Elasticsearch for larger things like transcripts of classes I teach or take.
Firestore is bad at some things (e.g. try running a "where" filter query for 'date > X', but "orderBy" some field that isn't 'date').
Also obviously Firestore is very bad for any kind of text searching, especially searching for fuzzy strings.
Also Firestore gets expensive if you're expecting to do a ton of reads/writes in a day.
But for 80% ~ 90% of projects I swear by Firestore.