Hacker News Reader

Search works brilliantly for your own notes because you know what you're looking for. You wrote the notes. It fails for problems that you can't describe well though, especially if you don't know the technical jargon to search for. Directories and tags can work for that problem because you can easily see relationships like a hierarchy of folders, or what other tags a tagged item has. Tags and folders also act as a prompt to classify and organise things, which occasionaly leads to discovering how things are connected.

"How to find stuff" is a space with many solutions to similar looking but subtly different problems.

> “You wrote the notes.”

HA! In my case I wrote my notes, but don’t remember what’s in ‘em. It’s endemic for learning.

folders/tags is a false dichotomy.

This is how I tell people to use Confluence. There’s no hope of ever having the general set of all articles organized in a way that makes sense so just try to make the article searchable. If it’s about an error, make sure the error message is in there. If it’s about configuration of a feature make sure the feature name and “configuration” is in the title. Also make sure to write a paragraph describing what the article is about and different times someone might need it.

It’s like internal white hat SEO

Making docs discoverable is good, and this works well for retrieving mostly static documentation and tickets. It gets complicated when when resulting items are used for troubleshooting or new documentation and tickets.

When results, e.g., include multiple tickets and documents referencing the same error or configuration, which do you use for troubleshooting? Which results are the authoritative / canonical / source of truth for documenting the latest configuration directives?

Versioning used well for the input helps. Organization and ontology is additional context for using search results well.

> So, all of the Python-related notes go under the "Python" namespace. All of the database notes into "Databases," and so on. All the did was shift the problem space. Which namespace does the note on SQLAlchemy go in?

You don't need to have a perfect heirarchy, you just need to have a heirarchy that you'll remember.

If you constrain yourself into trying to produce a Liskov free heirarchy you'll tie yourself in knots.

I'd probably put SQLAlchemy under Python because it is python-specific. Databases should be kept for your notes on PostgreSQL administration (and you may have python scripts in there but they'll be to serve the purpose of administrating a concrete PostgreSQL install or something.

And you don't need to find all the database stuff and collect it together, you just need to know where you put something. So largely put it in the folder that makes the most immediate sense to you and don't overthink it. Later if you hate it, refactor it. But remember that you're dealing with your own personal hash table for retrieval and not trying to implement group_by. The latter issue is where you'd need an index over your notes that was searchable, but I don't find myself needing it.

> To put it succinctly, my notes do not fit into a DAG.

I think you mean "tree" or "hierarchy"? A DAG would be a superset of tagging, and could place SQLAlchemy under both Python and Databases, as long as you consider the edges oriented (i.e. there's a parent-child relationship).

Doesn't solve the tag maintenance issue, though.

Technically, if their notes can't be represented by a DAG, they also can't be represented by a tree either.

Right but the point is that the notes, as described, can actually be represented by a DAG. They just can’t be represented by a tree.

That's a nice summary of the problem and a solution I think I can agree with. One quibble is I think your notes could maybe fit into a DAG, but (as I think you meant) not a tree. In practice I don't think there's an easy way to do that - it would mean lots of sym or maybe hard linking and I don't know enough about filesystems to know if that's feasible.

But the one area that curation carries a big advantage over search is in browsing. You can do maybe some things with topic modeling and recommendations to allow a kind of browsing, but with searching it's really hard to know whether you have thoroughly covered a part of the space via searching, while with hierarchical curation that is easy. Filling in that last part with a good solution would make searching a no brainer over curation I think, but IMO I don't think most search solutions try to handle that right now.

> I wound up with many articles whose tags no longer (or never did) match the content.

I'm curious about your tagging strategy. Could you show some example of this issue?

I feel like my tags are very different since they're barely curated, but I can't imagine ending up with a "wrong" tag. I may miss some and have to add them later, but i can't remember ever removing or changing a tag. That's both in pinboard and in my scanned documents.

for me, folders are just a limited form of tagging. since i switched from mutt with its folders to sup which uses tags for email, i don't want to look back. first of all, my extensive list of mutt folders was trivially translated to equivalent tags, so i could continue as i was used to. but then, tagging allowed me to create additional tags as i needed them. i didn't tag everything, and sup also indexes all mails and also has search, so in fact i get the advantage of both tags and search (and i can save searches and treat them like virtual tags/folders as well).

well, effectively, tags too, are just a specialized form of searching.

the one thing i don't see is the problem with tags getting out of sync.

yes, it happens. tags do become obsolete, but that is rarely a problem. it just means that i find get more results than i should otherwise. occasionally when i see way to many obsolete tags i go and clean up that particular category of tags. if it is just a few then i ignore them.

and if i look at a tag and i don't find what i need, then i search for it, and add the tag when i find it so that next time i'll find it faster, because with more than 2 million emails in my archive search alone is not good enough.

> for me, folders are just a limited form of tagging.

I understand that this is a very subjective thing... but folders offer an exclusion which isn't immediately available via tagging. Tagging is implicitly an OR operation.

Now, your search engine might offer ways to add "not X", but it's a tradeoff. It you have highly-compartmentalized bits of info you end up with the problem of "how much do have to explicitly exclude from my search?".

It's complicated.

Personally, I think we're missing a level of organization somehow.

> Tagging is implicitly an OR operation.

Doesn't have to be. I can't think of many tag-based search systems that default to OR; usually the default is AND, and that's pretty sensible.

There are plenty of text-based search systems that default to OR, though (see also: pretty much every mainstream search engine), and it makes searching a fucking nightmare, so I definitely get the aversion.

not sure i follow. can you give an example how exclusion works? all i see is that without using symlinks i can have only one folder/tag per item. i can choose to do that with tags too if i want, but i don't see how using folders helps with that other than forcing me to do it.

If you can nest folders, then a thing is all of its parent folders in addition to the folder it is directly contained in. This is the AND relationship, which you don't get with most tagging systems.

To highlight exclusion, consider an example- folder Animals with subfolders Cats and Dogs. A thing can be either a cat or a dog, but not both.

but that's just tagging discipline. if i tag an animal with cat and dog, that's obviously wrong. but i don't need folders to help me avoid that mistake.

if an object has five tags, it is all those things. that looks like an AND relationship to me.

the rest is search. if your tagging system only does OR searches, then that's indeed a problem, but of the implementation, not of the concept.

and there is no reason both can't coexist.

one application that i use extensively is kphotalbum. it allows me to organize photos by tags, categories, date, folders and other attributes all at the same time. i can navigate and search photos based on all of those attributes. usually i drill down a folder, and then pick tags from within that folder, or i pick tags at the the top level and then work with the folder structure of the selected images. ironically, kphotoalbum does not do well on OR relationships or exclusion of specific attributes. i often use temporary tags to help me there, which kphotoalbum makes trivially easy to use

The final step GP is describing would correspond to notmuch[1] for mail. Have you looked into that?

[1]: https://www.notmuchmail.org

> Everything is a Tag. Users are just Tags with passwords.

I like tags which are structured as a tree, so that you can tag something as “Python” and it will also acquire the parent tag “Programming”. When you add a new tag for the first time you still need to eventually curate it by giving it other tags as parents if applicable. I suppose this is roughly equivalent to a directory structure where each item can be in multiple directories at the same time.

That's a really valuable insight and well said, thanks for sharing.

I wonder if tags could work like spam detection. For each tag mark a few things as "tag" and "not tag" and let an AI figure out everything else.

If there's a tag you never look at, and so it's poorly trained, who cares? You never look at it. Commonly used tags will be better trained, because you will have marked more things as "yes tag" or "not tag".

How good is the search performance? I have used an a couple of apps that provide full text search, but it is very slow when the number of notes is large.

I think the best solution is a mix of 3, folder structure with tags, full text search when needed

I feel like you must be implicitly tagging your notes.

Let's say you have notes about SQLite. If you don't put "SQLite" in the note, how will you find them?

select * from notes where upper(text) like '%SQLITE%' :P

My point is "sqlite" is effectively a tag.

Right. If you search for "database" instead of sqlite it will only show up if you happened to use that word in the notes. If you're searching notes from years ago you might not remember the exact keyword you need. You need a search engine with a powerful enough synonym/grammatical similarity capability.

exactly the same sold me on spotlight search on macos. Have to sign "for all of their other faults got one thing extremely right" as well, sigh.