Tag Systems | Hacker News Reader

Tag Systems

buttondown.email

109 pointsfagnerbrack57 comments

The worst part about going from G to FB was going from buganizer to fbtasks.

The internal tool at G has a strict hierarchy called "components", and fbtasks is tags alone.

This means if you want to file a bug against anything at G, you could navigate this large hierarchy and find a rough spot for the bug. Regardless of that team's documentation, tooling, bug tracking helper forms, etc you could always file a bug that would get pretty close. Then someone on that team would move it into the proper component if you got it wrong.

At FB... you go to file a bug, but you need to know which tags! If you miss the right tag, that team may never see it. All teams implement their tagging differently, and many implement almost nothing at all. This devolves to pretty much all bugs being tracked through internal group posts, because at least internal groups have names and membership lists that are easily searchable. Then teams implement bots & programs to take every post filed in a group, and auto generate a task with the right tags! And pray that they append the `donotreap` tag...

Bugs need to be filed against specific systems, and those systems need to be owned by teams. Those teams are in orgs, those orgs are in companies. I cringe whenever I see a bug tracker for a large company that is not in a hierarchy.

You could have a tag hierarchy, or even multiple ones. I don't think tags prevent you from navigating the hierarchy, it's just that at some point you need to build the hierarchies and make them accessible, and just having tags won't push you there.

One thought that has occurred to me when using tagging systems is that they ought to have the ability to refactor tags. No matter what the tagging scheme is initially, it will surely mutate over time. It should be possible to rename tags, join tags and divide one tag into multiple tags. And while you are at it, perhaps it could use some kind of sentence embeddings to suggest tags that mean approximately the same thing.

Stemming would get you 95% of the way there in coalescing tags. Would sentence embedding outperform that significantly?

In my opinion this is largely a UI/UX issue. For some reason, applications with rich tagging systems have only the most raw basic interface for manipulating and searching using tags.

I think something like datalog queries would be perfect for refactoring and searching tags. Would also let you declare relationships between tags, like "tagged(x, calculus) -> tagged(x, math)"

From 2001, and still the same. https://people.well.com/user/doctorow/metacrap.htm

>2.5 Schemas aren't neutral

Nowhere is this more prominent than in traditional file systems where the folder hierarchy is often used as a kind of metadata tagging system. You take a picture while on vacation and copy it to your /pictures/2020/vacations/Hawaii/beach folder. But you could also copy it to your /2020/vacations/Hawaii/beach/pictures folder.

Your pictures, documents, music, and source code gets spread out using whatever hierarchy seems right to you on the day you copied or created it. Heaven forbid you decide to change the folder structure using anything but symbolic links so it breaks any stored references (i.e. full path) to the file(s).

at the very beginning del.icio.us had a hierarchy operator: anything tagged "foo/bar" would internally also get the tag "foo" as well. i ended up pulling it out.

for finding untagged things, i had "system:untagged" automatically added if the items had no tags because mixing inner and outer joins was a nightmare in mysql.

Good information that deals with tagging issues that I have been thinking about and implementing for several years.

I have been working on a new kind of data management system that is designed to replace traditional file systems while also doing a bunch of database operations. It is an object store that makes extensive use of tagging. Nearly all file metadata comes in the form of tags attached to each file. Even its name is a tag, so every file can have multiple names (each object's unique ID is a 64 bit number).

The system is currently in beta and still needs work, but it can do some amazing things so far. It can create 100M+ files and attach dozen of tags to each one (there is a limit of 255 tags per object). Searching for all photos with certain tags, for example, can give you the results extremely fast. Even with containers with 100 million objects can give you the 100,000 that have a specific tag in just a second or two.

Tags have context. Instead of 'John' as a tag, you attach Person.FirstName = 'John' to photos of John or documents written by John. The whole system is like a sparse relational database table where each row/column intersection can have multiple values.

We are looking for more beta testers and anyone can download and try the software for free. Here are a couple short videos showing a few things it can do.

https://www.youtube.com/watch?v=yLPNLHm9fIk https://www.youtube.com/watch?v=dWIo6sia_hw

Nice. I’ve wanted something like this for 15 years.

I’m curious how you implemented them. I’ve been mulling using a tag based system for a completely unrelated project. I figure one might get a lot of way there using hash tables and sets from any of the high level languages. But it seems you maybe used a different approach?

Each logical container (a Pod) has a set of data objects called Didgets (short for Data Widgets) where each one is assigned a unique 64-bit number as its ID. Each tag is defined with a data type (e.g. Person.FirstName has type STRING) using a schema like how a column in a relational table is defined.

All tags within the same context are stored together in a highly optimized Key-Value store (also stored within specialized Didgets) like how a columnar store database stores all the data for a single column separately. So if you attach a tag like Person.FirstName = 'John' to a photo of John that is stored in a Didgets with ID = 123; then within the tag the value of 'John' is mapped to the key 123 within the Person.FirstName tag store.

This makes it extremely efficient to find all the Didgets where 'John' was attached as a person's first name. It can also find everything where the first name starts with 'J' or a vowel (using regex). The values within each tag store are de-duplicated and reference counted. In the example, the value 'John' is stored once. If 100 things have 'John' attached, then its reference count is 100. This makes it extremely fast to find everything with the 10 most popular names.

The tags were so efficient and fast that I built relational tables using them that are often faster to query than traditional database systems. In this case the 'Key' each value is mapped to is the rowID within the table.

Here is a video showing how that compares to SQLite: https://www.youtube.com/watch?v=Va5ZqfwQXWI

This is wonderful, but unfortunately only for Windows.

If you want beta testers for a Linux version... I'm more interested in the Manager with an API, not the GUI Browser.

I love a good tagging system, if it is done properly. It makes it so effortless to find the content you want.

I broadly like this post, but I disagree with "The lesson: don’t let users encode logical paradoxes.".

If a user says they want (A && !A), just hand them back nothing. Let the expressiveness exist regardless. (Ideally, warn the user, though)

I want expressive tags. It makes everything so much easier. If I can't do (A && B) || !(C or D) then as a user I'm going to get frustrated if there's a lot of content to sift through. Grouping and negation are critical, IMO.

The post touches on this, but I'd say by far the largest issue is the moderation of tags. If only the author is allowed to tag, and there's no moderation or oversight from other users or official moderators, the tag system is effectively useless. Some users will be lazy and not tag effectively. Others will tag incorrectly by mistake or intention. You want the system to support new tags for a variety of reasons, but this allows typos to create new tags.

Generally, when I see that a tag system sucks, it's because it's been mis-used, not because the underlying logic is bad.

The paradox they were referring to was A := (B), B := !(A). That's not a expressiveness issue, that's just a dependency cycle—there's no reason to allow it.

Oh, assignment? Yeah, that's likely going too far. I completely misunderstood the issue they were stating.

Can anyone tell me if anything like this exists :

A program that runs in the background on your machine and watches processes and the file handles they open. Then, keeps a database of associations between files and programs. E.g. you open a pdf with adobe acrobat, and now it adds an entry with a relation between that specific file and acrobat, and between pdf file extensions and acrobat.

Then, this information is used to provide an intuitive interface into the file system. Searching for that .docx file you made last year and cant remember for the life of you where you put it? Just search for .docx, or files associated with "Word"

This to me is the holy grail of filesystem search, and it doesn't seem that hard to implement, but I've never found anything like it.

I haven't come across of any project like that, but in case anyone wants to implement this and doesn't know where to start, here's a way to do it on a freedesktop-compatible linux:

Make a userspace daemon process that adds eBPF tracepoints[0] to open{,_at} etc syscalls which match files of your user directories with specific extensions (e.g. .docx).

Associate PIDs that open those files with their .desktop entries[1]

Store results in some database like sqlite3.[2]

Search this database with your favorite interface, like a CLI script or a GNOME shell search provider[3].

I have seen this Rust project on HN which does something similar but with file attribute syscalls, you can use it as reference: https://github.com/javierhonduco/sweeper

[0]: https://github.com/iovisor/bpftrace [1]: https://www.freedesktop.org/wiki/Specifications/desktop-entr... [2]: https://sqlite.org/ [3]: https://developer.gnome.org/documentation/tutorials/search-p...

This is wonderful, thank you. I may try this out.

Originally wanted to do it for windows but I eventually concluded it would take some prodigious hackery to get it do work

I wish I could help you but I haven't touched windows in decades. Perhaps someone else can chime in.

So as far as I can tell the only way a user can access this type of info on windows is via procmon. Problem is its just a gui application, no programatic access afaik. I did find a project where someone reverse engineered it to some extent and it allowed you to make calls to it programatically. Anyway, it turned out there was a loooot of heavy lifting that'd be required to get it to behave the way I needed it.

Anywho, I'm happy for the info on linux. I was looking for a project that would require me to up my bash skills anyway. This sounds perfect

I'm pretty sure the latter (Win8 and on?) versions of Windows had some sort of framework for writing FS filter drivers or something, that the AVs are supposed to use?

Every NT version, in fact. Driver filters including filesystem filters are core NT feature.

So you'd have to write a driver that hooked itself at appropriate step in IO Request processing, then it could inspect and gather data just like eBPF.

Procmon uses diagnostic interfaces from outside kernel, might require debug permissions.

Spotlight on the Mac has done this for like 15 years. You can search for files by content and metadata attributes[0]. You can even create "smart folders" that save the search parameters and show an up to date instance of the search when opened. The Spotlight search is also exposed on the command line with the mdfind command.

Spotlight uses the LaunchServices database of file type associations to give friendly names to file types but has a hierarchy of type identifiers[1].

[0] https://support.apple.com/guide/mac-help/narrow-search-resul...

[1] https://developer.apple.com/library/archive/documentation/Fi...

Think about this solution for a second. You want a process that spies on other processes to build a database of what files they touch so you can remember where you put that file last year.

This is a problem that can be solved simply by better organizing your files. No security nightmare Rube Goldberg contraption needed. Ive been in that boat before and it just boils down to organizing your stuff better.

> You want a process that spies on other processes to build a database of what files they touch

It's basically an extension to the VFS that would maintain atime and some additional metadata to it, what's so horrible about it?

I mean I would write the damn thing myself, Im not worried about security.

I just think windows tools for navigating the file system are straight garbage and think it could be done a million times better by just indexing interactions between processes and files. But who knows maybe im barkin up the wrong tree.

Also- organize your files better : first off easier said than done and second off, my shit is already a total mess and I am not willing to take off work for 3 weeks to organize everythibng. I just want a file viewer UI that is actually useful in allowing my to find files.

Imo the folder hierarchy paradigm shouldnt have ever been exposed to users as the primary mechanism for accessing files. It makes no sense for 99 percent of use cases for the average user.

I'm not following your use cases. Why do you need an association database to look for files named .docx or .pdf? I could see it possibly being useful for apps like VLC which open a number of different file types, but then if I used multiple apps to open videos I'd have to remember all the apps I used in order to do that search.

Basically windows search is slpw as all fuck if you search by file extension. Itd be a lot faster if you just had a DB of associations.

But also in the case that I do not know the extension of a file, or want to find all files involed in configuring a program, etc, it would allow you to find that info quickly.

(Windows) File associations database is part of the OS, no? How is it different what you need?

Search is best done via Everything by Voidtools, which is instant, though it doesn't connect to this database, but instead uses manually entered lists of extensions

But you could write integration that extracts the list from the OS database and dumps it in Everything's lists

Or you could integrate Everything in you file manager and then add the OS file associations integration there

One very prominent use of Key-value tags is the OpenStreetMap database. The datamodel is deceptively simple: nodes, ways, and relations to represent geographical locations, and a list of key-values attached to any of them to define the semantics.

I've slowly drifted into preferring tags over log levels. Strict hierarchies always turn into dead ends sooner or later.

https://github.com/codr7/cl-redb

Edit: https://github.com/codr7/cl-slog

The best way is, in my experience, let the users tag away. Then add the tags to the item descriptions in the full text search (while also holding on to the normalized data, because refactoring tags is a thing). If your FTS system allows the definition of synonyms, even better. Users get the usual autocomplete, and suggested tags. Searchers get approximation.

I never implemented this before but the tags can also hold references to the tags they are most frequently tagged with to further add content to the depleted search results. Never thought how the implementation would look like though.

A tag system is actually a deterministic computational model

https://en.m.wikipedia.org/wiki/Tag_system

Implementable in vim keystrokes! https://buttondown.email/hillelwayne/archive/vim-is-turing-c...

If you think hard enough about tag systems you end up with the idea that the unix filesystem is a nearly ideal way to tag data.

My hierarchy of tag systems goes something like

key, key=value, hierarchical key(the unix filesystem), hierarchical key=value(oh shit you found ldap, abort, you went to far)

Iff you assume the use of symbolic links. A tagging system needs the ability to have arbitrary tags and the same item referenced by multiple tags.

If you're on one file system, hard links work as well. They have they added benefit that you can delete an entire tree but files that are linked to from elsewhere won't get deleted or end up with dangling refs.

A description of the unix file system but using tag vernacular instead of the more common forms.

a file is a generic unformed array of bytes, the application can store anything it wants in this array. files are selected via tags, a file can have any number of tags, however there is a limitation that each tag can only reference one file. tag prefixes are use to group and match common tags.

I think an important part of tagging systems is that you want some way to store what tags have been considered. Like, this story has "female-protagonist" but not "teen-protagonist": someone clearly looked at the characters and applied tags, but does that mean it does not have a teen protagonist? Maybe the tag was created after someone reviewed the story in order to apply "*-protagonist" tags. Maybe that person was only doing "female-protagonist" in bulk.

Having a system that can point out unknowns is useful both for search (by including them in results) and for filling in that data (by suggesting those stories for review).

> “should users be able to query a specific tag alias?” ie, can I search for stories tagged with the alias snarry? There’s no correct choice here. AO3 went with “no”.

This is a very interesting point. My initial thought was: there definitely IS a correct choice, which is "no". If the choice were "yes", that would imply there's a difference between the tags, but they're aliases, so there shouldn't be a difference. An alias should be exactly interchangeable.

Then I thought about how practically impossible that is to enforce, given the nature of language. The usage choice itself implies a difference; there are no true aliases in language.

I think it depends on the usage is tags, and how important the tagging system actually is, because there's a tagging system in wide use that allows for aliases. That system is reactj on Slack messages. For those outside the slack ecosystem, that feature allows you to an emoji to a message, even or especially if you're not the author. Thing is, as tiny pictures, people remember what they remember about the emoji, and people often remember an alias and may not even know the offical name. Slack further allows customization by letting users add their own reactj imbuing each slack workspace with its own personality.

If someone wants to type in dumbbell and someone wants to type in weight, and it goes to the same tiny picture, is that really so broken?

It always bugs me that the default alias for the fist bump emoji is "facepunch"...

I think we're ultimately in agreement — aliases should be 'transparent' and not imbued with any specific meaning. Of course, it's much easier for a very limited system like emoji to be able to deal with these things than an open-ended tagging system, especially one that caters for the entirety of Unicode.

AO3 has many, many decisions around tagging that are quirky, at best.

I am not enamored of tags, because tag systems break down when humans get involved.

The same person will choose different sets of tags for the same file, when asked on different days; they will end up with synonymous tags that should be coalesced, but nobody wants to do that drudgework.

The semantics of each tag will alter over time and by person.

An optimal tag system is inferior to a full text search system with an indexer that recommends keywords and has a reasonable syntax for includes, excludes, and boolean logic.

That does not really speak against tags in general. You can also predefine tags for instance.

While I absolutely agree with the downsides you mention in the 2nd and 3rd paragraphs, the first paragraph is kinda trivial: all systems break down when humans get involved. As for the last one... what about images? People tag them too, and no "full text search system" exist for them unless you count "textual inversion"/"interrogate CLIP" which would still requires quite some hefty amount of manual tagging for the initial training.

This is exactly what the tag alias system of AO3—as discussed on this page—is designed to mitigate. If you've spent any amount of time on that website, you'll realize their tag system works really, really well.

I've seen it. Their tag system relies on a large group of volunteers to reconcile the problems, and all the volunteers abiding by a large set of rules:

https://archiveofourown.org/wrangling_guidelines/11

The way i would frame this is that tag systems have a steep return on effort expended. Low-effort tagging is garbage, as you have observed. But when users stay on top of it for some reason, it can be extremely valuable. Someone else mentioned the example of AO3 here.

tagging wasn't meant as a way to organize information for other people. it was meant as a way to augment retrieval.

however, your suggested search system doesn't work - many people built these and they weren't right - because they don't encode whether or not the object had attention.

> I am not enamored of tags, because tag systems break down when humans get involved.

One of the things I'm curious to try out with GPT is auto tagging content and patching documents that have missing tags. You train GPT with your corpus and existing (and incomplete) tagging system, and have GPT fill in missing tags and revise tags that are synonymous or spelling variants.

Maybe there are already good non-AI system that do this, I haven't really looked deeply into other solutions yet.

Then there's the problem of synchronization. How do you take two bodies of tagged data and merge them? How do you create a degree of certainty that the tags themselves have any similarity?

Look, you can be a horse girl if you love horses. You don't need to own one to be a horse girl. Why gatekeep, huh?

I believe you are referring to the "is" vs "has" distinction.

Polymorphism or composition?

Reply on news.ycombinator.com