Hacker News Reader

Comment by agileAlligator | Hacker News Reader

> Pointlessness of Organization: my Calibre and Zotero libraries are a mess. But is that bad? Is there any point to organizing them? I can always find what I need, either by searching or browsing, because I have a spatial sense of where each book is in Calibre’s big grid view. If I went through everything in Calibre and Zotero, and fixed the titles, added missing authors, publishers, publication years, fixed the cover images—what then? What have I gained? Nothing. It is a waste of time to organize things too much.

On reading this, I felt like I already sort of knew this, and this internet stranger validated my thoughts.

View on HN

About once every year or two, I remember something I read maybe 15 years ago and realize it would be absolutely perfect for some reason, but I can't find it. And being an engineer and an HN poster, my brain immediately leaps to "Oh, if only I ran a system that archived everything I browsed so I could build my own personal search engine that could search on just what I've ever looked at."

Then I smack it down, because that is a crap-load of effort to recall a link every year or two. And let's be honest, the marginal value of that link isn't all that great either... in the moment the need may seem large, but sitting here typing about this I couldn't tell you even a single such thing I've forgotten about, because that's how important they are... just more ephemera in the stream themselves.

My MP3 collection is a bit of a mess. I've cleaned up the worst instances of "Band, The" "The Band" "Band" "Band - The" sorts of duplication, but that's about it. My book collection is similarly messy. Heck, even my family photos are basically sorted only by year and not much else. So what? I can fix it. I can fix it all. But it's hard to even so much as recover the time I'd put into it once over, let alone in multiples.

(Much more important, especially for the family photos, is not losing them. So I've got a backup solution. But it's just a fire-at-directory solution, not all gloriously organized by type either.)

So I've learned to just sort of let the desire to have greater organization pass over me, Litany-against-Fear style. It's just a siren call.

Photos are the worst.

I'm through at least three complete reorganizations where I even dusted off old backups and collected all photos (because I felt I was deleting photos too liberally last time), de-duplicated (and de-quadruplicated) them all, and built the new "forever" structure.

It sucks that I just know I'll do it again at most five years from now.

Photos are a good example of why a filesystem hierarchy is insufficiently expressive. You might want to search for pictures of me at parties, or pictures with me and my wife, or pictures from 1999, or pictures of LA, and the same photo might belong in all of those searches. No single category will ever be a good place for a photo.

I personally agree with your point (and find the loose textual search offered by phones these days to be mostly adequate).

But reading your comment gave me a thought: filesystem hierarchies are indeed insufficient, but what about filesystem hierarchies with liberal use of hardlinks?

Or tag based, such as https://www.tagspaces.org/

That seems equivalent to a graph to me, and yes, I'm unaware of any kind of search that a graph does not permit. Indeed it could be the basis for a system that, in my opinion, would dominate any of the existing knowledge graph / tool for thought products. It would consist of three more pieces:

  * A database for backlinks. (Links from file X to file Y would only be possible when X has an appropriate file format -- `.txt`, `.md`, `.org`, etc.)

  * A search grammar with the following primitives:

    * find children of (links from) query results
    * find parents of (links into) query results
    * take the disjunction (OR) of queries
    * take the conjunction (AND) of queries
    * group queries with parentheses

  * The ability to pipe files found via ordinary shell commands into that grammar.

Given the size of most peoples' knowledge graphs, you wouldn't even need to keep a text index (ala Lucene) -- `find` and `grep` would be more than sufficient.

I’ve been looking for a good tagging system for image and video files, something I can use to quickly and easily go through a stack of files and tag them, then search by tag later. Bonus points for being able to recompress on the way through, since phones seem to have terrible compression ratios compared to offline compressors.

That's a good example because it leads to truth that NO up-front tagging will ever anticipate all the searches you might make in future. There are so many possible searches.

I figure that a combination of wetware and software is the current sweet spot. My brain usually has enough associations and context to turn every photo search into a time or place filter - "I think it was downtown last year" or "some time in summer at home" or "it had my wife in it". The photo storage system need only provide search/filter on date and place to narrow it down to a few hundred thumbnails, plus machine-learning to tag people. Which is basically what iOS provides, no more, no less.

Any other up-front categorization or tagging is basically wasted effort.

For me, I wrote a bulk tool that renames my photo file names by reverse geocoding the GPS information via Open Street Maps. That way I can do text search for place, as well as 2d map search. It's at https://unto.me

This is what I want machine learning and face recognition for.

Problems local to my machine, not Orwellian nightmares.

That does seem like a really good solution. Google Photos will prompt you, if I recall correctly, to identify a few faces, and then automatically id the rest. That's fantastic, if you don't have to worry about putting their privacy in Google's hands.

Adobe Lightroom has it. Unfortunately it is quite expensive for the casual photographer.

For me photos are solved problem now. I no longer do any cleanup on them, just assume that Apple AI will show me best photos when I search for them. I think that it is simply good enough already.

this calls for an AI that recognizes pictures and categorizes them for us. what have we been solving all these captchas for?

photos and pictures organization should be a solved problem.

Does your smartphone not already do that for you? iPhone does, I think Android does and I think iPhoto on macOS does as well. It wouldn’t surprise me if Google online photos or Facebook do also.

(That is, let you search using words for things in the photo or themes like “winter”).

Google Photos search function leverages some automatic categorization, it's pretty good. I wish I had some way to run something at that level over my Lightroom 5 catalog.

I put all my photographs in directories named year/year-month/year-month-day. For instance:

    ~/pictures/2022/202212/20221225

And I tag them with as many tags as I can be bothered with using XnView. XnView lets me find pictures by name or by tag.

Digital photos are easy to organize... I store everything in d:\masterarchive\yyyy\yyyymmdd\ folders, and have since 1997

Tagging them with embedded IPTC tags is the way to go. DigiKam works (mostly) as a substitute for the late Picassa. (I'd use that, but the last version has a nasty bug in that it sometimes swaps faces in the recognition database, which then tends to corrupt it all).

The major problem I've had is that in the beginning I didn't really have enough free disk space to keep up. That is no longer an issue, nor is it likely to be again.

> My MP3 collection is a bit of a mess. I've cleaned up the worst instances of "Band, The" "The Band" "Band" "Band - The"

MusicBrainz Picard cleans and labels your music automatically using sound signatures, even when the file has no metadata. You can just give it your files and let it run, rarely have I felt the need to monitor it. It gets stuff right 99% of the time, the rest 1% is easily fixable whenever you come across it.

> "Oh, if only I ran a system that archived everything I browsed so I could build my own personal search engine that could search on just what I've ever looked at."

Well, that makes two of us.

Hey, I’ve been building that. It’s called A Personal Search Engine: https://apse.io

Oh! That is neat. Based on screenshots? Wouldn’t have thought of that.

I have many questions, about backup and disk space. I’m going to give it a try.

Thanks!

Bookmarks are a good middle ground between saving everything and not being able to find something you read later. It is integrated in the browser's address bar as autocompletion/search and if you can vaguely recall some words from the title you can find it. I've been using this system for years, works great with Firefox sync and Firefox on Android too (to remember articles I discover on both mobile and desktop in same place)

For sure. I get the appeal of Having Everything Organized. It's conceptually compelling. But I think the right way to look at it is in terms of minimizing total cost of retrieval and filing.

As an example, take physical paper. Receipts, bills, anything that ends up in one's mailbox and doesn't immediately get recycled. I used to oscillate between two approaches: over-elaborate filing systems and just ignoring the problem and letting the mail pile up in snowdrifts.

Eventually I realized that my love of elaborate systems was a giant fucking problem for my actual life. I thought about it like I was designing a production system. I very rarely needed to retrieve old documents; most of it was for "just in case" conditions. I needed to frequently file things, and if the cost of filing was too high, I wouldn't pay it. So I bought 8 filing boxes, each 3 or 4 inches high and big enough to comfortably hold legal-size paper. Each one is marked with a year, and almost everything for that year just gets tossed on top. A few exceptional kinds of paper then have their own separate file folders (e.g., tax documents, my current landlord, key retirement paperwork, key medical stuff). Once a year I throw out the contents of the oldest box and relabel it.

This works great. It turns out I almost never need anything from an old box. When I do, it's a quick rummage in one spot. With infrequent, hard-to-predict retrieval, storage-optimized organization is the best organization.

I have been handling physical organization by prioritizing filing. My reasoning is that if the house is a mess, filing isn't keeping up with retrieval, and I need to reduce friction on filing, because dealing with the mess is consuming more time than inefficient retrieval.

For paper, I don't have much trouble. Things go on the fridge if I will need them soon, or in one small sterilite plastic file box. It's nowhere near half full and I would not be surprised if it lasts 10+ years before I need any more storage for paper.

As an experiment I've been working on sorting things by category in a more general way, like the dewey decimal system rather than true categories, to remove the overhead of half full containers used to sort things. They're based on observation of what was already stored vaguely together rather than starting with an idea.

One common category is BAM, bulk artificial material. This includes paper towels, laundry soap, paint, water repellant spray, etc.

Another is TAM, tapes attachments and materials, containing tape, steel wire, foam, webbing, carabiners, key split rings, screws, and all similar things often having to do with either attaching things together or long things sold by the foot.

With wider categories I have fewer places to memorize, and organization within a category isn't that critical because they can be rummage-searched, without the overhead of a buch of individual drawers or boxes in some ever evolving system. It's just a formalization of random boxes of junk.

Yes, same spirit for sure! Rummage-searching is very effective as long as the total rummage space is low, like a single box.

The main place where I'll depart from that is with higher access frequency. E.g., I have a box that is sort of "LRU tech stuff". In there my various cables are sorted by type into gallon ziplocs. Finding the right kind of USB cable is something I do too often to want to pull it out of a 50-wire snarl.

Oh I didn't think of that. LRU makes a lot of sense, especially since it's just the default big mess with an expiry.

My first line of defense with tech is volume reduction on the stuff itself. Bluetooth over cables, software over hardware, USB-C over everything else, resisting any kind of random tinkering gadgets in favor of phone apps and zero-friction stuff I'll set up and never bother to upgrade, etc.

If you never buy random stuff just because it looks cool, then when you really do want something you can afford to future proof it and sometimes get one thing that replaces multiple separate things.

I have one box dedicated just to power(Which I may split up into separate cables and batteries boxes) that has all my USBs not in active use.

USB-C has been fantastic. If you don't do a lot of fancy stuff with high data rates, it's an amazing way to collapse down the number of unique objects.

> Each one is marked with a year, and almost everything for that year just gets tossed on top.

This is what I do. Things organize themselves organically this way - the important stuff bubbles to the top, and the rest is already sorted by linear time.

As a heavy user of TfTs (and one who finds immense value in them as a personalized reference guide more than anything else), I've come to a similar conclusion. If search is good, not much else matters (the bad search in Roam was one of many things that led me to leave it). I'm not exploring my own notes for fun, I'm digging through them for something actionable and usually have a vague idea of what I'm looking for. Linking, tags, and other organizational tactics (even just a directory scheme) are only there to assist that.

Anything more than that is a recipe for burnout - and I've definitely been there.

Because there's a certain element of dopamine release associated with this act of digital tidiness. And for a lot of people it's a good way to provide the illusion of feeling productive while stalling on the act of creation... which is hard.

Reply on news.ycombinator.com