Hacker News Reader

Comment by MarkusWandel | Hacker News Reader

Maintain access 20 year old fitness data? What for?

I'm fairly far along on the data hoarding spectrum. This extends, for example, to over 200K photos from my life, categorized and indexed. It's trivial to look at "what happened on this day 20 years ago" or at least on the nearest date to that day that had pictures taken.

And here's the thing: As the memories fade, so does interest in the photos. 10 years ago you still remember clearly. 20 years ago is another era.

And that's photos. Fitness data? Do I really want to see how I declined over the last 20 years? I see that already from the average speed display on my bike computer. Fitness data is the ultimate "looking forward" thing. But maybe that's my age; it may look different from a younger perspective.

Agree with others that just use a platform that lets you export open-ish files. .gpx doesn't do heart rate, but .fit does, and gpsbabel can translate that to another format as needed.

View on HN

I think this is a resolution argument. Data should be curated carefully and reduced in resolution as it gets older. If you’re never going to look at it, it’s probably not worth keeping.

As per photos you don’t need 11 pictures of those muffins you baked in 2003. One will probably suffice, if you even care about them 18 years later that is.

The same goes for health data. But overall trends over 20 years are really useful as you get older. Expected decline is inevitable but unexpected decline may be indicated earlier and result in some preventative action to improve your quality of life. There are a lot of long term metrics appearing in Apple health like walking stability on that front.

On photos, I’m not sure what I’d do with 200,000 photos. I have 7,623 which span 4 generations and 125 years at the moment. I spend a lot of time curating these photos, adding metadata, editing and pruning garbage. They are all managed with Apple’s native Photos app which is good enough for the job (non destructive, sqlite DB underneath it, easy to back up)

Out of interest, what is the argument for reducing resolution other than space concerns? Fitness data is not really space-intensive. I recently ran a marathon and the associated GPX file is less than a megabyte; the FIT file is about 180KB. I don't really think even 30 years of regular running data would come to much. I appreciate cycling data will take more space but not that much more.

My point is, it seems like once you have put in place the habits and technological solutions necessary to store fitness data, it actually takes more effort to reduce the resolution of older data. I'd rather have data and not want it than want it and not have it, but then I am an almost pathological hoarder so maybe it's just me.

It’s mostly the ability to be able to locate it afterwards i.e. to improve the signal to noise ratio. I track hikes I do with a GPS and keep the GPX files afterwards but generally don’t refer to them unless uploading curated copies to OS Maps here in the UK for other people to use. So after a few years I nuke them. As for health data, it’s quite large. My apple health data is around 100 Meg compressed now and a lot larger, in the order of 1Gb, decompressed.

I am not a hoarder as a counterpoint. I am always looking at ways to reduce what I have and remain focused. I lived with a hoarder in a house full of trash and don’t want to be that person to someone else. He also had 4TB of photos and videos that needed to be dealt with.

I think at the end of the day my life can probably be compressed to a few images and a few paragraphs…

>On photos, I’m not sure what I’d do with 200,000 photos. I have 7,623 which span 4 generations and 125 years at the moment.

Some of them might be historically interesting enough to preserve on Wikimedia Commons, for example if they depict places that look very different today, or stuff that you don't see in everyday life anymore.

That’s quite possible. I have around 150 historical photos i.e. over 100 years old which depict family life in Switzerland in the 1800s. I will look into that. Thanks for the suggestion.

Keeping lots of photos is easier than whittling them down to some percentage subset.

It is until someone leaves you 4TB of them to deal with. Where do you even start?

I am a big proponent of not leaving my kids a pile of trash to sift through. See my other comment on this matter for background.

But in 10 years or so, 4TB won't be a lot of data. It's also not really now. My NAS has about 85 now.

They could just have it on a portable drive and look through it whenever they want. It's not like they have to catalogue every picture :)

4TB is a huge amount of data. And it’s costly to keep data spinning. 4TB storage an in apple device is going to set you back a lot of money for example. Not everyone has a NAS or the ability to run one and even more importantly to back it up effectively. It becomes a large monkey on your back and a cost and not an advantage.

Looking at my device storage, I have 91Gb online for 5 people. That’s everything we have ever done. Because we care enough to look after it and curate it.

> As the memories fade, so does interest in the photos. 10 years ago you still remember clearly. 20 years ago is another era.

That's what makes them so interesting.

The photos from 15 years ago with my parents still alive and my kids as toddlers are more interesting than last year's holiday.

Fair enough. My older is just under 10 years old, so that kind of sets the "current era" event horizon. 20 years ago is relatives who aren't alive any more, parties full of people I can't even put a name to any more, trips of which the memories have faded, stuff like that.

(since photo management seems to be of interest)

I have approx. 220k photos, of which 210k are in camera raw, 10k are ‘processed’, which typically means exposure corrected, cropped, etc. They are stored in two parallel directories (raw and processed), each of which is structured into years which are further subdivided into ‘shoot’ folders – one for each distinct upload from the camera. I use Adobe Bridge to do the uploads and to change the raw files names to support my indexing approach. The year folder names are simple (e.g. ‘2021’) and the shoot folders have names like ‘2021-010 garden birds young’ (where 010 is the sequence # of the shoot in the year). The folders thus automatically list themselves in chronological order when sorted alphabetically. A processed image might have a name like ‘20210523_010_my-IRL-name-here_0019.psd’ (so that I can immediately find the raw or processed shoot folder given a file name). I insert my name so that if I a send an image to someone I am identified as a creator (image metadata contains additional contact / copyright info). I’ll often extend the automatically generated file name to include the photo subject as I process specific images.

My indexing system relies on my memory and the fact that the relatively small number of processed pictures are indicative of the contents of the wider raw set – so I can browse the relatively small number of processed shots (using Bridge) if I am looking for something thematically. I usually manage to cull obvious fails from the raw set when I do an upload before doing any processing. Storage costs being what they are I have not yet had to further cull raw or processed images to make room on my hard-drives – currently 1.7Tb of images.

I have left it too late to add meaningful tags to this collection, and because I am the only librarian, I don’t have a need to help other users find particular images.

Any suggestions for managing an advanced photo library locally with a concurrent Google Photos library to share with family?

I'd recommend using one of several mobile phone apps to sync your phone's camera/DCIM directory with a computer/NAS at home, so you have your originals.

https://photostructure.com/faq/how-do-i-safely-store-files/#...

I wouldn't recommend syncing from, say, your Google Photos library: even in "original quality," your files may be mucked with (it depends on your platform).

Once you've got them local, there are several apps to share photos via local webserver. I wrote PhotoStructure to deduplicate and manage my 500k+ photos and videos from the past 20+ years.

> Maintain access 20 year old fitness data? What for?

Why not? I have about 12 years worth because that's when I started tracking, no reason to ever delete it. Saving the 0.001% (actual value, I just checked) on disk space isn't a motivation.

It's fun to review old data. Occasionally useful, even. Mostly for fun.

> And here's the thing: As the memories fade, so does interest in the photos. 10 years ago you still remember clearly. 20 years ago is another era.

Wait another 30 years and those now-50 year old photos will be treasure.

I don't look much at my photos from 20 years ago, that's pretty recent so I remember it well. But photos from 50 years ago are quite the treasure. Even older ones (like when grandparents were toddlers) are invaluable. Wish there were tons more of those.

But yes that's photos. Fitness data is certainly less interesting, but still fun and costs approximately zero space to keep.

Hmm. While I don't really need my heart rate or speed for every activity I've done the last years, some of it is actually interesting. Like looking at accumulated time or distance of various activities over the years. I also keep a list of all big events I've been to and my stats from those, which I find cool. But of course, that doesn't have to be exportable.

Just a few days ago I could find out the answer to a Q about when in the year I normally do my first cross country ski session, by looking at the graphs per year. So on average I will be going skiing next weekend 8)

OT but...

> 200K photos from my life, categorized and indexed.

What app do you use to manage your photo library?

> What app do you use to manage your photo library?

It doesn't seem like a good idea to tie a photo library to any app. Applications come and go, the photo library should last a lifetime (or more if you pass it on). You don't ever want to get caught with your photo data in some proprietary structure when the app that created it is no longer available for whatever reason. My photo library certainly has outlived many applications I've used to process the photos.

After processing I store them in just a directory hierarchy. Top level dirs for each year, event/topic subdirectories under each year. That's it.

That's fine for file storage, but doesn't provide a way to find all photos with dog in them, or photos from 2007 - 2012 with bird and dog, but not cat, in them.

Most apps will let you store your photos how you want them to be stored on the filesystem.

(...he _should_ be using PhotoStructure, but I _may_ be biased...)

For medical diagnosis when you get older, having that data will actually be useful in the next 20-40 years to diagnosing things that will happen to you then.

Would love to know how you categorize and index 200k photos.

Homemade system that is not in any way publishable. It's a folder hierarchy but also viewable as a chronological timeline. The main key is to organize them as you go, and not fall too far behind.

> Homemade system that is not in any way publishable.

I can still access all the photos I took since around 2008, when I first started to nurture an interest in photography as a hobby. I never delete b-rolls unless they're absolute crap (i.e., I triggered with my lens cap on, too dark with no flash---blurry, out-of-focus shots do not count as absolute crap).

I also manage it with a homemade system! This system is closer to my darkest shame than proudest achievement. It kinda reminds me of Destin's pre-Linus-organized system[1] except way smaller and a few scripts more sophisticated (my system has redundancy, ha!). Despite that, I have maintained access despite moving across continents.

In fact, last year, slightly spurred on by the pandemic situation, a bunch of friends asked me if I still have copies of photographs I took years prior. The oldest query was only for 4-5 years ago but I am proud to say that I was able to oblige despite my shameful system.

Kids, don't do this in prod, but homemade systems FTW! :)

[1] https://www.youtube.com/watch?v=lcWSrIiR1tY

Reply on news.ycombinator.com