Ten Years of Logging My Life
chaidarun.com
chaidarun.com
As fascinating as the data could be (I never managed to transform it like this), the constant recording of every single thing made me stop enjoying things. It made me hyper-conscious of everything going on in my body as the primary goal was to figure out some health issues. Everything I felt or did seemed to have an "event handler" attached to it.
I now just stick to one line a day in a spreadsheet as a sort of diary.
To anyone attempting it:
1. Figure out what you want to do with the data first. Even if you just want some pretty graphs, create some random test data and set everything up so you can see these graphs. Don't just throw data into a black hole for years.
2. Make sure recording the data is as easy and comfortable as possible. That android app from the article looks good, but you have to be absolutely sure it's okay to use many times a day. Be aware that there's no good answer to "why do you keep reaching for your phone all the time?".
Apps like RescueTime (time spent online) and Withings (sleep and steps) are simply running in the background of your life, with no additional input needed from the user.
I'm always impressed when I see people who maintain these massive data-entry systems, I know personally I would never be able to maintain them which is why I opt for lazy tracking. The less data entry I need to do, the better.
Oura ring however has an open API, I don't own one or use it but worth checking out - https://cloud.ouraring.com/docs/
We currently support desktop agents (Windows, MacOS), Calendars (Google Calendar, Outlook), health monitors (Apple Health, Google Fit), as well as Chrome, Slack and a few others.
We're big supporters of MyDataGlobal and the OpenHumans movement, and have a few features for export in the product already. Happy to chat if you'd be interested in a way to feed out just the API data we're already managing so you can use it elsewhere, DM me or mail support@mana.ai and I'll see it!
"Avoid black holes" is a good tip when you struggle with forming the habit of logging. If you have a graph that auto-updates as you add data, it's fun and the habit will stick better.
Saying all that to say I truly admire people like you!
Chains are unstable[0]. Usually you miss a habit because something has gone wrong. This time, when something has gone wrong and you most need support, is when you lose all the support of the chain. In fact, you also have to deal with the shame of breaking your streak. "I always come short" is the inevitable outcome! There's no goal! They're good for taking 95% compliance to 99%+ (days without an accident counters, the best of the best at their activity), but not good for taking 20% to 80%, which is what most people need for most habits.
I had some good success (but not perfect) with what I call Fibonacci Streaks. You go for 1, 1, 2, 3, 5, 8, 13, etc. in a row. When you hit a goal, the counter starts over and you go for the next goal. If you miss, you start over at the first 1 and get some goals to hit again quickly. You could also have a variant where you try again for the goal you missed. In this system, the counter starting over is a normal part of things. Why Fibonacci? I considered doubling, but the jumps are a bit large. 1.5X seemed right, but gives fractions. Fibonacci is ~1.6X each step. I made a free sort of add-on app for tracking it in Todoist[1].
[0]https://en.wikipedia.org/wiki/Longitudinal_static_stability [1]https://habitsfortodoist.com/
I find this stuff really fascinating. About a year ago, Garmin added a new feature to “pause” the Training Status measurements the watch takes. Apparently when folks where getting sick they became incredibly disheartened by their watch telling them their lack of activity is not conducive to improving their fitness. It’s almost reinforcing the bad habit instead of trying to frame it in a more productive way.
When this happens, I set the whole thing aside for a year, every time, even though I shouldn't.
I can definitely relate to the frustration! It's been years since I've worked on the product side myself, but for what it's worth we do allow equipping up to 2 streak freezes now. I believe you can also restore a lost streak as an in-app purchase; I've never done so myself.
The difficulty that people run into with maintaining daily rituals is that they have no way of reliably enforcing the ritual during a period of low motivation / energy. My approach is to find a low-commitment version of the ritual to be performed on days where you don't feel you can really give it your all. How I approach this with diary writing is simple: on nights where I feel I cannot possibly spend much time writing, I write an entry of the form, "It is incredibly late and I do not want to write more before I go to sleep." Nothing more. By doing this, I am able to maintain accountability without torturing myself.
I'm really not persuaded by your story because you don't actually seem to be engaging in "never break the chain" behavior. Like, can you tell me exactly how many days in a row you've journaled? Like, if you had a counter going in Duolingo or put a red X on a calendar every day in a row that you wrote a joke like Jerry Seinfeld? You seem to be engaged in the system of "do the thing almost every single day, however small, forgive yourself when you miss it" which is a wildly more applicable and effective system, in my opinion.
I think the last time I forgot to journal was... maybe a year and a quarter ago? I fell asleep before I intended to, and when I woke up, realized I had missed a night. I have never ever decided not to journal on a particular night, and this fact is enormously motivating. That is why the chain system is useful.
One interesting question is, how do I journal when it is extremely impractical? The answer is, I've done whatever was necessary to maintain the chain. Lacking a journal, I've used a pen and scrap paper. Lacking any writing utensils, I've used my phone. Lacking my phone, I've borrowed someone else's phone. The important part is that I made the sincere choice the perform the action, in however nonstandard a way, and this reminds me of my commitment. I've essentially inducted myself into my own religion.
I wish I could form some actually positive habits this way!
It's intentionally non-machine-readable. Just to see the past weeks at a glance.
I can't praise this app enough. Main pros:
- all your data is stored locally not in the cloud, but you can easily back it up on Google Drive
- it lets you export your data anytime
- It comes with nice dashboards, weekly, monthly and yearly reviews. It let's you see the impact of each activity on your mood. It really gave me some interesting insights on my mood patterns.
My only complaint really is that the only numeric value you can track is your mood, everything else is yes or no.
A year or so ago, I've also started using Clockify [1] to track the time I spent on each activity. Again, I don't track everything, only a few habits I want to improve on. It has a mobile and desktop app. The UI could be improved but all the features I need are free, and most importantly data export.
It's the perfect time of year to set up a tracking system. My recommendation is start small. Track only a couple of activities first and add more as you build the habit.
[0]: https://daylio.net/
[1]: https://clockify.me/
Anyways: First experienced memory on that list was when I realized winter was reoccurring thing in spring of 1955. I was sitting on a sleigh on water-covered skating field and imagined I was a boat. I then suddenly remembered that this has happened before, but very long time go, maybe half a life-time ago.
A video camera 1984 solved the remembering problem totally. I know exactly what happened thereafter.
Anyways various archives seem to be more and more important with age, as human physical memory is only 7 years long. Thereafter there are only memories of memories, which often get twisted in the weirdest of ways.
"Dream of an apartment separated from opera restaurant by a glass wall only" is the first good one..
I hate to be that guy, but... you have no idea what you're talking about.
Remembering something is like pulling a note out of a cabinet. It gets saved from the back, where the papers tend do degrade. Then, it gets put at the front. The trick is, the original note isn't put at the front. A copy is made. So remembering tends to reinforce, but also can change during this lossy copying process.
None of them "churn".
"The memory-of-a-memory thing"
That "isn't a thing".
"I don't remember the exact science"
You also have no idea what you're talking about.
"I do remember the metaphor I came up with (evidence in my favor!)"
I have no words.
"Remembering something is like pulling a note out of a cabinet."
Not really, no.
"lossy"
Please just stop.
"copying"
Memory is a "copying process"?
Please guys... please just stop.
> None of them "churn".
By churn I mean "lose and hopefully be replaced". That definitely happens. Some brain cells die and are replaced, some live your whole life (or die first)[0].
The cabinet is just a simile for cued recall[1]. More recently learned/recalled items are remembered better (front of the cabinet)[2]. Recalling memories in a different context can change them[3], suggesting recall isn't a "read only" operation.
[0] https://www.livescience.com/33179-does-human-body-replace-ce... [1] https://en.wikipedia.org/wiki/Recall_(memory)#Cued_recall [2] https://memory.psych.upenn.edu/files/pubs/KahaMill10.pdf [3] https://news.northwestern.edu/stories/2012/09/your-memory-is...
Earlier in the year I discovered Joplin and I’ve stuck with it, enjoying its balance of structure and unstructure. If it had the instant-data-entry-into-CSV feature of Tap Log it would be perfect, but nevertheless I decided having one journal / personal data repository was better than having several, and set about importing all my entries from my previous system. I now have many, many thousands of entries spanning over nearly 25 years, and I’m happy to report that Joplin is coping well.
> What you see is some of the daily data I’ve collected during the last 7 years of my life (2,498 days between 01/03/2014 and 01/01/2021) charted as basically as possible in order to allow global readings and comparisons. (A nice method is just to scroll up and down with the mouse cursor pointing to the time of interest.) I update it every January. I started to log some topics later on, and there is a period where I did not log mood for some technical reasons in mid-2015. I have a lot more data on medical issues, food, people, and daily activities, which I chose not to show here.
https://news.ycombinator.com/item?id=28896170
It sends an email a day and I just respond to it, and the response is stored in an Sqlite db.
It's not a log, exactly, but could be used as one as I can respond multiple times for the same day.
But I use it to store not just what I did each day, but also my thoughts, or my reactions to things I read, or little things I learn, etc.
It took a little getting used to at first, but in the last 30 months I have never missed a day.
Although it's not the main use, it's fun to be able to tell friends, with near certainty, when exactly we did what.
I've unsuccessfully attempted various implementations of life logging for years and I wonder, how much of those failures were due not to poor system design, but rather a poor relationship with myself throughout time? I have some theories to test starting in a few days.
Some questions I have for OP and anyone else with similar habits: Do you journal regularly alongside / in addition to your activity logging? How are you recording emotional variables like stress? Any plans to expand your data capture to encompass more signal?
For example, I could record stress on a 1-10 scale. Do I just put 11 for a day of unprecedented stress? Or should 10 always represent the global maximum thus far? More generally, how do I ensure consistency with myself?
I do vividly remember the stressful times in my life and can see their effects in the blog post's charts. I figured it's probably best to preserve at least some of my own privacy, although I did touch on this a bit when discussing music.
All that said, I have friends who journal and I might end up trying it too out of fear of losing data (even if it's squishy). Maybe once a week would be a sustainable cadence?
I'm also a musician, and ironically, I will often increase my total stress levels by procrastinating certain tasks, using an instrument as my weapon of distraction. Sometimes this has instigated negative feedback loops, causing me to avoid playing for an extended period.
So my plan is to allocate a few minutes in the beginning and at the end of my day where I will reflect on the previous and upcoming cycle, including tasks and goals. Here I will have a -1, 0, 1 flag where -1 is "I am stressed," 0 is "I am relaxed," and 1 is "I am excited." I can either set these flags for an entire period (hour, day, week) or I can set them per object (task, goal). The default state is 0, as I'm more likely to set a flag if I'm experiencing emotion in either direction. Importantly, I will also set these flags retrospectively; so in the morning, I might flag a task as -1, but in the evening, it might be +1.
There are some useful data points here that I should be able to achieve with minimal friction. 1) how often am I stressed or excited? 2) how often can I easily identify the source(s) of emotion? 3) how consistent are my emotional flags over periods of time?
I also wonder what quantifiable data must exist in any form of extended journaling that can be unlocked via NLP or simpler heuristics. When I'm excited, do I tend to use more or less punctuation? When I'm stressed, does it take me longer to complete a journal entry?
Thanks for the inspiration to jot these thoughts down. They've been lingering in my head for awhile.
That's also a great point about NLP etc. I was kinda hoping that my "Average daily log entries" would show some kind of trend like that, but I guess it was too simplistic. It really might be worth recording journal data and just assuming that the tools for analysis will catch up later.
I do journal as well, using vimwiki (Where I implemented some simple tagging for things like happiness/productivity/stress/etc. ratings from 1-10).
However one of the side projects I'd like to work on at some point is a mood recognition project using a camera that I can put beside my door. The idea being to take a "photo" that then uses an ML algo to determine my mood (from 1-10) and record it whenever I enter/leave my home.
I ended up moving towards the mental side, to see if I could measure aspects of flourishing and wellbeing, and more recently to see if any core attitudes about the world are changing over time.
If you’re interested, check it out for comparison:
It seems designed for people who want to bill their time, but works just as well for people who want to track 24/7. It takes some patience for a week to get into it, but now it's second nature.
While reviewing the data at the end of the week is useful, I've found the greatest benefit is requiring myself to choose how to spend my time. Getting distracted (e.g. browsing Hacker News!) requires the mental step of stating "Now I'm going to distract myself".
For my system I have repurposed Org Mode's clocking feature and run a dedicated Emacs instance in a terminal window. I then have a bunch of R scripts for visualizations: https://github.com/he-sk/timelog
The main disadvantage is that I need to enter activities manually when I have been away from the computer. I just save notes in my smartphone but it's a bit annoying.
https://www.gibney.org/a_syntax_for_self-tracking
To make this fast and easy, I developed the syntax described in this essay. It turned out to work pretty nicely. I have logged over 14,000 entries over the last 2 years.
As I am not aware of any existing software to analyze causalities in such an irregular log, I started writing my own this year.
It is an interesting challenge. The approach I currently take is a visual one: I draw a chart on which every notch on the x-axis is a 24-hour interval. And two lines on it. One is the line where I did not do something (say take Vitamin D3) and the other line is where I did do it.
The y-axis is the probability of an effect (say having a headache) in that time interval.
The center of the graph is the 24-hour interval after the treatment. Then the 24-48 hour interval, then the 48-72 hour interval etc. And left to the center is the 24 hour interval before the treatment. Then the 24-48 hour interval etc.
So it would be a "perfect" graph if the two lines look very much the same before the treatment and very much different after the treatment.
Some treatment/effect pairs look promising, but so far, I don't have a result clear enough that I feel confident enough to publish it.
> I can make everyday plans with greater precision, knowing exactly how many minutes I’ll need to shower or drive or buy groceries or do laundry or water the plants.
I used to think like this, I tracked how fast different routes took to work to optimize and decide. I optimized where I put what during laundry to make it less bothersome, and so on. But do you need 10 years of this? For me, after doing it a couple of times I basically got it.
A shower, I don't need to track it to know I can do it between 2-15 minutes. Or less, if I REALLY need to get going.
I've also built my apps, and used pre-built ones. But each time I've stopped, I haven't really missed it.
I just put a big (so I can read it without my glasses) battery operated (no electric shock risk) cheap (cuz I'm cheap) clock visible from the shower.
A rough analogy: you can figure out from your HP bar how much damage you take from each hit, and it's probably worth doing so if you're going to be fighting these same baddies forever and have no other way of knowing, but the HP bar also serves other important purposes related to resource management: it gives you a sense of how you're doing overall, and it constantly reminds you that HP management is even worth caring about.
Sorry for the very personal question. You certainly don’t owe me an answer. I just find it fascinating how some people care a lot about tracking their lives and others (like me) do not. I’m curious how it relates to personality types.
I basically see myself as a garden variety introvert. I suppose that in an informal sense, I might be further along the spectrum than the median person off the street, but then again I suspect the same is true of many other software engineers - especially HN readers.
So generally no, but technically maybe? IANAD
1. How did you defined it - horizontal/unconscious/etc 2. Whether there are any 'outcome variables' it seems to cause 3. Whether your needs changed by age, or exercise level, or previous sleep duration (ie 'sleep debt')
Thanks for doing this! I always find them fascinating
2. It's not surprising that lack of sleep can get you feeling down. But since starting to track myself, I've been more conscious of the fact that whenever I'm feeling down, it's almost always after a night or two of bad sleep! I now take sleep a lot more seriously and ask myself first whether I've been sleeping well whenever I'm feeling down (as opposed to blaming other people, etc). This is especially important since I tend to ignore what my body tries to tell me.
3. The 7.5 hour average has remained roughly constant month over month for the entire decade. It's hard to say how much of an effect that other variables like exercise have since I always aim for 7-8 hours regardless. I do notice that if I get like 4 hours one night, I'll end up sleeping more than usual for the next night or two.
I've just uploaded a new APK that should hopefully work!
I developed an app installed on my phone which automatically detects SMSs transaction notifications and uploads them to my online app which is a simple General Ledger with simple and good enough reporting capabilities.
For cash payments (less than 10% of my transactions) the same app allows me to quickly take a picture of whatever I was buying or the receipt and uploads it to the online app for later posting. I even have a Google Cloud Run job which reads email notifications since some of my cards don't have SMSs notifications. I have been doing this since 2015 and it has been one of the greatest decision I have ever made.
If you're wondering where it could appear in the charts, any one of the Sleep/People/Exercise/Other categories would seem a possible candidate...
Easier to avoid it at least in the public copy. Though I'd argue even tracking it personally might cause issues if your new SO saw it and started to compare to past partners. Best to just leave it out entirely.
I don't think there's a taboo regarding sex and relationships in tech. People can discuss whatever they want. OP is free to choose whether to log (or share) their sex data.
- it's embarrassing to talk about your sex life, and maybe especially gaps in it, online? - it's naturally private and people value their privacy, particularly online? - people are so taboo about it that they don't even want to record it for their future self? - it just gets lumped in with "sleep"? - it doesn't take up a significant enough amount of time?
All of these depend on the person, of course.
People who value this type of privacy don't publish a decade of logs chronicling their daily life.
Is there anyone else with a better method?
I have been tracking the metrics of my life to varying degrees for close to four decades now. Every book I've read, every movie I've watched, every music album I've listened too, every science paper I've read through (rather than just glanced at), the bills each month, breakdown of grocery bills by product for the past 20+ years, every recipe I've ever cooked (though not every meal I've ever ate) for the past 10+ years, continuous screenshots of my desktop from my laptops and workstations, and recently (past 8+ years), multiple angle RGBD (colour+depth) photos of my home office and my home workshop. For a few years (between 2001 and 2008) I wore a homemade SenseCam that captured hundreds of photos per day as I went about my life.
I am always interested in other people's personal life logging habits so this is a fascinating insight in to it.
Where do you keep all this?
Did you consider any existing apps before making your own? Are there any apps like this that are available for download?
1. I have sole ownership of the data
2. The data is stored in an easily parsable format (TSV)
3. Duplicating a recent activity takes one tap
4. Pie and area charts are displayed in the app
My app is open source and available on GitHub[1] for any Android developer to compile for themselves, but if there's enough interest I can also get around to packaging it up for the Play Store/F-Droid/etc.
Like when I look at my watch it says 'work' and that should help me to allow myself to focus on work because I know my assistant will also 'schedule' news reading and other things.
Otherwise I'm constantly thinking and rethink ing what I want to do and end watching some yt because I'm to unmotivated at the evening to do something else.
Supporting this by visual cues around the flat with smart lights.
Of course just another, probably not working, tool for antiprocrastination.
With the stamina of the author (doing sport, playing instrument and keeping book for so long) I might not need a tool just his stamina :D
Same for wanting to do 5 hours of exercise a week, around work hours. What to focus on within working hours, etc.
With Alexa and Google Calendar, we're pretty close already.
But surprised how good the smartwatch vibration works for "time to move".
But just an hour ago I thought how I could tell Plex to stop playing and showing a screenshot to say 'its time, your show will continue in 30 minutes.' or the Amazon fire tv.
Might be doable with writing a Kodi plugin.
I've been looking to automate my tracking somehow for a bit, my best bet could be using the Google Fit API which already logs most of the stuff I want based on other devices I have (a smartwatch, for example).
My ideal setup would be something that can be self-hosted, exportable, and resilient, but haven't found the right recipe. I'm not good at logging things manually.
Some people might feel that they are not in control of their life even after a month of logging. Some that they should optimize every minute of their life for "productivity".
I suspect doing some introspection with help from a psychologist could be much more effective for both types.
[1]: https://mobile.twitter.com/giansegato/status/146896340342311...
The author writing their own Android app to accomplish the logging was interesting to read.
Anyway I didn't get really any insight from it so... idk if it's a waste of time or what. I still use writing a lot for personal thoughts/state management.
I was using Mailgun for the emailing/API and Raspi/Python/Apache/MariaDB for my stack.
I am aware using third party my stuff is passing through them, it's not super detailed just positive/negative sentiment with abstract context.
- https://www.coursera.org/learn/the-science-of-well-being
- https://hubermanlab.com/the-science-of-gratitude-and-how-to-...
Yeah I'm not sure, I think it's natural to have ups and downs. But for me personally there's always something worrying me. I don't know if once I escape being poor "for good" eg. have money invested/don't have to work. If I will lose this constant anxiety.
I still struggle with the whole "altruism" thing like I help people but I also in the back of my mind have this feeling of "mine". "I'd be further in life if I was an asshole" etc...
I think productivity is also subjective... if you setup businesses/have passive income/don't have to work. If you don't do anything are you not productive? Anyway I know what productivity means for me (getting things done) but that's because I can't stop yet. Not that I want to stop but yeah. It's like I don't want to wake up/sleep 9-5 but I need to because I have a job.
Happiness seems simple to achieve to me, do the opposite of what makes you unhappy unless you need something like medication.
idk, this whole 'studying happiness' sounds like you're trying to trick yourself to be happy, but that's an ignorant take without looking for myself.
I know can really get down in the weeds, hierarchy of needs, Buddhism, philosophy, etc...
I think that as a sentient/alive being, your mind will always be on/running. And actually having problems like being broke distracts you from the fact that time counts down and you exist regardless of what you do. Mainly to stay alive and avoid pain.
Even if you have money and didn't have to work/have freedom, how long till you're bored. Anyway I have plenty of reasons to be happy you know like I have family in a third world country with parasites, barely have access to water, here I am paid to write code, running water, hot shower everyday, etc... I used to stack biscuits into a box for a job and hated myself, I changed it.
Life is not so bad (but it can be better, Wonder woman meme).
Anyway I'm just rambling here
Yes, I have.
Emotions are their own realm, and the time you spend observing and reflecting on them, as we are doing in this thread, is valuable.
Not only does this give you a good idea of what you did each day, but it’s extremely beneficial for both budgeting and tax purposes.
The base plan for $10/mo does this. You can use Microsoft Todo to set the habit of consolidating every morning with your coffee.
I spent around $1200 on alcohol in the 2 year Melbourne lockdown, around $2000 on coffee, and a whopping $40 on entertainment venues (indoor sports, movies, tickets, etc). Aside this, I learnt that I spend less than I expected on eating out vs buying groceries, and all things considered it’s probably cheaper for me to rent a car permanently than own one if I want a nice car.
I used a non connected Android app before, but since I moved to a phone without Google Play support, I'm writing my own app.
Don’t think I have the dedication to track things similarly, but love the data. I use a Whoop fitness band to track sleep, vitals, exercise, and enjoy seeing all the data. Would be cool go log more than health, but that’d require some extra effort… :-)
Malcolm Gladwell often trumpeted "deliberate practice." Does the OP receive feedback regarding the quality of his musical practice or the form displayed during physical exercise? Does that yield greater gains per time?
More (most) importantly, what are the GOALS of these activities? The ultimate assessment would be the extent to which such data, discipline, and tracking actually spark progress in the desired direction.
I'd probably improve faster by taking music lessons, but improvement is really a secondary goal after having fun. For what it's worth, I do still get some feedback from bandmates and my own ear - recording your own playing and trying to mix it into something presentable makes it painfully clear what needs work!
It's also a great way to estimate how long projects take - pretty much thanks to past data. And you can use the data to tell how productive a company is - I had one whose project iteration times are a full 50x longer than what it took to do it yourself.
Just to share my experience, the time spent recording data often feels like it's worth it; there's a tangible reward that's more obvious after a while. At first it had novelty value but years later I recognize it as a distinct value separate from novelty. The act of recording by itself has a noticeable impact and anchors a feeling of self-control. (I use a sloppy tag-and-gather method though, not an app, which seems like it could be really fun to create)
I agree that logging is a great steering mechanism but I guess that you first must have a goal what you want to achieve.
I found the list years later. "20. Forgot to eat. Got sick."