A Syntax for Self-Tracking (2020) | Hacker News Reader

A Syntax for Self-Tracking (2020)

gibney.org

81 pointsgkbrk32 comments

Holy moly, my 3 year old post on the front page! The power of blogging. Cool.

Quick update: Yup, I am still doing this. I enhanced my tooling in VIM quite a bit, and logging is super fast now. Earlier today I entered log line number 30558.

I also wrote a bunch of software to visualize and analyze cause and effect in this type of log. Some of the many potential cause/effect relationships look promising.

Since the data in this type of free-form log does not fit into the typical statistical tests very well, I have no "significant" results yet. That's why I have not posted an update so far. I first want to improve my statistical abilities.

If anyone knows of established tools to analyze such a log, or has an interest in building new ones, get in touch!

for unstructured message analysis -- it's not a silver bullet, but you may want to try BERT or another newish embedding tool

(you can do this on your laptop, you don't need to upload the data anywhere)

then use like sklearn OPTICS to cluster (clustering method that doesn't require you to know the number of clusters in advance)

If there are obvious clusters, you may be able to label a few useful categories. If you get lucky and most entries have a cluster, you can potentially manually label the outliers.

I think I saw an example where you said you use it to answer questions like 'How many curls did I do with these weights last time' [1]. That was from a couple of years ago, and I'm assuming you are the commenter there. If that's correct, do you have a way to make this system portable? I feel like the place I'm most likely to need to answer such questions would typically only have my phone available.

I greatly appreciate the data you're collecting and the overall method of doing so. However, I'm curious to learn about the portability of this information, and how you access it when you're away from a computer with vim.

[1]: https://lobste.rs/s/ri5utx/syntax_for_self_tracking

> If that's correct, do you have a way to make this system portable? I feel like the place I'm most likely to need to answer such questions would typically only have my phone available.

If your phone runs Android, you can just install Termux and then do everything exactly the same. Stick the files in a git repo, edit with vim. If the ergonomics bother you, I think it's possible to have termux open files in an android native text editor app, but I unironically read and edit my files in (neo?)vim and it works fine.

Previously submitted here: https://news.ycombinator.com/item?id=29064409 (2 comments)

Interesting. I wonder whether they have learned anything from their observations. I would probably let numbers be part of the tracked event, because it might be important for things like. "Took 500mg of painkillers" or something.

I think that would be formatted as "Painkillers: 500mg" in the described syntax.

That syntax would be valid, but I usually let a dice decide, so the value is Yes or No.

And since I usually take 400mg ibuprofen, I have introduced a string for this in particular: Take_Ibu400. So in my log it looks like this:

2023-03-23 12:44 Take_Ibu400? Yes

If I would not have a specific string, I would track it like this:

2023-03-23 12:44 Take_Ibuprofen? Yes (400mg)

I’ve been doing a similar thing for over 1,000 days using an app called Daylio. It’s been both useful and rewarding to have a large archive of daily activity to look back on. I try to review old entries on an interval, usually at the start of the year. https://rickcarlino.com/2020/journal-apps-with-reminders-hel...

I loved the pragmatism, even if you don't log your life, I think this article is very valuable for everyone that need to be pragmatic

Some other posters have noted there are apps for broad personal data, but this is a whole other level that I think could be super valuable for those who want to build a scheme as they go. The blocker for me is that it’s not an app - if I’m going to do this kind of logging, it needs to be something I can put on my phone.

Apple Shortcuts or similar can handle this efficiently. Can even pump it straight into other apps like Apple Health to visualize a variety of metrics that you want to log. Multiple apps too, one tap can enter a metric into Health, log it into another app, and then into a text file.

Takes a bit of upfront work to personalize it, but it's mostly just tedious not actually strenuous.

What works for me is that when I'm on the go, I just write events into a text file on my phone.

2023-06-27 16:00 Drink_Coffee? Yes

And type them into my log when I'm at home. Sounds cumbersome, but it's in fact quite easy. I also log the time I spend on the self-tracking project and while it's one of my main projects, it only takes 10 to 15 minutes per day.

As always, I’d recommend ISO-8601 style timestls so which would only require one additional character (space swapped with T and a trailing Z) for the same semantic meaning, but could be extended to support time one offsets, fractional seconds, etc. without changing the parser.

> alias coindecide='if (( RANDOM % 2 == 0 )); then echo Yes; else echo No; fi'

I would probably (put intended) not use that. IANAS[tatistician] but I'm seeing suspicious runs of 1's or 0's which have a low probability:

  $ for x in $(seq 1 10); do echo $((RANDOM % 2)); done
  1
  1
  1
  1
  1
  1
  0
  1
  1
  0
  $ for x in $(seq 1 10); do echo $((RANDOM % 2)); done
  0
  1
  1
  0
  0
  0
  0
  0
  0
  1
  $ for x in $(seq 1 10); do echo $((RANDOM % 2)); done
  1
  0
  1
  1
  1
  1
  1
  1
  1
  1 
  $ echo $BASH_VERSION   # on Ubuntu 18.04, x86-64
  4.4.20(1)-release

If this is just an interactive thing you run manually, you may be better off tossing an actual coin.

If there are runs of zeros, this should result in a number greater than 25000:

    for i in {1..100000}; do echo $((RANDOM % 2))$((RANDOM % 2)); done | grep 00 | wc

For me, it always comes out very close to 25000.

Hmm. I also tried this with four bit codes:

  $ for x in $(seq 1 1600000); do \
    echo $((RANDOM % 2))$((RANDOM % 2))$((RANDOM % 2))$((RANDOM % 2)); \
  done | awk '
  { hist[$0]++ }
  END { for (code in hist) print code, hist[code] }'
  0100 99668
  0101 99832
  0110 100026
  0111 99837
  0000 99899
  0001 100451
  0010 99846
  0011 99980
  1000 100246
  1001 100182
  1010 100116
  1011 100383
  1100 100014
  1101 99470
  1110 99535
  1111 100515

Doesn't seem like anything deviates too much. If there were excessive runs of 1's and 0's, the 1111 and 0000 codes would be excessively frequent.

Nice. I do something like this except less complex. I use Blitzmail (there's an iPhone equivalent, but I don't know what it's calle) -- basically, it's an ultra simple app that pops a window to email only one person (usually, yourself) and have replicated a script on my desktop to do the same.

Then I have another script which just makes them ALL "todo" items in the journal of http://zim-wiki.org, and I just "check" (triage?) those every day.

okay, every few days, but i'm trying to better about it.

Either way it's replace a LOT of individual "apps" for me.

I started down the path of building a narrower version of this to track my personal activity. The syntax was a similar date log format (time optional) but it was unit aware and it used commas for labels. Example:

    2023-27-06 Walked 2 miles, cloudy day, felt good

That would parse to the following object:

{date: "2023-27-06", activity: "Walked", measurement: {value: 2, unit: "miles"}, labels: ["cloudy day", "felt good"]}

The idea was to eventually generate a static site with automatic graphs and tables but I eventually just switched back to a spreadsheet. It is hard to beat a spreadsheet for flexibility.

Makes me think of https://www.conventionalcommits.org/en/v1.0.0/ for your life. Any recommendations for tracking while away from your computer? I love ease of use but I think I would lose large parts of my day & weekends when not at my desk.

I would just use a cloud product like Google drive or Google notes for this. Then at the end of the day paste it into the correct location in the log.

I've thought about this type of tracking in the past, but I haven't been confident enough that it would actually find useful patterns to put in the work to implement it.

If you're using a phone, you could email yourself with a journal specific subject then use an API to automatically add those observations to a log. (Might be able to do that with Google Drive or notes too, I'm not sure if they have API hooks).

If you don't want to keep a phone, a small journal is also a possibility.

This is cool. Would love a system like home assistant w/ little blueprints that you could add. For example maybe you want to track your location and have that get defined automatically, i.e. anytime I visit the coffee shop, I’m working, so define that as work.

If you're on iOS, Life Cycle[1] can do this kind of thing for you. You can assign locations to categories (and it often does a good job of guessing them anyway) and it'll tally them up on a weekly/monthly/yearly basis.

[1] https://apps.apple.com/us/app/life-cycle-track-your-time/id1...

Since it is just a text file, it is easy to merge it with other data at any point in the future.

So you could log your position on a different device. And only when you want to analyze the data, you write a script which converts the positions to lines like:

2023-06-27 13:50 Work (At_coffee_shop)

And inject those into the self-tracking log.

https://jrnl.sh/en/stable/

I feel like this tool is similar in spirit to this idea.

This is a really cool idea. I would use a self tracking app, especially if I could quantify, search, or visualize the findings.

I have tried some similar things. I like your approach and write-up a lot!

You asked about tools. Well, from the "when you have a hammer" department.. you could use plain text accounting tools for some quick reporting wins. Some examples:

If you have a.dat:

  2020-05-28 18:41 Eat Pizza
  2020-05-29 09:00 Slept with the window open
  2020-05-29 09:00 Headaches

This is close enough to hledger's timedot format to do some reporting. Each line is interpreted as an empty transaction, which you could query by date or description:

  $ hledger -f timedot:a.dat print 
  2020-05-28 * 18:41 Eat Pizza
  
  2020-05-29 * 09:00 Slept with the window open
  
  2020-05-29 * 09:00 Headaches

  $ hledger -f timedot:a.dat print date:2020/5/28
  2020-05-28 * 18:41 Eat Pizza
  
  $ hledger -f timedot:a.dat print desc:eat
  2020-05-28 * 18:41 Eat Pizza

You could transform your data to a plain text accounting format, with quantities. Eg, make it TSV:

  $ perl -pe '$c=0; $c++ while $c < 2 && s/ /\t/' a.dat > c.tsv
  $ cat c.tsv
  2020-05-28 18:41 Eat Pizza
  2020-05-29 09:00 Slept with the window open
  2020-05-29 09:00 Headaches

and use hledger CSV conversion rules to customise and enrich it:

  $ cat c.tsv.rules
  fields date, time, description
  
  # save the time as a tag
  comment time:%time
  
  # count each item as one "event" by default
  account1 (events)
  amount1  1
  
  # special cases
  if pizza
   account1 (food)
   amount1  200 cal

Now you have a (single entry) accounting journal:

  $ hledger -f c.tsv print
  2020-05-28 Eat Pizza  ; time:18:41
      (food)         200 cal
  
  2020-05-29 Slept with the window open  ; time:09:00
      (events)               1
  
  2020-05-29 Headaches  ; time:09:00
      (events)               1

Allowing quantity reports:

  $ hledger -f c.tsv balance -MATS cur:cal
  
  Balance changes in 2020-05:
  
        ||     May    Total  Average 
  ======++===========================
   food || 200 cal  200 cal  200 cal 
  ------++---------------------------
        || 200 cal  200 cal  200 cal 

  $ hledger -f c.tsv activity -D desc:headache
  2020-05-28 
  2020-05-29 *

  $ hledger-bar -v -f c.tsv cur:cal
  2020-05        200 ++

  $ hledger-ui --all -f c.tsv   # explore with a TUI

More at https://github.com/simonmichael/hledger/tree/master/examples...

With "tools" I meant statistical tools, aka statistical tests.

Because the log is a time series where events happen at random times, the typical tests like the t-test are not readily applicable.

One way I am currently generating p-values is to look at questions like "When I did X / did not do X - how often did Y happen within the next 24 hours?". And put those numbers into statistical tests like the t-test.

But since "Y" often is not just something that happened but something that has a value (For example "High energy / normal energy / a little tired / very tired") this discards some data. This and many other specifics in a random-interval-event-log probably allow for better tests than I came up with so far.

Is the complete log a single file? I would be interested in seeing an excerpt of the data as an example

Yes, it is a single file. 30558 lines so far.

Some lines from today:

    2023-06-27 06:40 Wakeup
    2023-06-27 06:40 Last_night_sleep_time: 07h21
    2023-06-27 06:40 Last_night_sleep_interruptions: 1
    2023-06-27 06:40 Yesterdays_Steps: 11898
    2023-06-27 08:49 Temperature: 24.8
    2023-06-27 08:49 Humidity: 40%
    2023-06-27 09:21 Take_Iron? No
    2023-06-27 09:21 Take_VitaminD3? No

The first 4 lines have the same timestamp because I use a vim macro to ask me about these 4 values. The macro then logs those 4 values at the wakeup time.

I would pay for a mobile app built to rigorously implement this with encrypted backups (Joplin style).

A vim plugin would be a nice enhancement to the OP.

Reply on news.ycombinator.com