The idea maze of personal logging (2016) | Hacker News Reader

The idea maze of personal logging (2016)

pcmonk.me

136 pointspcmonk46 comments

This is a very interesting read, because I took the opposite approach.

I started out with the premise that manual logging is very hard to do well. It takes effort on my part (and thus hinges on my discipline), can be inconsistent, and can change depending on what my important metrics are.

I automatically collect as much information as possible. I log every time I change a window or tab and what I change it to (extending from native windows on macOS all the way through to virtual windows in gmail.com). I log the same from my iPhone. I throw all of my data in a Postgres data warehouse, and then use a BI tool to normalize it and put some KPIs in a dashboard.

This system allows me to ask anything from "what articles did I read last week?" to "what percentage of my time at work did I spend programming?" to "how much did I sleep this week?" to "who do I spend the most time messaging?".

It's also an exercise in understanding that metadata is incredibly revealing.

Sounds interesting! Sounds like you've hand-rolled this, have you published anything about your system? I'm sure the details would be fascinating.

I haven't yet! I should really get around to doing that.

Is it possible to collect this kind of information on iOS without rooting the device?

Nope. I have a background in jailbreaking and use a jailbroken phone :O

I've done a similar thing with https://github.com/0x09/watch

It's extremely simple but it allows me to do things like plot a color coded calendar of app usage, which can provide very revealing info about things like productivity spikes and lulls or sleep schedules.

I have a companion Safari extension for tracking active tabs which I did not release since it seemed like such a niche and strange thing to use, but the main code to do that is simply https://gist.github.com/0x09/d1220a7b0a0bda64b4f542067cd43ec...

I had also written a tool like yours: https://github.com/ifree/ActivityLogger

Depending on the ease of use of this thing, I will pay up to £25 one time if you'll let me host the data myself.

If you bountysource, I will pre-pledge before you have delivered.

I think you're sitting on something useful. I currently use RescueTime but I'd be more comfortable with one time fee and holding on to the data myself.

I only mention the money in case you're wondering how much it's worth to other people.

I treat Google Calendar as a time-series database.

I have 14 calendars logging approx 500 events per month, and I have a UI for all of this... Google Calendar.

I also have API endpoints for all of this... Google Calendar. Which can be triggered by IFTTT for automatic stuff (i.e. did I Instagram something or Tweet?).

My email is linked to Gmail, so flights, itineraries, online shopping delivery, restaurant bookings can all flow in.

So the only thing I need to do manually are the few things it cannot automatically know, such as what film I just watched or what book I finished reading.

I only log public things, that is... data that exists in someone else's database somewhere. I do not log private things, i.e. Signal conversations or anything 100% offline that didn't involve a public database.

For me it's a simple thing... this data exists, can I make store it myself in a structure that I can do something interesting with it.

I have about a decade of this data.

Oh, and Google Calendar is really long in the tooth and is becoming both slow to use, hard to search, and clunky.

> Oh, and Google Calendar is really long in the tooth and is becoming both slow to use, hard to search, and clunky.

In your personnal case (because of the many events) or in general ?

I feel like calendar has been forgotten by Google (no material design for instance) and could be on the list of next things they chop off.

In general.

I don't think they'll chop it because it's critical to the GSuite business side of things.

But it's probably not receiving love as whilst Ai has seen things like Photos revamped and the emergence of Google Assistance, the applicability of this to the business stuff is more difficult for compliance reasons and politics, etc. It's easier to focus on consumer when innovating.

The weirdest bit of all of the Google offerings, is the ever-diverging difference between the capabilities of the consumer Gmail accounts and the GSuite accounts. Google's tagline of "One Account, All of Google" is long dead in reality, and with it there seems to be a very different focus, innovation happens to the consumer and not to the business side, and never the twain shall meet.

Consumer accounts cannot use Google Calendar's "Find a time", and business accounts cannot fully use a list of products so long that it's boring listing them.

Google Calendar is central to how I run my life... I even logged phonecalls with links to the audio files uploaded to Google Drive, as well as scans of post received (and categorised in Google Drive but linked from Google Calendar).

But the usefulness is being reduced, silo'd, splintered from the general usefulness of Android and the Google related family of products due to the split of Google into consumer-focused vs business-focused with less overlap.

I've not bought a Pixel phone for this reason either. What's the point when so much of it's selling point is software and that software doesn't work with GSuite accounts.

> I even logged … scans of post received (and categorised in Google Drive but linked from Google Calendar)

Oh, very nice idea. What's your workflow for doing this?

Imagine a 1990s batch flow system :)

I have a Fujitsu ScanSnap S1500, I scan through that... it auto names my post based on timestamp and drops it into a folder.

I rename it how I want to file it within my "Files" folder in Google Drive but leaving the datestamp intact, i.e. "2016-05-03T07:13:01 Finance - HMRC - P60.pdf" and I move it into another folder.

A cron comes along, looks at everything in that folder, fires a little Go util that uploads it to Google Drive (splitting " - " into subfolders), and finishes off by adding it as a calendar ever.

This would be a lot easier on Windows or Mac as I think you have Google Drive clients that sync whereas on Linux this doesn't exist.

I thankfully don't receive a lot of post, but when I do it's only 30 seconds to scan something and a few seconds to rename and move into another folder.

I made my side project (https://socialite.ooo) to do personal logging, but I had a slightly different problem. I would remember small things about people but not their names. I was going to the dog park with my girlfriend a lot and couldn't believe how often we'd remember someone's dog's name, or the walk we met them on, but we wouldn't remember them.

My solution was to correlate people to events, and events to locations. This allows you to not only remember what you were doing, but pivot off of any piece of information to better recall an event. Logging into a web app to enter information is a bit of a slog. I think making a mobile app is really crucial. That could even lead to some automation of the logging. For example, "You just called [CONTACT], would you like to create an event for this? ", or at the end of the day, "You texted [CONTACT] today, give a quick summary of what you spoke about". From there you can make the analysis smarter and smarter.

It's really strange, all these companies have so much data about us. We don't have access to our own data in a decent format though. My call history is with my telco, my location history is with Google (and probably many others), my conversations are spread out among so many different apps. What happens when I want to query the data about me though?

> Socialite is a paid application. We made this choice in order to give you peace of mind. With no advertisers involved, you won't have to worry about who is getting their hands on your information. Pricing will be targeted at $10/year. Don't worry though, we won't charge you until you're making use of the application.

> Socialite is a paid application. We made this choice in order to give you peace of mind.

That sounds very funny. I would get much more peace of mind if you said that your client app is open source and uses strong encryption before saving data to your server. If you pay somebody, they must be trustworthy? That logic is laughable.

I think the argument there is that their incentives line up with yours. For someone else to get your data, they'd have to get hacked. Most companies just give that data to the highest bidder. Of course, that business model means that they should be able to encrypt it all client side, and I think that would be wise.

It it's free, you're the product, not the customer.

If it's proprietary software, it uses you.

Can you expand on that?

If it's not free, it doesn't mean you are not a product.

The problem of encrypting data and then making it searchable is tough. My career is in security so I did think about how to implement this for a while. It is its own product though, and security conscious users (myself included) are the minority. I decided to make an application and not sell user data or use it to sell ads. That is what that statement is saying.

Sounds interesting, I'll check it out.

> My call history is with my telco, my location history is with Google (and probably many others), my conversations are spread out among so many different apps. What happens when I want to query the data about me though?

That's a real problem right now. I asked that question in almost the same terms many times when I worked for Urbit [0]. I no longer work for them, but they still seem to have the best long term solution to giving you access to all your own data (specifically by putting that data on your own server to begin with).

[0] http://urbit.org

Personal storage servers would be awesome if they got popular. I'll check urbit out. They would need a really flexible api for apps to hook into. If that happened though, I'd be stoked to sign up for an application and instead of choosing an authentication provider you chose a backend.

I really like your approach of parsing the natural language to create log events. Going the other way, from data to descriptions could be really cool too. Finishing grad school is taking up my free time for the next few months, but maybe we could come together on these ideas later on.

I remember using AutoHotKey [1] (windows only) to randomly prompt me what I was up to every hour.

This was many years ago so I don't recall the details but my intention was to understand where/how I spent all those hours in the day as a student.

While there was no formal syntax that I followed that would have helped to analyze the logs programmatically, it did help me get a general understanding of my schedule and identify things I should change.

Ref : [1] https://autohotkey.com/ [2] http://lifehacker.com/5582372/use-a-daily-log-to-keep-yourse...

Beeminder's TagTime does this, but according as a poisson process averaging once every N minutes.

In college I had to keep a life log for about a week as part of a psychology class. It was a completely manual process (i.e. pencil and paper; this was quite a while ago), and I was supposed to log in as much detail as possible what I was doing throughout the day. At the end we were supposed to create categories from the log, and add up all the time spent in each activity.

The purpose was to find out how much time was really spent on each activity, rather than what we thought we did. It was an eye opening experience.

I did something similar back in university after reading about Sen. Bob Graham [0]. For a period of 6 months, I would write down every single action with place and people involved in a small notebook. At the end of the day and the week I would review it. It was really eye opening and allowed me to change some bad habits.

[0] I can't remember which article it was, but this one describes his logging activities: http://quantifiedself.com/2009/05/politician-as-self-tracker...

I've been using RescueTime (http://www.rescuetime.com) for around three years now, which pulls data from all of my workstations as well as my phone, but it lacks the ability to handle soft skill logging like social encounters and dietary habits with the finesse that pcmonk is working towards. Right now, I use it to track things like how much time I spend in IDEs, in StackOverflow, Hacker News, etc.

For some previously unexamined reason, I am giddy about tools that enable me to surveil my own productivity and procrastination metrics, but feel slightly uneasy about trying to capture daily intangibles.

I'm not sure if I would want to know how much time I've spent with each of my friends; the analytical side of me excises and optimizes, and I worry how that might manifest in negative ways if I start to quantify friendships and phone calls.

Anyone identify unexpected truffles and/or landmines in the exploration of your personal logs?

DayOne could help you, it can add a little popup on your device asking "what did you do today?" where you could enter little or lots of details of your encounters which are not loggable.

My very first (and maybe not so useful comment): I really love your approach. I was thinking something more automatic (I have some withings stuff) but you convinced me to start small and manual. Really interesting. Thank you for sharing.

How about screen video capture? Rewind at 60x at the end of the day and take notes in whatever structured form you want.

It would also help to have an app that reacts to inactivity, that is you stepping away, and ask you what you did.

This works well if your day revolves around a desktop or maybe Android, but not so well with iPhone or iPad.

Sounds like a fascinating idea, but I suspect a video will be unmanageably large. However, a relatively low resolution time-lapse would probably be sufficient.

I tried, it works fine. Using screenflow on mac.

You could sample say one screenshot every minute. It wouldn't take up too much space.

Those files will add up very quickly. On Linux I had screenshots for every click for months and the results were numerous enough to bring 'ls' to its knees

I think the idea was to analyze and cull at the end of every day

Well the goal was to make a video. The screenshots could be compressed into a single video file. At one screenshot a minute, and 24 frames per second; 24 hours of screenshots would give you a one minute video.

At work I use an app called Timesnapper. It takes a screengrab every second. Great for time sheet filling and ad hoc documentation of tasks.

Really well thought out. Would also map well to an Alexa skill, like:

Alexa, tell plog I ate a burrito at 8am.

Being able to log and analyze your data is great. It not only provides insight, but also provides motivation ("I'm logging this to improve. Let's not screw it up." etc).

I think my three big tools that have helped in this space are: my fitbit (tracking exercising), habitica for recording daily tasks and todos, and private journaling at the end of each day.

I've been terrible about routines and such, but all of this has really helped get things back on track.

Overall, I think also logging has to feed back into both organization and action, such as new goal setting. Which adds new things to log.

This is great. The structure and simplicity is bomb and the data will be easy to mess around with while still being easy to clean up even if you mess around with verb usage like "scarfed" instead of "ate".

There's some evidence that this kind of behavioral monitoring can be useful in a number of areas, especially weight loss, and other gradual lifestyle shifts. This seems like a particularly smart way to implement that.

Most of the human-subject research done in public health, sociology, and similar domains uses data collected in a similar way. They basically ask people to take notes on specific behaviour or fill short surveys multiple times a day (they call it ecological momentary assessment), then they analyze it in the context of their research.

So yes, a tool like this has value in helping with better understanding ourselves, but also can help population research as well.

GUI is not a problem, you can always tweak it or delegate it for little money. The big problem is how to force yourself to do it regularly in long term, at leat in my case.

I think that good idea is to integrate it with bot on your IM platform of choice. Messenger bot, in my case, could ask randomly what I'm doing and track it.

I've used Timing.app on my Mac for a loooong time. Love it.

http://timingapp.com/

ibGib's current web app used today by my girlfriend and myself for personal logging while our shower is getting remodeled: https://www.ibgib.com/ibgib/Saturday%20II%5E818E0366E6B009CA... _(NB: After a temporary session "login" at the bottom of the page, you will be redirected...also, the links are long because the last part of it is a sha-256 hash that guarantees the integrity of the content of the ibGib data)._

I post this because I think personal logging is a perfect fit for ibGib's engine (and it's partly why I've written it), and it may interest you as a back end. It would allow you to basically have a graph database for a back end, while also allowing you to create and evolve your semi-structured data much like evolving an Event-Sourcing event in version control like Git (since each and every datum is versioned and maintains a complete audit history). It also is designed to be Big Data friendly, with the ibgib website's datastore acting with completely Open Data.

Also, I wanted to mention that I've actually used it recently in the exact use case given by @TACIXAT where I was walking my dogs in my neighborhood and I wanted to be sure I remembered the neighbors' names down the street when I met them a month or two ago (https://www.ibgib.com/ibgib/Walk%5EE3EE1755557357849C41C1511... - just doule-click the big yellow circle). I definitely like how the socialite app looks, and it's great in that it helps you remember things in context. This is also what is primary about ibGib, allowing for pictures, text, links, etc., to be nested "inside" other "objects" (so each node acts as both a file with content and a folder with relationships to other nodes).

So for your personal logging, you could create a semi-structured "type" with your subject-verb-DO-etc. and then create instances of these. Then you can relate them via any named relation to any other construct (called an ibGib). So it's very much like a graph datastore that also has data hashing for all "children" nodes, which makes it a merkle graph. There's all sorts of neat properties, and like I said, I thought your use case would be a good fit for the engine. (Btw, the structure also lends itself to auto-generating dynamic and collaborative blogging from your data, e.g. https://www.ibgib.com/ibgib/WaffleGib%20Jr%5E59B6D3828D35209... or https://www.ibgib.com/ibgib/Grillin%20burgers%5E9E0151A452C1...)

Reply on news.ycombinator.com