Learning from 20 years of personal analytics | Hacker News Reader

Learning from 20 years of personal analytics

blog.stephenwolfram.com

352 pointshendler59 comments

As someone who's trying to optimize his life better, what strikes me the most is this part of the post:

  For my consistent experience has been that the more 
  routine I can make the basic practical aspects of my life, 
  the more I am able to be energetic—and 
  spontaneous—about intellectual and other things.

This reminds of the book Uncertainty that I'm reading. Very interesting indeed.

"Be regular and orderly in your life like a bourgeois, so that you may be violent and original in your work."

Gustave Flaubert, Letter to Gertrude Tennant (December 25, 1876)

This is a wonderful apologia for the needy/homely lifestyle... conserving energy for something important.

And the flipside of course: pity on the extravagant, wild, dress up and party lifestyle: a way to comfort yourself if you don't have anything deeply rewarding to do.

Funnily enough I think this applies to business as well. I was at the YC User Acquisition conference the other day and what was striking was just how many processes the most successful companies have locked down. It frees you up to be creative.

It's part of an increasing awareness and importance of habits in a business context. The surge in popularity for game mechanics was largely directed at creating habits for consumers, creating an entertaining and engaging use case where they didn't have to think too much. Thinking is hard.

A business, at its core, is made up of a people. Well defined processes are the equivalent of habits for a business - common practices where the expected behavior is straightforward enough that the company doesn't have to think very much. This, as you say, frees you up to be an active thinker on many other topics.

There's a great book by brazilian explorer Amyr Klink[1] where he tells the story of how he became the first person to cross the Atlantic by rowing a small boat.

He starts out rowing whenever he feels like it, sometimes rowing to the excess of 16 hours in a single day.

So Amyr adopts a regular "work" schedule, rowing at most 8 hours a day, but allowing 2 extra hours whenever he feels energetic and motivated.

He then becomes so productive that he's able to cut 40 days from his initial schedule (it took him 100 days as opposed to 140 days to cross from Africa to South America).

Edit: link to the book

[1] http://www.amazon.com/100-Days-Between-Sea-Sky/dp/0747549478

The scope and scale of this data is breathtaking! However... it strikes me that the best that he could do with this data was plot it and say "oh, I remember those events". I wouldn't feel like all of that effort was worth it, if I were him. What did it DO for him? Apparently very little.

Some people just like data, the way others like walking on the beach or playing football or reading a book. Some things just don't "do" a whole lot for you except entertain or inform you.

It's pleasant to look back over your life - the graph probably reminds him of great stories that are too private to share :)

"I wouldn't feel like all of that effort was worth it, if I were him"

It's a hacker instinct. Not everything has to be justified, and immensely valuable to society. Sometimes doing it cause you can is part of the satisfaction.

This is amazing. I think the quantified self movement is going to be huge. With more personal tracking gadgets (fitbit, jawbone, nike fuel), measurements are going to become more and more seamless, passive, and complete. Not only will you get historical insights into your blind habits, but you can finally have an objective feedback loop on your behavior, and make necessary adjustments.

But one of the biggest challenges is going to be privacy...

It is pretty amazing, but in a way, Wolfram is also demonstrating how not to do Quantified Self: the point is not stamp-collecting, but data-collecting + experimentation + analysis. Right now, Wolfram has maybe #1 and #3, but he doesn't have any of #2, and he doesn't seem to really appreciate this:

> And as I think about it all, I suppose my greatest regret is that I did not start collecting more data earlier.

Hopefully he can change that. For example, his observation about the 7% use of the backspace key is an obvious one. If he eliminated backspace entirely, he could speed up typing by as much as that much - nontrivial given how much time he apparently spends on email! An experiment would be using some typing tutor program for a few months and seeing whether the keylogging notes a decrease in backspace.

Good point. When I saw his email volume, I wondered how I would feel looking about at an entire decade and seeing how much time I spent on emails. I'd probably feel a lot of regret. But like you're saying the point of QS shouldn't be regret (or novelty), but identifying a 'problem' and fixing it asap.

I wondered how I would feel looking about at an entire decade and seeing how much time I spent on emails. I'd probably feel a lot of regret.

Why should you feel regret? There is nothing more human than communicating with other people!

A prediction engine (aka adaptive shorthand, or similar) would give even better ROI than manual practice in standard typing.

Why are predictive keyboards so popular on phones but not desktops and laptops?

Do they actually work for coder-style usage? Would I actually be faster in Emacs if I was constantly looking to see what my keyboard had been redefined to this time?

Intellisense/autocomplete in Eclipse and Vim is the one are of desktop computing where I do get assistance, and it works great. It's not autocorrect, but it's two characters and then tab of crtl-space or whatever and tab or arrow key to the word of my choice.

I've been doing it in bits for a few years, mostly because I'm a sucker for interesting-looking graphs of data, but imo it's a mixed blessing. I've definitely found interesting trends, but measuring is also something I'm actively conscious of, and, as is often the case, it's hard to measure things without the very fact of measurement changing them.

I find myself struggling to avoid falling into the trap of doing or avoiding-to-do things specifically so that they'll show up in the analytics in a certain way that looks nice, which feels like a weird "teaching to the test" but applied to my actual life. Some things I've since stopped tracking because I really disliked the meta-ness of it all.

Privacy will be one challenge, but I think seamlessness (as you mentioned) is currently the bigger challenge.

I'm a runner, so diet is important. For 9 months I tracked everything I ate. Weighed portions, counted calories, logged everything. The data was insightful. I learned that I need to eat 3,300 calories a day to maintain my weight, with a range of +/- 300 calories.

After 9 months I stopped tracking things because it became time-consuming. Partly was because I learned what it "felt" like to be full, but mostly because tracking it all was really annoying. If all of that could be automated somehow, I'd love to have a spreadsheet of data which I could graph and chart. "At 8:16am, you drank 7.8oz of Orange Juice, totalling 108 calories." Cool, my O.J. container and glass measured that automatically, and uploaded it to my database!

But right now doing it is extremely manual. All of Wolfram's amazing data comes from some sort of automated system he setup once and never worried about. A pedometer. A script chewing through 20 years of archived emails. That form of automation doesn't exist for many types of data people would like to collect, such as food intake, or blood sugar testing for diabetics.

Inventing these non-invasive automated sensors is key for society-wide personal data collecting. Otherwise most data gathering will be sentenced to the dedicated few, rather than the busy many.

My grandparents are diabetic, and I've long imagined an implant that could measure blood sugar levels and report it back when they sit down at the computer or pick up a device. NFC is used in subdermal implants today, it could be done. My grandfather has a pacemaker, perhaps it could be a combined function. The problem would be getting blood samples without actually causing damage to vessels, but I am no medical researcher so I don't know all the hurdles.

I agree with you that measuring is the most intensive task of tracking personal statistics. I can enter how many servings I had of what in any fitness tracker website, but what if I had 1.3 servings? What if I had .2 servings (just a taste)? How do you estimate that without measuring it first? And if you're measuring everything on a scale, where has the fun gone in life? What's the difference between medium intensity running vs high or low? If I biked at 15mph it might be medium, but what if I kept that 15mph steady up steep hills? Or what if I kept 15mph down a hill without pedaling? We're built for rough estimates and are amazingly adept at convincing ourselves we're right.

I have enough difficulty with time tracking at work either breaking my productivity or cutting into my limited "me" time, and that's just a petty little thing. I usually end up estimating at the end of the week then fixing discrepancies that always come up.

woah, "NFC is used in subdermal implants", is that already being done? I've googled and seen it as something on the agenda, but have no idea if people are already doing it?

Well, if you define RFID as a near-field communication. This could be a matter of debate, I guess. Obviously it's well-known for having your pets "microchipped", but there's also this:

http://en.wikipedia.org/wiki/Microchip_implant_(human)

These kids developed something similar: http://www.diabetesmine.com/2011/04/kids-get-innovative-with...

> That form of automation doesn't exist for many types of data people would like to collect, such as food intake, or blood sugar testing for diabetics.

When I was looking into it, I noticed that a lot of the blood sugar testers were electronic and could record the series of readings for you. You still had to prick yourself, but it looked pretty easy.

(Sadly, the one non-invasive blood sugar recording device I found was apparently discontinued for causing electrical burns.)

It probably wouldn't be too tough to make an arm-band device which periodically pricks you, possibly beeping a few seconds before it happens. I'm not sure when diabetics need to calculate glucose levels, if it's either after each meal, or periodically throughout the day.

At 8:16am, you drank 7.8oz of Orange Juice, totalling 108 calories.

It just occurred to me that the new breed of payment services (Square, Google Wallet etc) should combine with food vendors to offer this service.

In Australia at least fast food restaurants publish the calorie count of each food option. It wouldn't be impossible for the payment company to do the correlation.

Also sports (and weigh loss) food vendors should embed NFC chips in their packaging so you can scan them before you consume them. There are plenty of cross-promotion possibilities there...

Vendors are already tracking all my itemized purchases and sharing with every advertising company in existence, it would be nice if they would share the data with me, too.

"Google Takeout" indeed.

I would say that this 9 month period was important because it probably trained you to intuitively know how much to eat. I went through a similar process 5 years ago: I had to monitor my vitamin K intake so I grabbed the USDA nutritional database, wrote up a simple little web app for my own use (made public at cookingspace.com) and after using my little app for about 6 months, I had trained myself to intuitively know how much vitamin K I was consuming every day.

Motivation is also another big challenge. Very few people care about their long-term health, and thus very few will care about tracking their health stats.

If you're curious about the setup he uses there's more here. http://stephen.wolfram.usesthis.com/

I am looking forward to moving into a house so I can have that noisy hot supercomputer in the basement on Gigabit Ethernet / wifi to the living room.

If this sort of thing interests you, check out Kevin Kelly's Quantified Self collaborative project.

http://quantifiedself.com/

wow, so he was and is writing a minimum of 50 up to 200 emails per day ? Insane, that would take up my whole day, but since hes mostly managing his company his job probably is mostly about writing stuff to people, but still amazing to keep that up for so long.

Wow! Just wow! So, he's reading about 500 emails per day, and writing over 100 emails per day, and spending about 10 hours per day on the phone, and has an average of 10 meetings per day... those numbers just don't add up. The meetings are obviously held over the phone, and he's apparently writing and reading emails while being in those meetings, but I still don't understand it. Are all his emails one-liners that don't require any thought or research? Is he some sort of genius multi-tasker? How can it possibly work?!

He says a lot of the emails were automated--system status emails during the launch of alpha and so on.

Those are the inbound emails, not the outbound.

He perhaps uses email (in part) the way you use instant messaging, testing, or twitter.

And a 10 minute chat may count as a meeting.

For anyone that wants to track their own typing, check out the whatpulse project: http://whatpulse.org/

My typing (http://whatpulse.org/stats/users/210575/) seems to match his in frequency, around 10 million per year.

My first thought on that is "people will just install a keylogger?"

Then I read their privacy policy and it makes specific note of "we don't log keys, just keystrokes". It must be the truth, they repeated that entire paragraph twice in their privacy policy.

Haha, well they get that a lot. You can monitor network traffic if you like, they don't track your individual in-order keys.

My side project http://AskMeEvery.com helps with personal data tracking. It asks you a question of your choice (ie how many phone calls did I have, how many commits, anything) every day and graphs your responses over time. Might be useful if you're interested in this.

Why would you ever ask a human how many commits someone made? # of phone calls I can almost understand, but we should be beyond that as well.

When I was reading James W. Pennebaker's "The Secret Life of Pronouns," I started consciously collecting as much personal data as I could for this exact reason. I'm 23 and I've got most of everything I've ever done on a computer since I was around 14 logged. I started thinking about the amazing insights that all of this data can reveal to me in the future. Every IM conversation, email, blog entry, text message, tweet everything I've liked on Facebook. I can only imagine in 20 years, having a psychiatrist ask me what my childhood was like and being able to show them a piechart of how many times I complained about something to someone.

I would love for all of the little tools and scripts he uses to keep track of this data to be released publicly. Half out of curiosity (how exactly does he do it?) and half out of interest in doing this myself. Although his post doesn't seem to make any important conclusions from the data, I'm sure that there are some really interesting correlations, patterns, etc.

> one can type and use a mouse just fine while walking on a treadmill, at least up to—for me—a speed of about 2.5 mph.

Anything you can do while typing and talking on the phone isn't much exercise. This seems far more annoying and inefficient than simply taking a 10 minute 5-6mph jog on the treadmill around the block before lunch.

If this kind of thing interests you, I made an app that graphs last.fm song listens in the same way as Wolfram's graphed his emails.

I find changes in listening habits correlate well with big life changes.

http://markhansen.co.nz/scatter.fm

I see that you're making the API requests on the client side (a good idea because of last.fm's API restrictions), but you should really consider caching them server side.

It takes quite a while to fetch all plays if you have a lot, and currently, if you want to share the graph with your friends, they would have to wait too.

Yeah, I hear ya, it takes ages to load my data too. If there's a way to cache on the serverside without running into the API throttling, I'd love to do some caching.

Another idea: log which windows are open and whitch one is active:

http://darcs.nomeata.de/arbtt/doc/users_guide/

If you are interested in seeing this sort of data for yourself - it is exactly this experience we are building at BrainPage.

Leave your email on http://signup.brainpage.com - we'd love your feedback as it gets ready.

This kind of data would be so cool for body related things

- Calory intake per day

- caffeine

- minutes exercised

- blood pressure

I could imagine in the future this would be quite feasible. The biggest barrier probably isn't technological, rather the resistance to the idea of injecting/carrying a little digital monitor

Many people, myself included, are keeping exercise data with sports-tracker / Endomondo type apps. It quite nice to have this history.

And I'd love an app that collated calls, email, meeting data from varous points over the longer term. I guess for most people this will be Facebook.

All of this data is already going into our computers. I bet people would pay for a slick app/service that visualizes it well, like RescueTime but more holistic.

I wonder how he reliably recorded the phone call data?

The same way I do it - itemized phone bills :P

Good point didn't occur to me!

How do you scan 230,000 pages of paper?

Auto-feed, like how a printer consumes paper.

i can measure so i can mess with it. What did the person learned more then if he would have asked collegues and his fam members. Humans are imho the more important data filters and aggregators. I prob.missed the point.

Reply on news.ycombinator.com