I put my whole life into a single database
krausefx.com
krausefx.com
In case you don't make it through to the project's webpage howisfelix.today, and read through to the end, here is the creator's conclusion:
Overall, having spent a significant amount of time building this project, scaling it up to the size it’s at now, as well as analysing the data, the main conclusion is that it is not worth building your own solution, and investing this much time. When I first started building this project 3 years ago, I expected to learn way more surprising and interesting facts. There were some, and it’s super interesting to look through those graphs, however retrospectively, it did not justify the hundreds of hours I invested in this project.
I’ll likely continue tracking my mood, as well as a few other key metrics, however will significantly reduce the amount of time I invest in it.
This is my general feeling towards all quantified-self projects. Perhaps it creates interesting insights for some people, but I can imagine the kinds of conclusions I'd actually get from it will be things like "I'm happier on sunny days" and I'll say "yeah..."
As a side note, people tend to be very unimaginative in the initial stages of data analysis: very often they just take the data they have instead of thinking what data they need to have. And obviously the results correspond to the initial limitations. So much effort is put into data munging and other stages when your inputs are incomplete or plain irrelevant to what your business actually needs.
If you think you should be feeling good today because you're well rested, you probably will feel better.
This year I added in daily mood tracking, and it's mostly focused on taking 5 minutes a day for myself to just jot notes down quickly. I'm using dailo and I have a few categories too (work good/ok/bad - food homecooked/takeout - vices) and just the space I've created to do that has been worth it. I don't expect to get anything too interesting long term besides patterns in takeout/cooking, but I find myself more connected to my day good or bad, and that presence of mind to combat the autopilot is worth it.
The signs I easily ignore are a little bit of brain fog, some reduction in physical energy, and dry skin. The dry skin is meh, and the other two are solved* with food that provides energy.
* = It does not actually resolve it, but it does “enough” and has a long enough lead time that it’s easy to have an instinct of “Oh I’m feeling run down, time to grab a snack” followed by “that didn’t quite get where I expected, I’ll give it time to digest” before forgetting about the whole thing as the project reabsorbs my attention.
There were 3 customizable preset capacities, but even using those required a couple taps each time. Drinking a non-standard amount and manually adjust it required extra effort and taps. Sometimes I’d forget to record it right away, then have to try to remember if I was double-counting a refill. The mental burden was clearly out of balance with the (practically nonexistent) reward.
I quit tracking it when I found myself adjusting my behavior, like avoiding the natural habit to top off my water bottle when it was less than half full and I was getting up from the desk anyway. Ultimately it would be interesting to have that data, but it’s only worth it to me for zero cost/effort.
From there, it's also valuable because it's clear that in the past I might have a dehydration headache, eat something, and still have a headache because I was thirsty. By just incorporating a routine and simple 1 click tracking, that consistency has helped me to easily manage this. But as you said, for you it works "naturally". For me, I was pretty out of tune with that.
Didn't do it for politics, just for lolz and prototyping a concept. Got to Twitch's advertising unlock, but stopped running it because it was a headache to maintain with my home setups/remotely.
But yeah, not trained on my comments, but someone else's, and it didn't take a ton of content to get it going for gpt-2.
Edit: Here's a clip for reference
However, if enough metrics were automatically collected (when did I fall asleep and heart rate lower?) and there was some system that could sort through them (eating a 4th meal increases odds of ruby team scoring additional points by a factor of 1.3x) then you could possibly learn some neat things.
Some of the most interesting things have been pointed out to me by others. Things that are related but I didn't notice. Foods that seem to increase my chances of sickness and things like that.
Conversely I can see when I have been likely to have been drinking from my heart rate.
I think the key here is the passive data collection with intuitive data presentation to drive low barrier to entry analytics.
He has to be in the top 1% of polluters, given his obsessive flying and consumption of technology.
Dozens if not hundreds of charts, depicting every single thing except the thing that is destroying our world.
I filed an issue on his github.
I'm tracking my life by a similar system than OP, but the error with tracking is to start tracking useless things. For example OP track his calorie intake, it is a lot of work, and it is not really usefull at the end. It is unlikely to correlate with any other measures he is doing (apart from weight).
I personally only track mood, activities and position. Each tracking has a specific goal, and only take me 1 minute per day. I have a lot of headaches and could be tempted to track headaches to understand what triggers them. But now that I know what triggers them (when I'm hungry, or I spent too much time exposed to the sun) I would never have been able to detect that correlation from data.
So my advice to people wanting to track their life, identify first what you need the data for, and increase the number of data you track if you have a lack of data to achieve a specific goal, but don't track for the sake of tracking.
For example I track my position, to know where I have been, but also to warn my parents in case I go missing and nobody know where is my last position
I track my mood, to know if I am doing better over time, or if I'm able to improve my sleep not only by feeling it, but by having numbers on
I track my activitites, because I want to know If I work too much, if I do enough sport, or if I spend too much time on my computer in a week
Just reflect on yourself at the end of the day.
Tracking calories is very very simple if you use an app like MyFitnessPal. For most foods you can scan the barcode or you can just enter the calorie data from the label.
Secondly, isn't weight a pretty important measure to track? It can impact a lot of things and therefore if you see you start to have too many days in a week that are above your calorie goal you can catch any gain before noticing it on the scales.
My take on this is Timelinize[0], which is capable of downloading all your data from various digital records like social media, your phone, arbitrary files and unstructured folders, your Google account, etc. So it can aggregate data from Google Photos, Location History, Twitter, Facebook, text messages, and more, onto your computer, organized into a single timeline.
(It is the successor to Timeliner[1].)
I suppose Timelinize can support personal body metrics from various devices, and it is planned to support manual entries as well like journals. Analyses like his would also be possible.
Data can be automatically imported on a regular basis via APIs (like, backing up your new tweets every day). The arbitrary file importer can also be scheduled to watch a directory for changes: just write a file or folder to that directory, and Timelinize will automatically import it (and optionally delete it when finished).
Timelines are stored locally on your own computer, as plain files on disk, with a sqlite database as an index, so it should be a breeze to back up and use/read without proprietary software.
The idea is to:
- Back up your precious data that is otherwise held more-or-less hostage on somebody else's computers
- While creating a detailed, personal journal for your family history
- In a way that can show you your life in context with others' timelines and the wider world.
[0]: https://twitter.com/timelinize (No actual website yet.)
Hoping it can suck in my missives from Facebook, Twitter, Reddit and Hackernews as a minimum!
Twitter yes, Facebook somewhat (for now -- the API and data graph are very complicated!), Reddit probably not initially (but on my lists; need to learn how to get the data), and Hacker News probably eventually (also need to learn how to get the data).
Why did you stop making updates to Timeliner?
(Also, I'm in the market for a better, available name.)
My own take on this problem is Chronomize [0], which I've been working on since 2020. It's interesting how we independently came up with similar names.
Feel free to reach out if you'd like to chat about personal data management.
I like your projects' name better too... would still like to see if I can find a better one for mine that is available.
[1] https://www.theguardian.com/environment/datablog/2009/sep/02...
It's a fine balance between caring about stuff and wanting to raise awareness while also seeing other human beings for who they are, assuming good intent.
I disagree (although 'I fly lots' would be a strange thing to ever be 'proud' of imo, regardless of the climate issues). While we should all be mindful of our individual impact, asking people to restrict themselves personally (or shaming them) on such an important and positive freedom as being able to travel widely just sits wrongly with me.
There are much lower hanging fruit as regards climate change to tackle first at a much higher level (e.g. government, multi-national business). I feel our focus should be on that. If you want to make personal sacrifices that's noble but shaming others for not making the same sacrifices is petty and makes it seem like your reasons for making the sacrifices are for appearance rather than because you care. NB: I mean this more generally, it's not wholly directed at your specific comment which wasn't super judgemental.
Maybe, besides political leaders and businesses, that should be limited to inner circle family and friends with whom you have a more complex relationship, and where there is a little bit more receptiveness towards critique. Even in that context it's probably better to just live the lifestyle yourself and hope they will follow.
On the other hand, if you put something out there, to be discussed publicly on HN, I don't feel obliged to only give positive feedback necessarily. Also, the author picked this up in a really nice way and has already added carbon stats to the page.
But man, it's so hard if you deeply care about something to keep your mouth shut. I don't know if you have any topics that make you feel like that?
Edit: actually responding to the rest of your comment, I'm a strong believer in the power of individual choices. The whole plant-based movement for example in The Netherlands (and outside) over the last few years was fully driven by individual consumer choices and little bits of money shifting in other directions. European politics actually tried to hinder it for a while, but at some point when something becomes the popular thing to think, politics will have to follow.
I agree. There are certainly climate-related things I try to hold my extended family to account over (e.g. basics like recycling). I guess where we disagree is where the line is and how much criticism we’re willing to accept and impart :)
Thanks for discussing it with me so pleasantly.
In other words: I am not saying that he should be ashamed of being a pollutant. But I am wondering why he isn’t ashamed of it.
Btw, he isn't traveling just for the sake of traveling — he has conferences around the world, and he's a European working in USA which means he probably flies back and forth from his home country a lot more than an average American. (Doesn't help that North America is far away from everywhere else).
For what it's worth, he makes an attempt to schedule all his conferences such that he wouldn't have to keep flying to/from North America in between conferences.
Felix Krause, keep doing whatever it is that you do and enjoy your life.
Then, they actually let you donate/spend a bit of money towards offsetting your individual carbon footprint, towards whatever initiatives you choose (personally donated towards planting trees).
It's not like it would have as much of a profound impact as stopping the mega-corps would, but it's still something and pretty easy to do.
Disclaimer: not affiliated with them in any way, just thought it's a cool idea.
A little = 140€ / month. Wow. What do people do, redecorate every year?
edit:
11 tons of CO₂ per year
That's 64% less than the Germany average for your household size.
I guess the two of us are pretty ecological ;) But I just realized they didn’t ask about heating (afaik oil based and I have only costs, no usage numbers), so that might make some difference.
Buy one new couch and there goes your budget for the year.
Once you have the basics of collect>summarise>publish metrics, it’s trivial to add just one more.
Do people really need every single aspect of their lives measurable to this degree?
Does anyone (other than your mother) really care?
I, for one, would hate for this to become a thing. It’s like a personal Facebook, but with infinitesimally more extra junk that I don’t want to know.
I feel like keeping a rough journal and manually revisiting would be more helpful than quantifying and correlating. But financial tracking can be useful, to a smaller degree, and tracking mood changes over big life changes can be valuable I guess. Otherwise, does feel vanity
Sometimes, it feels like there's so much going on that it's hard to know if it was the sleep, the diet, the events, or something else. That's where the data might be nice.
Although, I probably could do with some life changes, they need to be manageable.
> Overall, having spent a significant amount of time building this project, scaling it up to the size it’s at now, as well as analysing the data, the main conclusion is that it is not worth building your own solution, and investing this much time. When I first started building this project 3 years ago, I expected to learn way more surprising and interesting facts. There were some, and it’s super interesting to look through those graphs, however retrospectively, it did not justify the hundreds of hours I invested in this project.
But calling it a vanity project seems overly dismissive. Sometimes you start a project and don't know how it's going to end up. Sometimes it's the journey and not the destination that makes a project worthwhile. I think Felix deserves a lot of credit for writing up his conclusions and sharing them.
it was a vanity project - Felix just found out that it's not worth it later than sooner
Enlightenment
What's your problem with vanity projects? And since when is doing side projects for your CV wrong?
> Do people really need every single aspect of their lives measurable to this degree?
No, but it's fun? "Do people really need X" can be said about anything.
> I, for one, would hate for this to become a thing.
No-one is forcing you to look at it and even less participate in it.
Tracking mood is great. A simple telegram bot -> a google sheet lets telegram do the mobile/desktop UI and you just control the logic
One overlooked thing is automating financial tracking/categorization. In the US there's no good cross banking api except for email. Almost every credit card, bank, etc can email a receipt (set the minimum amount of alerts to be $0.01). The flowchart for parsing is bank alert -> gmail -> gmail filter by from address adds a label -> google pub sub notices that label and pings a little VPS somewhere -> gmail api -> parse email -> telegram bot
As I type, it sounds complicated but each step is well trod and has tutorials available. Now you have the ability to make custom push notifications for any transaction with < 1 second latency, instead of using the bank app. And it's all under your control. So the telegram bot allows you to categorize those purchases, you can add little notes or put it in categories. The nice thing is instead of looking at the end of the month and having 100+ little purchases and trying to remember if you ate at Starbucks 4 weeks ago, you categorize things as they happen in just a few clicks. And adding a category (one click) at the time implicitly marks the purchase as 'known good', not a suspicious purchase. So you can cross check at the end of the month for purchases you didn't categorize as suspicious.
It's a bit heavier on the automatic data aggregation side, but has some manual inputs as data sources too. I'm hoping to make it easier to interface with existing tools for data analysis, dashboards, notebooks etc.
i feel like there's such a gap, that it takes such dedicatiom & effort to have the digital world we interact with report back & give us the facts of whats occured. i dont know what futures this obstructs but it's a drain on general human possobility & energy that man-made reality, unlike the gods, is so deeply resistant to revelation & exploration.
i've seen comments calling the submission a vanity project, and it admits to it's owm uninterestingness to a certain degree. i think karlicoss's far broader spanning digital self sensing & ordering has a far higher chamce of helping us distill out meaning & sense from the world. but what a bold & hard sojourn just to start, just to begin to marshal the data. critics abound but i for one think we have so much, are only just starting to understand & calobrate ourselves in a technoaugmemted mode.
currently the technology all resists technoaugmentative exploration. this is a crime against humanity, against our base human spirit. we dont know what is possible, what we could do. we cannot explore better possibilities. we have now two examples of goong far, but all the systems around us resist this open exploration: we are all chained to the wall of plato's cave, and few bother to break free. i await & pursue an emergence.
1. I've tried out many todo apps and they all aren't quite right for me (atm it's just a pen and paper), so I want to try design UX that's tailor made for me.
2. I found that arbitrary points and day streaks go a long way in making my habits stick. So I want to have an app that can add points/ streaks to anything that I do. (e.g. I complete "go to the gym" from my todo list, and it automatically adds a log to a table. Then that log is parsed to show some arbitrary point & streaks for going to the gym)
3. I want the app to have a REST endpoint that I can use to integrate with other apps to automate data entry.
Honestly, it's more of a procrastination thing than a productivity thing :p but have a lot of fun thinking about/ building the app.
I try to keep track of work notes and such, but when I've tried to be super strict about it, I always fall off.
Edit: after reading the full thing something that jumped out to me was summer/winter comparisons. If you have the freedom to work anywhere, chasing sun seems like it could have quite a positive impact on your general health and well-being.
I really hope that the world stops idolizing the digital nomad lifestyle.
Either I'm just being jealous, or he is indeed this vain, or both.
edit: it is not, he uses about 3x as much carbon flying (14.2 tons of CO2) as emitted by a typical passenger car per year (4.6 tons of CO2).
I can't help but think that flying will hopefully become a taboo topic, something that people do occasionally but definitely don't brag about.
I hope we can figure out more environmentally sustainable flight, not hope we just stop flying
How do you ensure that you consistently track? Automated pings? Physical buttons/sensors at your desk or I guess computer uptime maybe.
Other thing is did you have any concerns about making this info public, I guess it's not locating you exactly.
Automation is key for 99% of the data with only a small amount requiring manual entry.
* Please note that a Happiness score of at least 70% must be maintained to retain employment. Additionally, you must think about Megaco in a positive way for at least 2% of your out-of-work hours."
All bleak jokes aside, interesting work by the author.
I also keep a personal website with some list of things I did and this has proven useful to serve as my backup memory.
As for logging of structured data, I've sometimes thought of coming up with some kind of annotation scheme, so that, if I need, in my writing, to log a measurable quantity of some kind, I could just annotate it in a special way. And then, later on, if the need should arise, I could just extract the annotated data from the text document. I've not taken it any further, but I think it could be useful. A sort of a "hybrid approach" that would let me focus on the writing and the flow of thoughts, while also letting me log tidbits of structured data in between.
Initially I thought this was going to be some quip, like "I put my whole life in a single database.... and it was two tables of ten rows each".
But getting your personal data out front and in the open, does worry me somewhat how it might be exploited by nefarious parties, or even insurance companies, finance companies etc.
I would like to push back on the position that if there are no insights gleaned then there is zero value in this work.
There is plenty of value in owning your data and bringing awareness to what companies are storing about you. Further I find tracking my data satisfying and meditative in itself. Digital collecting. Example: https://data.tomhummel.com/sneezes/
> How does decomposing and displaying my life in a database change who I am or who I want to be?
The quantitative display is interesting as well; there are multiple sources where this information could be gathered from by OEMs. It does make me wonder if someone is using Apple Health, Apple Credit Card, an iPhone, and using Apple Music if most of this information isn't already available and potentially displayed in a similar manner.
Apple of course is just used as an example.
My wife tracks just about everything in her life - and me almost nothing. I’d like the narrow the gap just slightly!
So I definitely welcome them to get inspired to do more.
I'm surprised most sentiments on HN are "super cool" rather than "scary".
I guess some people prefer attention over privacy.
- The data is stored in my own, private database, not connected to any service - Much of the data visualised on that page, is data many larger companies already have about you. I figured, why wouldn't I want to benefit also? - The data shown above seems very personal but doesn’t actually expose any real sensitive information. For example, disclosing your current location, your home address, or stores and places you frequently visits is sensitive and potentially dangerous. Even looking at the map of my movement data on my page, you won’t get any more information that what most people have in their public Twitter bio.
I made sure all graphs are non-interactive, and had all personal information stripped out.
- The data is stored in my own, private database, not connected to any service - Much of the data visualised on that page, is data many larger companies already have about you. I figured, why wouldn't I want to benefit also? - The data shown above seems very personal but doesn’t actually expose any real sensitive information. For example, disclosing your current location, your home address, or stores and places you frequently visits is sensitive and potentially dangerous. Even looking at the map of my movement data on my page, you won’t get any more information that what most people have in their public Twitter bio.
I made sure all graphs are non-interactive, and had all personal information stripped out.
When I was looking at the stats, I realized I can never make something like this. Just alone the list of flying spots mentioned are huge for me. I don't think I have travelled at much internationally. Having a location graph for myself would be mostly few cities and maybe 1 or 2 countries(connecting flights).
Maybe 1 day I get to travel a lot. :)
> Overall, having spent a significant amount of time building this project, scaling it up to the size it’s at now, as well as analysing the data, the main conclusion is that it is not worth building your own solution, and investing this much time. When I first started building this project 3 years ago, I expected to learn way more surprising and interesting facts. There were some, and it’s super interesting to look through those graphs, however retrospectively, it did not justify the hundreds of hours I invested in this project.
(I don't have any big opinions about the effort, I just wanted to hear what was the database chosen that was able to track all different data formats... and it could have been sexier, but yes - Postgres would do.)