Imagining a personal data pipeline | Hacker News Reader

Imagining a personal data pipeline

joshcanhelp.com

122 pointssurprisetalk45 comments

I also think about this general problem a fair amount, but this:

> ... There is a whole bunch of toil baked into this hobby and I'm wary of creating an endless source of digital chores for myself

Always stops me from pursuing it seriously. Also that I already do a fair amount of ELT at work and couldn’t tolerate it as a hobby.

But this framework makes sense. Seems like the idea is connector, schema mapping, and datatype standardization configured in one place. It’s a well thought-out framework and I actually have an internal platform at work that accomplishes something very similar, albeit for a totally different purpose.

But I also personally wouldn’t see a ton of value from this, except if it were used for bills, taxes, and financial management. But then the privacy aspect becomes paramount. There’s a reason people just have to do that stuff manually.

I would be surprised if Apple and Google didn’t eventually start to build something like this. Google is already pretty good at unifying email and calendar. It’s something that’s really only possible to deploy at the mobile OS level, because any other alternative would involve sending all of your data to a third party platform, which for most people these days is a non-starter. Plus with LLMs being a thing, the perfect interface to centralized/standardized personal data now exists.

> except if it were used for bills, taxes, and financial management.

I have another tool I built for finances:

https://github.com/joshcanhelp/budget-cli

I'm trying to think of a way to get the two working together because there is a lot of overlap in what they're trying to do.

> I would be surprised if Apple and Google didn’t eventually start to build something like this

They could certainly do it but I think it goes against what they want to do with the data, which is gather and centralize it. Maybe folks would go for it but the core idea here is saving for yourself, not storing it in another platform.

> It’s a well thought-out framework

Appreciate that, thank you!

HN user SimonW who created Datasette gave a talk in 2020 on “dogsheep” his tool for harvesting and processing personal data from a series of third parties.

https://simonwillison.net/2020/Nov/14/personal-data-warehous...

Here is more about dogsheep — (“ Dogsheep is a collection of tools for personal analytics using SQLite and Datasette.”)

https://dogsheep.github.io/

I was surprised, I thought the article was going to be about dogsheep, especially when the article makes a point of linking to prior work.

Absolutely no shade, it just goes to show how hard it is to keep track of All The Things.

I’ve got a small number of links to such things here — (which is how I could track down dogsheep quickly)

https://wiki.secretgeek.net/personal-data-liberation

Thank you for this! Looks like a kindred project. I have hard of Datasette before but not Dogsheep, I'll give it a look.

My friend and I had a similar idea a few years ago, so we’ve built a prototype of a tool that converts personal data exports to a single SQLite database: https://github.com/bionic/bionic (repo includes “popular Spotify songs when I’m in transit according to Google Maps” query). Unfortunately, we haven’t found ourselves actually using the aggregated data: we’ve looked on it a few times, but it didn’t end up solving some real pain. It was fun to build though!

I've had a few uses for PDPL so far and I'm trying to add functionality when I need it rather than taking a shortcut and just writing the script I need in $LANGUAGE. But, like you said, it was fun to build and I learned a lot so if it's not useful to me or anyone else, that's OK too!

I have built a timeline thing to gather all of my data as an augmented diary.

https://nicolasbouliane.com/projects/timeline

The newer version is basically a static site generator running on top of my data. The older version was actively fetching the data from various sources, but it was getting a little unwieldy.

The biggest challenge is to automatically get your data out of Google Photos, social networks, and even your own phone. All of my handwritten notes and my sketches are stuck on my iPad and must be manually exported. It's tedious and unsustainable.

Same with social networks. Data must be manually exported. There is also no simple, human-readable file format for social media posts. You have to parse the export format they choose to use at the moment.

This is beautiful and something I wanted to accomplish, in my own way, as a part of this project. I love this:

> It collects all your crap, and it puts it on a timeline. > I thought it would be neat.

Everything you're describing in your comment here is what I'm trying to do with PDPL, get the data out of those services so you can use it. In your case, it looks like Timeline focuses on files ("The filesystem is the database") so you could (assuming it works for you) have an output function that formats the filenames in a way that you can use and adds the date.

This is the whole main challenge of the quantified self movement all over again.

There's a lot of attempts to solve this problems but not much has been found, possibly because the whole setup of ELT processes is a lot of chores (just think about the whole inconsistent formats of data across services). It's like having a second job in data engineering, and I'm not even remotely in the software/data industry! I just like and do coding as a hobby.

I'm wary about adding chores to my life, that was something I mentioned in the article. I've been running the API crawler for a while now and, so far so good.

The quantified self movement is adjacent to this but not quite. If you are gathering data from a service with an API, this can grab that and you can work with it. I used to wear the Oura ring and now I have 2+ years of data sitting in JSON on my laptop that can be ported to anywhere. I'm not really looking to, like, optimize productivity or physical output or anything, I just want to have and work with data from services that make my life a little easier.

This is why we built CloudQuery (https://github.com/cloudquery/cloudquery) an open source high performance ELT framework powered by Apache Arrow (framework is open source, our connectors closed source). You can run local pipeline and write plugins (extractors) in Go, Python, Javascript and any other language and save data to any destination (files, SQLite, DuckDB, PostgreSQL, ...)

(Founder here)

Oh, this is really fantastic, I'm going to check it out! Does you end up with an archive of the data that's being queried?

In case we might be a good fit ... https://www.joshcanhelp.com/hire-me/

I have a project I've built that's somewhat like this, ironically called Pipeline [0]. It's a manual entry timestamped note taking system, and the UI is like messaging yourself. I've set it up over a wireguard VPN server and it connects all of my devices, it works offline as a PWA, and I've tested it on chrome/Firefox/safari on iOS/Linus/android/macos/windows. It mostly works on all of those platforms and some of my friends/family use it to take notes for themselves.

The fundamental query I usually use is substring search. The only contents is text, because I believe in the primacy of plaintext. The notes for the last 4 years of my life takes up 60 megs, and it takes half a second on a 5 year old android phone to parse all of it, and less than 50ms to search through all of it, so I can do it on every keystroke/ incrementally.

[0] Pipeline Notes: https://github.com/kasrasadeghi/pipeline-js

I'm not a web developer by trade, so if anyone has any feedback on security/UI/service workers, please let me know!

I've always thought that the messaging/texting interface is really nice and intuitive. It's like your terminal history but more ... personable?

I got it running locally and, while I wasn't sure all of what I could do, it was responsive and search worked well. Nicely done!

About a decade ago, I took on a personal toy project to try and teach myself larger scale programming in java, as well as the APIs provided by multiple internet services (google, twitter, facebook, ....)

The project was to try and collect and make searchable my "internet self". something I called "personal search". i.e. the idea was to try to index every web page I look at, every e-mail I get / see. Social media content shared to me by my social graph (using said APIs), and further indexing pages shared.

The indexing itself wasn't the hard part (per se, as at the time, the APIs facebook, twitter et al were very expansive, much more limited these days, one can attempt to deal with it with intelligent dom scraping, but that's a never ending race where your sources are consistently changing things and you are chasing their changes), the question is how does this information really create significant value for myself? i.e. how often am I going to actually be searching this personal archive. I search google many times a day to find new things (or refind things I already found through it), but how often do I search for things that are within my personal index during a normal day? a handful at most I'd think (and many times, not even once).

With that said, the concept that I then decided to try and teach myself was trying to write a browser extension that could do a similarity search (of sorts) between the documents in my personal index and the web page that I'm currently looking at + content related to that (ex: looking at a news article about current events, idea is that it should surface other articles your friends have shared on the topic and their comments on it). That ended up being an area i didn't have the time (or expertise) to really go far with, so it sort of ended there.

> how does this information really create significant value for myself?

This was a really important question I asked myself. If I build it, will I use it? I had two answers.

First, I do, somewhat regularly, have reasons to look back into this archive. If it mission-critical? No. Is it nice to have, yes? I like being able to see what happened on a certain day in the past, I like seeing how trends between activities come together, and I like tying systems together locally rather than in the cloud. I use Obsidian so bringing everything together locally is a big motivation.

Second, this is just a hobby, like any other. Having conversations like this, contributing to the universe of open source, connecting with people on the things I feel strongly about (data privacy, local first, life logging) ... it's fun and interesting and a good use of my life. There is a tendency to get a little obsessed with it all (family history of OCD forces me to be mindful about it) but I've enjoyed the process so far.

I love your browser extension idea. Managing content from the web that I want to save has been a big challenge locally.

For the subproblem of being able to unify and query various data sources in different formats, I would suggest to take a look at Datalog and specifically Mangle, my implementation of it. I don't want to plug the project here but more describe the approach.

Usually your data will comfortably fit in a file. Your data getter emits these files in facts (essentially relations). If you want structures data, it can also be a single column that is of some struct type (similar to protobuf).

With all data available the problem becomes one of querying. With a good enough query language and system, you write these can data transformations via Datalog rules which roughly correspond to database views.

It is always possible to write queries in code in a general purpose language, but is a bit clumsy and hard to get an overview or reuse. It may also be possible to do SQL but it SQL is not very compositional and you ask yourself whether the base data representation should be adapted refactored. Essentially you do not want to think about the optimal schema or set of structs but just do transformations you need in the lowest friction way.

With Datalog you may benefit from a unified representation (everything is "facts") and the transformations to useful different formats (different kinds of facts) can be factored and reused. It may mean duplication and denormalization but usually that does not matter.

Mangle supports aggregation and even calling some custom functions during query evaluation. The repo is at https://github.com/google/mangle and obviously there remains a lot to do, the API is unstable, there are bugs and the type checker is not finished... but a number of people and projects seem to use it. Even if you do not use it, it may give you how to use facts (relations) as a unified data structure for your project.

Thank you very much for writing all of this out!

My abstracted YAML representation worked pretty well for this initial PoC but I'm seeing places where this is just not going to fly for more complicated queries. I'd much rather string together more competent technologies to accomplish my goal rather than spending a year reinventing the wheel.

Along the lines of your comment ... trying these queries in a general language and in SQL brought me to the same conclusions you have here. One of the foundations of this is open source so if I write a recipe and contribute it, I want someone else to be able to take it and tinker with it or extend it but do that without having to deal with too much boilerplate.

I will take a look at Datalog. I know I've seen it before but don't know much about it. Thank you again!

This is actually a great use case for something like Windows Recall. Ingestion of data after the fact requires the data to be discoverable.

If there was a way to add a meta-prompt to Windows Recall like "Create a log entry every time I watch something with its title and URL" it could serve as a history whether things were watched on YouTube, Vimeo, or any other site, without requiring plugging into each service individually. Repeat ad nauseum for each thing to be logged, or perhaps someone can come up with a more clever query than I that catches everything sufficiently.

The level of granularity on many services might be surprisingly large, preventing introspection of the data at a useful level.

This is a horrible use case for Windows Recall. Even if we ignore all the privacy implications of having a third party screenshot you every 30 seconds and making the files world readable, it's a bad idea.

Recall has lost a ton of useful metadata you already have - both URL visits and streaming are clearly discernible actions, both at the network stack level, and from your browser history. Throwing that away to trust an LLM to re-infer the same data is both reducing data fidelity and significantly increasing processing cost.

If you want to see this done reasonably well, I'd suggest looking at e.g https://beepb00p.xyz/promnesia.html (which not surprisingly bears a strong similarity to what the article discusses)

LLMs don't add much value here, outside of tightly locked down systems where screenshots are the only way of exporting.

Sorry when I said something like Windows Recall, I didn't mean Windows Recall but software with similar capabilities. I think in my mind I was imagining some sort of ongoing screen capture along with a meta prompt or prompts, and some sort of output.

The value the LLM adds is interpreting/processing data without having to tailor input streams. Imagine if formats change, fields get renamed, and so on. The maintenance would be a headache if this was done on a per-service level. I think the reduction in fidelity seems like a reasonable tradeoff, but that's for the user to decide of course along with local/cloud processing and proprietary/open source software.

Even things like invoices from the same service change format over time.

I've been using https://www.manictime.com for maybe close to 20 years now, although not the pro version that offers screenshot recording (curiously the website doesn't mention the existence of a free "standard" license). It records window titles and presence/away times.

A prompt every few minutes that would ask "What are you doing now?" would be interesting to me, as a professional procrastinator. Maybe an even better one would be one that says something like "In the last 10 minutes, you spent 90% of it on Hacker News".

The (non-privacy related) issue is the same - if you resort to screen shots, you've thrown away tons of valuable metadata.

And, as long as there's an API, I am fairly certain that maintaining a compat layer is a lot less work than retuning the LLM when the images change. (And you'll want to adjust your tuning, at least with current SOTA, or your error rate will reach Unpleasantville fairly quickly)

Yes, it seems easier on the face of it - but the reality of building an LLM pipeline will quickly point out a lot of edge cases.

Browser history does that pretty well currently.

I've been talking about this with a friend/colleague of mine. Some of the greatest sources of data are "recorders" that are running in the background. Oura or any other activity tracker, Google Maps with location history turned on, browser history. Data privacy concerns aside, those sources can really help tie things together.

Personally, I stopped using my Oura ring and don't have location history collected anywhere (that I control or know of). One of my big ideas for this is a native application for your phone and laptop that collects everything it can in the background and surfaces it somewhere you control (synced via a service you choose, lands on a machine you own). Maybe overkill for many people but being able to access that without giving it to another service would be something I'd use.

I always wonder what people would do with this data. I have a more "tears in the rain" approach. I just don't think there is that much value in, say, my old workout logs. I cannot see how capturing and analyzing it all would make my life much better. I feel if there was one crazy hack that would, say, make me stronger, the people who's job is to get as strong as possible would have already found it. (It's probably PEDs.)

> I just don't think there is that much value in, say, my old workout logs.

To you, perhaps, but witness the excitement of historians and archaeologists when they find things like the Ea-nāṣir complaint[0] or an Akkadian shopping list[1].

You never know what might be interesting to future peoples.

[0] https://en.wikipedia.org/wiki/Complaint_tablet_to_Ea-nāṣir

[1] https://news.artnet.com/art-world/akkadian-cuneiform-tablet-...

In my case, I use it as an enhanced photo stream slash diary. It let's me see what I was up to on a given day. I see my location, my photos, my diary, my search queries and my transactions, among other things.

I second both the sibling comments here but I'll add a few things:

- Having a detailed meeting and project history from work helps me during interviews, writing resumes, reminding myself of previous approaches. A lot of this comes from detailed note-taking but making connections with people I've worked with, companies I've interacted with, etc adds a lot of color and detail.

- In a similar vein, tying together people with contacts (from Google or iCloud), events, and notes helps me be a better friend, neighbor, family member in my personal life. I don't have an excellent memory so I build it out in Obsidian.

- I like being able to do simple things like figure out how many bike rides I took since I've been laid off (65 bike rides, totaling over 645 miles and 138,000 feet of elevation gain).

Does this all change my life drastically? No but I like the work, I like how present in my life it makes me, and it's fun.

I made something for just this use. It's very simple, but it gives you all your data in compressed JSON. I use it myself for mostly diet and exercise, but I also log other things like movies and books I'm reading. I made it because I wanted a nice interface to review my logs.

https://www.idiotlamborghini.com/strategies/weave

I would love to hear the story of where your domain came from! I love it.

The idea behind Weave is something I would really like to adhere to. I take a lot of notes but get bored by tracking.

That said, the origin story you have on that page really got my attention. Around that year that's mentioned, I was going to 24 Hour Fitness and was pushed by a trainer to keep a food log. I did and ended up losing almost 100 pounds total. What I learned from that log stays with me today and I'm able to keep a vague log in my head when I need to.

The promise of computing that never materialized was having software to track everything you've ever done, and leveraging that data for your own benefit and no one else's.

I love this and agree 100%. I feel like we, as a species, had to go "full internet" to understand the trade-offs of connectivity. First it was all local, then all internet, now it's feeling like the pendulum is swinging back the other way. Maybe I'm just in an echo chamber ...

I appreciate your response. I hope it swings back.

Well... I have mine in Emacs/org-mode/org-roam managed notes, integrated being in a single integrated platform with no need of extra code, with my mails (notmuch), contacts (org-contacts), financial transactions (beancount), file (org-attach-ed) etc down to the infra/OS config (NixOS tangled from org-mode, org-babel blocks, as per the Emacs, zsh, mplayer, ... configs).

The point is that with classic tools IS EASY, limited only by the current sorry state of IT things, with modern tools is a nightmare that demand much effort to be done.

oh man... we had so much of this done years ago with Locker, but old services die and new services are born, and existing services change their APIs constantly... it takes a lot of work to keep them up to date! https://github.com/LockerProject/Locker

Lots of prior art here so I knew I was not reinventing the wheel. I guess I just had to see what the challenges were first hand!

FYI, if you have control over the repo ... the URL on the sidebar, "lookerproject dot org" goes to a scammy-looking crypto thing, might want to take that down.

I think the domain was lost long ago, sadly

I totally get the allure of building this stuff :)