Surfer: Open-Source Personal Data Warehouse

Even so, would a rate limit be wise?

So it's not really a Personal Data Warehouse (PDW?) at all but rather an open replacement for Twitter/X, at least Bluesky is that. However Bluesky is really an MVP for ATProtocol which is meant to serve as an open protocol to allow decentralized data sharing where each user is in control over their data (at least that's my understanding of it).

So if you consider the spectrum of centralization -> decentralization for current web apps and data. Things like Twitter/X and any web 2.0 app are fully centralized where the app controls all your data.

Things like Mastodon and the Fediverse are decentralized but suffer from fragmentation and a lack of a global picture of the Fediverse.

ATProto tries to find a middle ground to reap the benefits of both. Users have a decentralized identity (did) which they can control and their content is tied to that and stored in a PDS (Personal Data Server). To bootstrap the process, Bluesky creates and hosts PDSs for new users but you can self-host that if you like (see https://atproto.com/guides/self-hosting). To reap the web2.0 benefits of a global picture there are relays which catalog and index the data from the PDSs. The architecture is explained quite well in https://atproto.com/articles/atproto-for-distsys-engineers.

So Bluesky is in a sense just an atproto MVP and people are building other things on top of it. Some examples:

* WhiteWind (https://whtwnd.com/): atproto blogging platform

* Smoke Signal (https://smokesignal.events/): atproto events platform (Meetup clone)

I think by default all the atproto/PDS data is public so maybe not what you want for a PDW but maybe you could make it work with per item encryption keys so that you can selectively share data with people if you want? Depending on what the goals are that might not be suitable from a performance perspective but I'm just brainstorming here.

I'm still trying to get my head around it all and so far the main innovation I see is around providing the decentralized-id (did) although the creation appears to still be centralized (but this is the part I understand least so far). Of course the other big thing they bring to the table is that they have about 25M users now so there is an actual network of users so this could take off. There are many great decentralized schemes but most of them never grow beyond a niche set of users.

I hope something like ATProto takes off so that users and humans/meat-persons can regain sovereignty of their data and we can create a free and open internet.

P.S. There's an interesting Github Issue in the ATProto repo: #3409 Call for Developer Projects (https://github.com/bluesky-social/atproto/discussions/3049#)

I have been using the python sdk on ubuntu and it's been working well, so far at least!