Hacker News Reader

Comment by apitman | Hacker News Reader

The things we could build if S3 specified a simple OAuth2-based protocol for delegating read/write access. The world needs an HTTP-based protocol for apps to access data on the user's behalf. Google Drive is the closest to this but it only has a single provider and other issues[0]. I'm sad remoteStorage never caught on. I really hope Solid does well but it feels too complex to me. My own take on the problem is https://gemdrive.io/, but it's mostly on hold while I'm focused on other parts of the self-hosting stack.

[0]: https://gdrivemusic.com/help

View on HN

Absolutely this. I would LOVE to be able to build apps that store people's data in their own S3 bucket, billed to their own account.

Doing that right now is monumentally difficult. I built an entire CLI app just for solving the "issue AWS credentials that can only access this specific bucket" problem, but I really don't want to have to talk my users through installing and running something like that: https://s3-credentials.readthedocs.io/en/stable/

Most apps, however, assume POSIX-like data access. I would love to see a client-side minimally dependent library that mounts a local directory that is actually the user's S3 bucket.

Linux has FUSE, which is a framework to develop user-level filesystems. Mounting S3 buckets is a very good use case. Sshfs and httpfs are more or less similar in this regard.

Yep, and WinFSP and dokany are two options for FUSE on Windows. I'd recommend using rclone or maybe check this list: https://winfsp.dev/doc/Known-File-Systems/

Such a system would be amazing. It would really force companies whose products are UIs on top of S3 to compete hard because adversarial interoperability would be an ever present threat from your competitors.

It really is such a shame that all the projects that tried/are trying to create data sovereignty for users became weird crypto.

I agree with both halves of your comment, but I realized I can't identify the connection between S3 oauth and data sovereignty. Could you elaborate?

So the idea would be that you have an account with AWS (or realistically a more consumer friendly service that's Amazon branded) where all your data lives. Then when you use say Dropbox you can pick "Use my own storage" and grant Dropbox via OAuth the ability to write to /dropbox in your bucket and all your files would live there instead of Dropbox's servers. Lots of the data sovereignty solutions also include a database like interface you can grant apps the ability to use but I can't imagine that catching on initially.

Apple actually already does this with iCloud storage but hides it really well so it feels seamless.

Isn't this essentially how the Dropbox API already works (for apps that support using it)? I've used many apps over the years that offer this option alongside some alternatives.

You can get close with a Cognito Identity Pool that exchanges your user's keys for AWS credentials associated with an IAM role that has access to the resources you want to read/write on their behalf. Pretty standard pattern.

https://docs.aws.amazon.com/cognito/latest/developerguide/co...

edit: I think I misread your comment. I understood it as your app wanting to delegate access to a user's data to the client, but it seems like you want the user to delegate access to their own data to your app? Different use-cases.

We're building this at https://puter.com

You mean you're implementing something like this to be used by puter.com?

Apache Iceberg is kind of this, but more oriented around large data lake datasets.

Reply on news.ycombinator.com