Comment by jeromeparadis | Hacker News Reader

Comment by jeromeparadis | Hacker News Reader

Parent Full thread

Hi!

Thanks for this update. This addresses a great security concern so that our buckets cannot be abused (especially filled) from outside of our apps.

I still have a remaining concern which is more about consistency. If you authorize users to upload for your app with a policy, they still can upload anything to their will without your app even knowing about it. Sure, the new policies can ensure that they won't be able to read what they upload if it never reaches your app but they still can upload huge amounts of data in your buckets without that data even being used or known of by the app.

I sure could write a backend scheduled task to clean up unknown files in our buckets.

But, what I would eventually like to see is a way to acknowledge that file handles we're received by the app. If, after a specified timeout no acknowledgement has been received from our app, then these files would be deemed dirty and you would then be able to clean up your servers and our buckets from these files.

Does my concern make any sense?

In any case, thanks again for this update. This remaining concern can be worked around and this update really mitigates the risk of someone wanting to abuse our buckets for malicious reasons.

Hi Jerome.

Thanks for the kind words. In regards to your concern, the way we imagine this playing out is that in your app, the client asks to upload something. You use an ajax request to ask your server, with your cookies, if this is allowed. If the user is allowed, you can create a policy and sign it. You should also set the min and max size on the policy. Then you can pass this along. Thus, you know of and approve of every upload request.

Does that sound reasonable? We has also considered a post upload hook where you approve of it. However, we imagined that people would not only want to know the metadata, but maybe even the contents before approving it which seems silly to pass the entire data around.

I think what will really help is if you set up the policy as outlined above, and then, do the clean up that you mentioned when we do webhooks where you could sign up to receive a notification after every upload.

Yeah, we'll manage it exactly as you mention and do some cleanup.

As for the idea of webhooks, in our case, we wouldn't need the content, just the meta data. That's what we already do anyway: we only store references to the S3 files we created. The idea is that it simplifies be able to run multiple Web server instances by having this user generated content offsite.

My concern was really more about us confirming what we receive to you so you could do the cleanup of non acknowledged files instead of us doing it. What you are suggesting is an additional verification that can have value but doesn't address the question of constituency between what ends up in our S3 buckets and what we end up being aware off in our app. In any case, no big worry but I'll email you details.

My bad, your webhook proposal, thinking of it, would indeed facilitate immensely the integrity between our database and the S3 files.

It would require us coding a scheduler. But it would be easy to write and would be way more efficient (than what we can do today). Would simply need to check your notifications against our database and accordingly clean up/validate the files in the S3 buckets.

Great stuff.

Reply on news.ycombinator.com