Hacker News Reader

Comment by stevekemp | Hacker News Reader

Uploading a single file to S3 seems like it should be a simple job, that is well-documented:

https://docs.aws.amazon.com/AmazonS3/latest/user-guide/uploa...

There are parts of AWS that are hard to use, and non-intuitive. But S3 didn't ever seem to be one of them, though perhaps I'm forgetting how hard it was initially.

You might consider a "friendly" system for static hosting if raw-S3 is too hard though, such as netlify. There are a lot of services out there which basically wrap and resell AWS services. (I run my own to do git-based DNS hosting, which is a thin layer upon the top of route53 for example.)

View on HN

Ha, I can't upload anything into S3. I'm unable to, from memory, create or modify any of my own buckets. My account is in some weird half-suspended state from a past billing incident, long resolved, where things mostly work - but modifying S3 is not one of them. Support insisted everything was fine even as I uploaded videos of the failures and eventually I just gave up.

I leave a few legacy things running there (billing works, of course) but these days just put personal stuff on Digital Ocean, which seems to meet basically all my needs without the complexity and cheaper to boot.

Yeah I managed to get the file into the bucket. However I couldn't get the permissions to let it be publicly visible despite working through several tutorials.

I ended up on Netlify, it's 1000x more my speed.

I've never used S3 to host a website before so I decided to try this and see if it's really that complicated.

First, I searched for "host static website on S3" and found an AWS docs page with a walkthrough.

Then, I created an S3 bucket through the console. I uploaded a "hello world" index.html file to it.

Then I went to Properties, checked "Use this bucket to host a website" and gave it the name of my index file. When I clicked Save, I had a hosted HTML file that I could navigate to.

I'm struggling to see how it could be simpler than that. What exact problems did you run into?

So this was a while ago and I fully admit it might have been me being stupid.

However when I was doing it I don't think there was a "Use this bucket to host a website" button. Instead you had to give the bucket a special type of permissions which meant anyone could see the contents but no one could write to it. I kept following the guide as closely as I could but I couldn't get it to take.

Maybe it's been updated since then. This is the guide I followed: https://docs.aws.amazon.com/AmazonS3/latest/dev/HostingWebsi...

Ok yeah I think it was changing the policy that access I couldn't get to work.

Again it may be that is super easy and I fucked up something obvious however I tried a lot of times a lot of different ways until I gave up in rage.

Uploading to S3 is easy until it isn't. It works pretty nicely for one-offs but when you have to blast a few hundred gigs into it or large files from a bit of software you wrote it's a royal pain in the butt. The phrase "multi-part upload" makes me cry inside.

Why is that hard? The CLI automatically does multi part uploads. It’s also simple with the various AWS SDKs.

It looks like it works but it's not reliable so your reliability concerns then get externalised into your application which multiplies complexity terribly.

What do you mean by it's not reliable?

I’ve never had the experience myself, but I would assume he means you would have to build in some type of retry logic in your script.

Just from a cursory glance, I couldn’t find any samples of how to do a multipart upload with retries in Python with Boto3.

This is an example of how to do multipart uploads though.

https://medium.com/@niyazi_erd/aws-s3-multipart-upload-with-...

Fun Trivia note: when you do a multipart upload, the S3 hash of the object is not the same as it is when you do a single part upload. I had a file with the same contents but a different hash when I used Python than when it was transferred with the CLI or CloudBerry. The quick and dirty way to fix the hash is to copy the file to itself with Boto3.

Reply on news.ycombinator.com