If you store data and your web app doesn't need to read it back, you can use asymmetrical cryptography. It's slower but it's safe in that threat model. You'll have to download the files to decrypt them.
But even symmetrical cryptography have some value. If the attackers can download the files from S3 but they can't crack the web app, they can't access their content. Only very few employees should have the encryption key. If they know that only 3 of them have it, they should think twice about doing something wrong. If everybody knows it, it's a free for all.