Pro tip : for such scenarios (push static documentation), services such as Netlify, Cloudflare Pages, Firebase Hosting are preferable. Their costs are lower, free tiers are highly generous, and cache invalidation is handled automatically.
For gatsby sites and webpack apps, I did ultimately figure out and simplify my s3 / cloudfront deploys and only invalidate index.html.
Ultimately it looks like I'm going to be migrating away to sphinx-doc. It's not up to date with the best practices in static site apps these days.