Part of the problem is all those companies who use AWS "standard practice" services, who assume the cost of bandwidth is just what AWS charges, and compute-per-page is just what it is, and don't even optimise those (e.g. S3/EC2/Lambda instead of CloudFront).
I've just compared AWS egress charge against the best I can trivially get at Hetzner (small cloud VMs for bulk serving https cache).
You get an astonishing 392x(!) more HTTPS egress from Hetzner for the same price, or equivalently 392x cheaper for the same amount.
You can comfortably serve 100+ TB/month that way. With 10,000 pages times 1000 bots per month, that gives you 10MB per page, which is more than almost any eCommerce site uses, when you factor that bots (other than very badly coded bots) won't fetch the common resources (JS etc.) repeatedly for each page, only the unique elements (e.g. HTML and per-product images).