Hacker News Reader

Comment by db65edfc7996 | Hacker News Reader

>Running our test suite 2-3 times over the span of a couple hours would trigger the rate limit…

Not to single you out, but not too hard to imagine why companies start enabling rate limiting. I oftentimes wonder the obscene amount of bandwidth being consumed by this developer automation. This page[0] indicates that PyPi serves 900 terabytes daily.

Seems obvious we need more turn-key solutions that enable caching for all of the popular package distributions (PyPi, NPM, Docker, etc). Too hard to configure and everyone will default to the path of least resistance (ie hammering the origin servers).

[0] https://dustingram.com/articles/2021/04/14/powering-the-pyth...

View on HN

The fun thing about the docker rate limit is that the layer tarballs are stored on S3 in us-east-1.

If you do a docker pull inside us-east-1, you’ll get a direct S3 link instead of a proxied download, and thus neither you nor docker need to pay AWS anything for bandwidth!

> If you do a docker pull inside us-east-1, you’ll get a direct S3 link instead of a proxied download, and thus neither you nor docker need to pay AWS anything for bandwidth!

I thought cross-account data still cost something; is that not the case?

> You pay for all bandwidth into and out of Amazon S3, except for the following:

> Data transferred from an Amazon S3 bucket to any AWS service(s) within the same AWS Region as the S3 bucket (including to a different account in the same AWS Region).

https://aws.amazon.com/s3/pricing/

Ooooh very neat thanks for sharing:) That has... interesting potential...

Also consider enabling S3 gateway endpoints:

https://docs.aws.amazon.com/vpc/latest/privatelink/vpce-gate...

That avoids paying NAT Gateway charges for traffic to S3 and in my testing also made a bit of a latency reduction, which came in handy once when I had a one-off data small file migration.

Thank you:)

If you work in a large company, for security reasons you may already be required to use in-house mirrors of libraries or container images.

There are commercial products for doing this, e.g. https://jfrog.com/artifactory/

All the command line package management tools then need to be configured to point at the custom in-house package mirror, rather than connecting to the default public package server over the internet.

Agreed! I don't mean to complain about docker, they build great products.

Reply on news.ycombinator.com