AWS does the optimizations over time based on access patterns for the data. Should have made that clearer in the article.
The problem becomes unusual burst load - usually from infrequent analytics jobs. The indexing cant respond fast enough.
The problem becomes unusual burst load - usually from infrequent analytics jobs. The indexing cant respond fast enough.
> 3,500 PUT/COPY/POST/DELETE requests per second per prefix
> 5,500 GET/HEAD requests per second per prefix
Most of those APIs don't even take a delimiter. So for these limits, does the prefix get inferred based on whatever delimiter you've used for previous list requests? What if you've used multiple delimiters in the past?
Basically what I'm trying to determine is whether these limits actually mean something concrete (that I can use for capacity planning etc.), or whether their behavior depends on heuristics that S3 uses under the hood.
I'm fine with S3 optimizing things under the hood based on access my patterns, but not if it means I can't reason about these limits as an outsider.
S3 simply looks at the common string prefixes in your object names and uses that to internally shard objects, so you can achieve a multiple of those request limits.
aaa122348
aaa484585
bbb484858
bbb474827
Would have same performance as:
aaa/122348
aaa/484585
bbb/484858
bbb/474827