Provisioning for the worst possible case is sometimes necessary (e.g. hard real-time use cases), but can also be very expensive.

Since HTTP connections are typically kept alive between requests, it's very common for them to be idle. It could be the case that 99.9% of the time, X bytes of memory is enough to avoid transfer delays, but to cover the 100% case, you'd need 10X that.

I think it's reasonable to decide that avoiding delays in the 100% case is not worth 10X the cost in memory.