Cutting LLM Batch Inference Time in Half: Dynamic Prefix Bucketing at Scale

Part of the Daft team here! Happy to answer any questions