DigitalOcean

Teknoloji & Yazılım

Continuous Batching Improves Your P50 and Can Wreck Your P99: The Measured Tradeoff | DigitalOcean

We measured continuous vs static batching on a DigitalOcean H200 GPU Droplet with vLLM, and what chunked prefill does to p99 stalls.