- [2026/01] BatchGen v1.0 released with support for DeepSeek-R1/V3-671B.
BatchGen is a high-throughput batch inference engine designed to minimize batch completion time (BCT) for large-scale batch workloads and MoE-based LLMs.
BatchGen introduces the sequence coroutine compute model, which treats each sequence's computation as an event-driven coroutine that can be paused, resumed, and reorganized. A static planner optimizes batch configurations through lightweight profiling, while a dynamic sequence scheduler yields, combines, and migrates sequence coroutines at runtime—enabling larger expert-level batches for sparse MoE models, mitigating long-tail stragglers, and maintaining high device utilization across GPU clusters.
- Large-scale offline inference and data processing pipelines
- Synthetic data generation
- Model evaluation and benchmarking
- Test-time scaling (e.g., chain-of-thought, self-consistency)
- RL rollouts and post-training workflows
- 🚧 Kimi-K2.5 (1T) support and H20 performance results.
- 🚧 Stabilize and release the partition and migration primitives.
- 🚧 Develop more adaptive scheduling policies for better resource utilization and workload balance.
- 🚧 Support OpenAI-compatible tool-calling.
Host Memory: Must be larger than the model size. Additional memory is used for the host KV cache, which stores KV states for sequences waiting to be processed. Larger host KV cache sizes generally result in better throughput.
For DeepSeek-R1-671B (~700GB model weights), we recommend at least 1TB host memory: ~700GB for model weights + 300GB for host KV cache.
git clone https://github.com/batchgen-project/batchgen.git
cd batchgen
./scripts/install_deps.shThis script automatically installs:
- PyTorch with CUDA 12.8 support
- flash-attention 3 (Hopper optimized)
- FlashMLA
- DeepGEMM
- BatchGen
For manual installation, see the Manual Installation Guide.
For complete deployment instructions including model download, checkpoint conversion, server setup, and submitting jobs, see the Deployment Guide.
- Deployment Guide - Step-by-step guide for multi-node deployment
- Server Flags Reference - Complete list of all server configuration flags
- Client API Reference - Python client usage and parameters
- Manual Installation - Step-by-step manual installation instructions
We learned from the following projects when building BatchGen:
- SGLang - High-performance serving framework for LLMs
- vLLM - High-throughput and memory-efficient inference engine