Optimizing Large Language Model Decoding Performance with NVIDIA MI100 GPUs: A Case Study
Introduction
As artificial intelligence practitioners and system architects push the boundaries of large language model (LLM) deployment, hardware optimization plays a crucial role in achieving efficient and cost-effective inference. This article presents a detailed overview of optimizing decoding throughput using a high-performance server configuration featuring four NVIDIA MI100 GPUs. Through custom software modifications and architectural optimizations, it is possible to significantly boost decoding speeds, making AI workloads more accessible and affordable.
Server Configuration and Hardware Features
The system employed consists of four MI100 accelerators, each equipped with 32 GB of VRAM, interconnected via NVIDIA’s Infinity Fabric Bridge (XGMI). This high-speed interconnect provides remarkably low latency and high bandwidth during data accumulation steps, surpassing traditional PCIe connections in performance. Such hardware architecture is well-suited for demanding LLM inference tasks, including those requiring extensive context handling and high concurrency.
Software Optimization Approach
Initial benchmarks using the standard vLLM framework yielded only modest decoding speeds around 15 tokens per second. To unlock the full potential of the hardware, significant software engineering efforts were undertaken. This involved developing a forked version of vLLM tailored specifically to the MI100 architecture, incorporating:
- Custom kernel implementations optimized for the ROCm platform
- Fine-tuned numerical computation routines, including 8-bit integer (int8) quantization with optimized rounding
- Enhanced memory management strategies to reduce latency and improve throughput
- Compatibility fixes addressing numerics and memory tracking issues in upstream versions
The result was a dramatic increase in decoding speed, achieving over 1,000 tokens per second during decoding and more than 5,000 tokens per second during the prefill phase at eight concurrent streams.
Key Technical Innovations
The optimization process included several noteworthy advances:
- Development of custom kernels that take advantage of MI100’s hardware capabilities
- Fine-tuning of int8 quantization parameters (group size 128 with precise rounding techniques) to ensure numerical stability and accuracy
- Implementation of compatibility patches for models such as Qwen 27B, which previously experienced reliability issues like garbled outputs at long contexts
- Extensive testing with high-context, high-concurrency scenarios to verify reliability and quality over prolonged operation
Performance and Cost Benefits
These enhancements drastically outperform publicly available APIs, reducing operational costs from approximately $80 daily per instance to a more manageable expenditure, especially valuable for mid-range setups. The specific features of the MI100, particularly the XGMI interconnect, contribute to superior performance during the model’s accumulation phases, facilitating faster and more efficient inference.
Open Source Contributions and Distribution
The custom modifications and optimizations are publicly available through a dedicated GitHub repository:
https://github.com/curvedinf/int8-vllm
This fork includes custom versions of the AITER library and pre-trained 27B checkpoints, optimized for int8 quantization with parameters fine-tuned for the platform’s numerics.
Practical Use Cases
The optimized setup has been employed to run large language models on a low-power profile (C6 state) with supported contexts of up to 256k tokens. Applications include:
- AI assistants and chat agents
- Business process automation
- ML research and model training
- Development of reliable, high-throughput inference pipelines
Hardware Efficiency and Economic Considerations
The system consumes approximately 540 watts at the wall under load, with noise levels slightly above typical desktop environments. Despite this, the cost-to-performance ratio is highly favorable, especially considering that such a multi-GPU setup was acquired for just over USD 6,000. The investment proves advantageous given the substantial performance gains and flexibility it offers.
Conclusion
Achieving high-performance large language model inference requires both robust hardware and tailored software optimizations. Through dedicated kernel development, numerical tuning, and software fixes, it is possible to unlock the full throughput potential of multi-GPU configurations like the NVIDIA MI100. These efforts not only improve efficiency and reduce operational costs but also provide valuable insights and tools for the AI community to deploy larger, more capable models effectively.
For practitioners interested in enhancing their LLM inference pipelines, exploring custom forks and optimizations akin to those described here can be a rewarding endeavor, leading to substantial performance improvements and richer experimentation opportunities.
