FP8 Quantization in Production: Lessons from Deploying Quantized LLMs

Why FP8?

In the landscape of model quantization, FP8 occupies a sweet spot. It delivers 1.4-1.8x throughput improvements over FP16/BF16 while maintaining greater than 99% accuracy recovery compared to full-precision baselines. For production workloads where quality matters, FP8 is the optimal tradeoff.

The Numbers That Matter

After deploying FP8 quantized models across multiple enterprise clients, here are the benchmarks we've observed:

• VRAM Reduction: 50% compared to FP16, allowing us to serve 70B-parameter models on hardware that previously couldn't handle them
• Throughput: 1.6x average improvement in tokens-per-second across our workloads
• Accuracy: Less than 0.3% degradation on our internal benchmark suite spanning code generation, legal analysis, and financial reasoning
• Latency: Sub-800ms first token on H100, sub-500ms on our Flash model

Production Considerations

Calibration Is Everything

Dynamic quantization is convenient for prototyping, but production deployments demand static scaling factors derived from representative calibration datasets. We calibrate on traffic patterns that mirror actual enterprise usage - not generic benchmarks.

Hardware Matters

Native FP8 acceleration requires NVIDIA Hopper (H100) or newer GPUs with specialized Tensor Cores. On older Ampere hardware (A100), we fall back to INT8 quantization which offers similar memory savings but slightly different accuracy characteristics.

Don't Forget the KV Cache

Quantizing model weights is only half the story. The KV cache can consume more memory than the weights themselves at long context lengths. We apply FP8 quantization to the KV cache as well, which often provides the biggest practical gain in terms of concurrent request capacity.

FP8 has become our default quantization strategy for all production deployments. It's the rare optimization that improves performance without meaningful quality tradeoffs - and for sovereign AI where every GPU matters, that efficiency is transformative.