Model Quantization
Compressing model weights and activations from FP32 into INT8, INT4 or other low-precision representations to cut memory footprint and speed up inference, at the cost of predictable accuracy loss. Physics-AI models are sensitive to absolute magnitudes, so quantization must be calibrated and verified per operating regime.
What it is
Model quantization replaces FP32 storage of weights and activations with lower-precision representations to reduce memory and bandwidth and raise inference throughput. Common tiers are INT8 (roughly 4x compression versus FP32) and INT4 (roughly 8x). Weights may be quantized, activations may be quantized, or both.
Unlike mixed precision, quantization happens at deployment time: after training, the model is exported to ONNX and the inference engine quantizes it, producing an execution plan for the target precision.
Key points
- Quantization error grows with numerical dynamic range: physics-AI outputs usually carry
explicit physical units (K, Pa, m/s) and the magnitudes of different quantities differ by orders of magnitude. A single scaling factor per tensor can push small-magnitude physical quantities close to zero. Per-channel or per-physical-quantity quantization is usually much safer than per-tensor.
- The calibration set must cover real operating regimes: boundary conditions, mesh densities
and load combinations seen in production all belong in the calibration data. Insufficient calibration is the direct cause of quantized models that look fine in testing and drift in production.
- Quantization and distillation are often used together: when quantization costs accuracy,
distill the original model's soft-label information into the smaller model first and quantize afterwards; that loses less than quantizing directly.
- Keep the numerical space consistent: if different operators of the same model are executed
by subgraphs at different precisions, the differences accumulate at the seams.
Common pitfalls / boundaries
- Quantization is not lossless compression. FP32-to-INT8 rounding is usually acceptable for
relative output error, but can be large for small-magnitude yet physically critical quantities such as conservation-law residuals or boundary-layer gradients. Check those separately instead of only looking at overall RMSE.
- Quantization changes numerical precision, not model structure. Using quantization to substitute
for distillation to recover accuracy goes in the wrong direction.
- Quantization error couples strongly to the calibration distribution. After changing the
operating regime or mesh density, re-evaluate rather than reusing the previous calibration.
- Metrics designed for quantized vision models (IoU, mAP and the like) do not fit physical field
evaluation. Use absolute and relative errors in physical units plus conservation residuals.