The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis
Hari Prasad, Ritam Pal
Why It Matters
What makes this one worth your time
Understanding the joint effects of quantization and sampling temperature on LLM safety is crucial for deploying models that are both efficient and aligned with safety standards.
Quantization and sampling temperature affect LLM safety alignment differently, with quantization being safety-neutral and sampling increasing instability.
Summary
The paper investigates the combined effects of quantization and sampling temperature on the safety alignment of large language models (LLMs). It evaluates 8 instruction-tuned models across various configurations, finding that quantization is generally safety-neutral, while higher sampling temperatures increase decision instability. The study suggests that quantization can offset some temperature-induced degradation and emphasizes the importance of multi-benchmark evaluations.
Key contributions
- Comprehensive evaluation of 8 instruction-tuned models across 144 configurations.
- Introduction of a Compound Degradation Index to measure the interaction between quantization and sampling temperature.
- Demonstration of the inadequacy of single-benchmark evaluations for assessing safety risks.
Notable insights
- Quantization is generally safety-neutral, contrary to concerns about low-bit deployment eroding alignment.
- Higher sampling temperatures significantly increase decision instability, highlighting the need for careful temperature management.
Possible limitations
- Not stated in the abstract
Abstract
arXiv:2606.29581v2 Announce Type: replace-cross Abstract: Modern LLM deployments often combine quantization with higher sampling temperatures to reduce cost, latency, or repetition, yet safety evaluations usually treat these as fixed implementation details. We test whether models that are safe at FP16 with greedy decoding remain safe after quantization and stochastic sampling, or whether the two factors amplify each other. We evaluate 8 instruction-tuned models from five families across 3 precisions and 6 temperatures, covering 144 configurations on 7 harmfulness benchmarks and generating about 2.0 million responses, which are scored by a six-judge safety ensemble. Contrary to concerns that low-bit deployment erodes alignment, we find that standard quantization is approximately safety-neutral: for 7 of 8 models, AWQ INT4 keeps attack success within about 1.6 percentage points of FP16 or lowers it, with clear degradation only for SmolLM3-3B (34.5% to 44.1%). However, the larger risk comes from sampling: higher temperatures sharply increase decision instability, with DFR reaching 41.9% at T = 1.0, even when average ASR changes only modestly. The two factors do not compound: our Compound Degradation Index remains sub-additive (-0.071 to +0.018), indicating that quantization partially offsets rather than amplifies temperature-induced degradation. Finally, a per-benchmark breakdown shows that single-benchmark evaluation badly understates risk: several models scoring 0% on AdvBench exceed 80% on ManyHarm. Standard INT4/INT8 quantization can therefore be reasonable for well-aligned models, but safety claims should report multi-sample stability across multiple benchmarks rather than rely on a single benchmark at greedy decoding.