quantization
also: bit depth reduction, model quantization, precision reduction
Quantization is the process of converting continuous or high-precision numerical values into a limited set of discrete levels, commonly used in audio/video compression and machine learning model optimization.
Quantization reduces the number of bits needed to represent data by mapping a range of input values to fewer output levels. For example, converting a 32-bit floating-point number to an 8-bit integer representation reduces memory usage and computational overhead.
In audio processing, quantization determines how many discrete amplitude levels are available to represent a sound wave. Standard CD audio uses 16-bit quantization (65,536 levels), while lower bit depths like 8-bit produce more audible distortion.
In machine learning, quantization optimizes neural network models by reducing precision of weights and activations, making models faster and smaller while maintaining reasonable accuracy. A model quantized from 32-bit floats to 8-bit integers might run 4× faster on compatible hardware.