fp16
also: half-precision, half precision floating-point, float16
A 16-bit floating-point data type that uses half the memory of standard 32-bit floats, commonly used in machine learning and GPU computing for faster computation with acceptable precision loss.
FP16 (16-bit floating-point) represents numbers using 16 bits instead of the standard 32-bit IEEE 754 single-precision format. It allocates 1 bit for sign, 5 bits for exponent, and 10 bits for mantissa, reducing memory footprint and increasing computational speed.
FP16 is particularly valuable in deep learning frameworks like TensorFlow and PyTorch, where neural networks can be trained faster on GPUs with minimal accuracy degradation. For example, a model using FP16 tensors trains roughly twice as fast while consuming half the VRAM compared to FP32.
Modern GPUs (NVIDIA Tensor Cores, AMD RDNA) have specialized hardware for FP16 operations. A typical PyTorch example: model = model.half() converts a model to FP16 precision for inference or training.