bf16
also: bfloat16, Brain Float 16
A 16-bit floating-point data type (bfloat16) that uses 1 sign bit, 8 exponent bits, and 7 mantissa bits, offering reduced precision compared to float32 while maintaining the same exponent range.
bfloat16 (Brain Float 16) is a machine learning-optimized floating-point format developed by Google that truncates IEEE 754 float32 to 16 bits. It preserves the full 8-bit exponent while reducing the mantissa to 7 bits, allowing it to represent the same numeric range as float32 but with lower precision.
Unlike float16 (half-precision), which has a smaller exponent range and can cause numerical instability in deep neural networks, bfloat16 maintains float32's dynamic range while reducing memory usage by 50% and improving computational speed on compatible hardware like TPUs and modern GPUs.
Example: A value like 3.14159 stored as float32 uses 32 bits; stored as bfloat16, it uses only 16 bits with minor precision loss but identical exponent capability, making it ideal for training large language models where dynamic range matters more than mantissa precision.