FP8 (8-bit floating point)
Lower-precision floating point format using 8 bits instead of the standard 16 or 32 bits, reducing memory requirements and computational costs while maintaining model quality through techniques like periodic accumulation to higher precision (FP32)
As defined in The Engineering Unlocks Behind DeepSeek | YC Decoded
Other definitions
A low-precision numerical format using 8 bits to represent floating-point numbers, reducing memory usage and speeding computation compared to 16-bit or 32-bit formats, at the cost of reduced numerical precision
The Engineering Unlocks Behind DeepSeek | YC Decoded
Where it comes up · 1
Most important claims · 5 of 5Search all claims for “FP8 (8-bit floating point)” →
- DeepSeek's fp8 accumulation fix periodically merges calculations back into a higher precision fp32 accumulator to prevent small numerical errors from compounding, resulting in far more efficient training across thousands of GPUs while maintaining model qualityThe Engineering Unlocks Behind DeepSeek | YC Decoded
- At FP8 precision, AI training typically achieves only around 35% model floating-point operations utilization (MFU), meaning GPUs are only used at peak potential about a third of the time, with the rest spent waiting for data movement between caches or other GPUs.The Engineering Unlocks Behind DeepSeek | YC Decoded
- Deep Seek's FP8 accumulation fix periodically merges calculations back into a higher precision FP32 accumulator to prevent small numerical errors from compounding, resulting in far more efficient training across thousands of GPUs while cutting cost and maintaining model quality.The Engineering Unlocks Behind DeepSeek | YC Decoded
- At fp8 precision, AI labs typically achieve only around 35% model FLOPs utilization (MFU), meaning GPUs are utilized at peak potential only about one-third of the time while the rest of the time GPUs wait for data movement between caches or other GPUsThe Engineering Unlocks Behind DeepSeek | YC Decoded
- Deep Seek V3, combining multiple efficiency innovations (8-bit training, FP8 accumulation fix, mixture of experts, multi-head latent attention, multi-token prediction), is one of the most impressive base models on the market and has been available for some time (released December 2024).The Engineering Unlocks Behind DeepSeek | YC Decoded