GGUF
also: GPT-Generated Unified Format
GGUF is a binary file format designed for efficiently storing and loading large language models, prioritizing portability and inference speed across different hardware.
GGUF (GPT-Generated Unified Format) is a modern file format created by the GGML project for storing quantized machine learning models, particularly large language models (LLMs). It was developed as a successor to earlier formats like GGML to provide better compatibility, metadata handling, and performance.
GGUF files are structured to enable fast loading into memory and efficient CPU-based inference, making them popular for running LLMs locally on consumer hardware. The format supports various quantization levels (reducing precision to lower memory and disk usage) while maintaining reasonable model accuracy.
Example: A 7-billion parameter language model might be distributed as a GGUF file (e.g., model-7b-q4.gguf) that can be loaded and run using tools like llama.cpp or similar inference engines on a personal computer, rather than requiring cloud resources.