LoRA
also: Low-Rank Adaptation
LoRA (Low-Rank Adaptation) is a machine learning technique that efficiently fine-tunes large pre-trained models by updating only a small number of additional parameters rather than retraining the entire model.
LoRA is a parameter-efficient fine-tuning method that adds trainable low-rank decomposition matrices to a pre-trained model's weights. Instead of updating all millions or billions of parameters during training, LoRA freezes the original weights and only trains small adapter layers, dramatically reducing memory and computational requirements.
In Linux environments, LoRA is commonly used with frameworks like transformers, peft (Parameter-Efficient Fine-Tuning), and llama.cpp to customize large language models and other neural networks on modest hardware. For example, you might fine-tune a 7B parameter model on a single GPU by adding just 1-2% trainable parameters via LoRA adapters.
The technique produces separate adapter weights that can be merged with base models or swapped between different model instances, making it practical for deployment and experimentation in resource-constrained Linux systems.