LoRA freezes the pretrained weights and learns a low-rank update to selected weight matrices (often attention projections): the update is the product of two small matrices with rank r (commonly 8 to 64), scaled by an alpha factor.
Only these small matrices are trained, so memory and compute drop sharply; after training they can be merged into the base weights or kept as separate adapters.
QLoRA keeps the frozen base model quantized to 4-bit while training the LoRA adapters in higher precision, letting large models be fine-tuned on a single GPU.
Higher rank gives more capacity but more parameters and overfitting risk.
Loading…