DoRA (Weight-Decomposed Low-Rank Adaptation) is an advanced parameter-efficient fine-tuning method developed by NVIDIA Research and published at ICML 2024 as an oral paper (top 1.5% acceptance rate). It builds directly on LoRA but closes the accuracy gap between LoRA and full fine-tuning by decomposing pre-trained weights into two distinct components: magnitude and direction. LoRA then operates only on the directional component, while the magnitude is trained separately as a simple learnable vector.
The motivation comes from analyzing how full fine-tuning actually changes model weights. The authors observed that full fine-tuning tends to make large directional changes with relatively small magnitude changes — a pattern that standard LoRA cannot efficiently replicate because it entangles both magnitude and direction in a single low-rank update. By decomposing the problem, DoRA allows each component to be learned through the most appropriate mechanism.
In benchmarks across LLaMA-7B, LLaMA-13B, LLaMA2-7B, and LLaMA3-8B, DoRA consistently outperforms LoRA by 1.0–4.4% on commonsense reasoning tasks. Remarkably, a variant called DoRA† that uses half the rank of standard LoRA still outperforms it — achieving better results with fewer trainable parameters. DoRA introduces no additional inference latency, as the magnitude and directional components can be merged back into the base weights at deployment time.
Every weight vector in a neural network can be mathematically decomposed into two components: its magnitude (how large it is) and its direction (which way it points in high-dimensional space). This is analogous to polar coordinates — instead of representing a point as (x, y), you represent it as (distance, angle).
Any weight matrix W can be decomposed as W = m · (V / ||V||_c), where m is a magnitude vector (one scalar per output dimension) and V / ||V||_c is the directional matrix (each column normalized to unit length). This decomposition separates “how strongly” a neuron fires from “what pattern” it responds to.
The DoRA authors conducted a detailed analysis comparing how full fine-tuning and LoRA modify pre-trained weights. They found a striking pattern:
This mismatch explains why LoRA often underperforms full fine-tuning, especially at lower ranks. The low-rank constraint forces LoRA to compromise between magnitude and directional adaptation, doing neither optimally.
DoRA addresses this by separating the two concerns:
Step 1: Decompose pre-trained weight W₀
W₀ = m₀ · (V₀ / ||V₀||_c)
where:
m₀ = ||W₀||_c (column-wise norms → magnitude vector)
V₀ = W₀ (the weight matrix itself serves as V₀)
Step 2: Apply LoRA to the directional component only
V' = V₀ + BA (standard LoRA update to direction)
Step 3: Train magnitude separately
m = m₀ + Δm (Δm is a learnable vector, one value per column)
Step 4: Combine for the final weight
W' = m · (V' / ||V'||_c)
Trainable parameters:
- LoRA matrices B ∈ ℝ^(d×r), A ∈ ℝ^(r×k) (same as standard LoRA)
- Magnitude vector Δm ∈ ℝ^k (tiny — just k scalars)The magnitude vector Δm adds negligible parameters (e.g., 4096 scalars for a 4096-dimension layer) compared to the LoRA matrices, but it gives the model a dedicated mechanism for magnitude adjustment that doesn’t interfere with directional learning.
DoRA adds only one trainable vector per adapted layer compared to standard LoRA. For a typical transformer layer with d = 4096:
DoRA requires computing column-wise norms of V' = V₀ + BA at each forward pass for the normalization step. This adds a small computational overhead during training (roughly 5–15% more training time compared to LoRA, depending on implementation). However, this overhead disappears at inference time because the final weights can be precomputed and merged.
from peft import LoraConfig, get_peft_model
# DoRA is a simple flag in the LoRA config
config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
lora_dropout=0.05,
use_dora=True, # Enable DoRA decomposition
bias="none",
task_type="CAUSAL_LM"
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# Slightly more parameters than LoRA due to magnitude vectorsDoRA’s primary value proposition is achieving near-full-fine-tuning quality with parameter-efficient methods. On the LLaMA-7B commonsense reasoning benchmark, DoRA scored 83.7% — a +3.7% improvement over LoRA (80.0%) and approaching full fine-tuning performance. On LLaMA3-8B, the gap was even larger: +4.4%. For applications where LoRA’s accuracy shortfall is unacceptable but full fine-tuning is impractical, DoRA is the natural solution.
The DoRA† variant (using half the rank of the LoRA baseline) consistently outperforms standard LoRA. This has practical implications: if your GPU memory is constrained and you need to use a lower rank, switching to DoRA lets you maintain or even improve accuracy. DoRA at rank 8 beats LoRA at rank 16, while using roughly half the adapter parameters.
DoRA was validated not just on language models but also on vision transformers (ViT) and multimodal models (LLaVA). For image classification with ViT-B/16, DoRA improved accuracy from 87.6% (LoRA) to 88.3%. For visual question answering with LLaVA-1.5-7B, DoRA achieved 82.0% vs 80.1% for LoRA on ScienceQA. This broad applicability makes DoRA a general-purpose upgrade over LoRA across modalities.
DoRA is fully compatible with QLoRA — you can load the base model in 4-bit NF4 and apply DoRA adapters in higher precision. This combination (sometimes called QDoRA) gives you the memory efficiency of QLoRA with the improved learning dynamics of DoRA, achieving the best of both approaches.
use_dora=True in the LoRA config and works across language, vision, and multimodal models.