PiSSA, short for Principal Singular values and Singular vectors Adaptation, is a parameter-efficient fine-tuning method that keeps the familiar LoRA adapter interface but changes what the adapter represents at initialization time. Standard LoRA freezes the pretrained weight matrix and learns a low-rank update from a randomly initialized factor paired with a zero factor. PiSSA instead decomposes the pretrained weight matrix with singular value decomposition, places the leading singular components into the trainable adapter, and freezes the residual matrix.
The practical goal is faster and stronger adaptation without giving up the deployment properties that make LoRA attractive. The trainable parameter count is still controlled by rank, the adapter can be handled by the Hugging Face PEFT stack, and the method composes with quantized fine-tuning. The difference is that the trainable low-rank branch starts from the model’s highest-energy directions rather than from noise.
The PiSSA paper reports consistent improvements over LoRA across models from 184M to 70B parameters and across natural-language generation and understanding tasks. A representative result is Mistral-7B on GSM8K: PiSSA reaches 72.86% accuracy under the paper’s setup, compared with 67.7% for LoRA. The same idea also helps in 4-bit settings: QPiSSA reduces early quantization error compared with QLoRA and reports 86.05% accuracy for LLaMA-3-70B on GSM8K, compared with 81.73% for QLoRA.
LoRA adapts a frozen weight matrix W by adding a trainable low-rank delta BA during the forward pass. If W maps from d_in to d_out, the two adapter matrices use rank r, where r << min(d_in, d_out). This makes fine-tuning cheap because only the adapter matrices receive gradients.
Singular value decomposition factors a matrix into left singular vectors, singular values, and right singular vectors. The leading singular components capture the largest directions of variation in the pretrained matrix. A rank-r truncated SVD can be read as the strongest low-rank approximation of the original matrix under common matrix norms.
PiSSA’s central claim is that the most useful place to spend low-rank trainable capacity is not a random direction but the principal structure already present in the model. It initializes the adapter from the leading singular triplets of W and moves the remaining lower-energy components into a frozen residual matrix. Fine-tuning then updates the important low-rank component directly.
PiSSA does not require a new inference-time architecture. It uses LoRA-shaped factors, but initializes them from the pretrained matrix’s principal singular vectors and singular values. That lets existing LoRA infrastructure train, save, convert, and serve the adapter with relatively small changes.
For each targeted linear weight, PiSSA computes a rank-r decomposition of the pretrained matrix. Conceptually, the matrix is split into two pieces: a principal low-rank component that becomes trainable through the adapter, and a residual component that remains frozen in the base model path.
import torch
def pissa_split(weight, rank):
# weight shape: [out_features, in_features]
# Full SVD is shown for clarity; production code should use fast or randomized SVD.
u, s, vh = torch.linalg.svd(weight.float(), full_matrices=False)
u_r = u[:, :rank]
s_r = s[:rank]
vh_r = vh[:rank, :]
principal = (u_r * s_r) @ vh_r
residual = weight.float() - principal
# LoRA-shaped factors. Frameworks may choose the equivalent transposed layout.
lora_B = u_r * torch.sqrt(s_r)
lora_A = torch.sqrt(s_r).unsqueeze(1) * vh_r
return residual.to(weight.dtype), lora_A.to(weight.dtype), lora_B.to(weight.dtype)The exact tensor layout depends on the adapter implementation, but the invariant is simple: the adapter’s initial product reconstructs the rank-r principal component, while the frozen residual carries the rest of the original matrix. Before any fine-tuning step, the residual plus adapter path should preserve the pretrained model’s behavior up to decomposition and numerical error.
During fine-tuning, gradients flow into the PiSSA adapter factors rather than the residual matrix. The optimizer is therefore allowed to reshape the dominant singular directions of the layer for the downstream task. This is the opposite allocation from ordinary LoRA, where the full pretrained matrix is frozen and the trainable branch begins as a small independent update.
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-hf",
torch_dtype="auto",
device_map="auto",
)
config = LoraConfig(
init_lora_weights="pissa_niter_4",
r=128,
lora_alpha=128,
lora_dropout=0,
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
model.print_trainable_parameters()A PiSSA adapter can be converted into a LoRA-style delta against the original base model by subtracting the initial adapter product from the fine-tuned adapter product. In matrix terms, the deployable update is A B - A0 B0 under the implementation’s factor ordering. This conversion allows serving systems that expect ordinary LoRA adapters to use the result without carrying a separate residual-model artifact.
PiSSA is useful when LoRA converges slowly or requires more steps than a training budget allows. Because the adapter starts on high-energy model directions, early optimization can change behavior more directly. This matters for reasoning, math, code, and domain adaptation workloads where every epoch over curated data is expensive.
QPiSSA applies the same idea in 4-bit fine-tuning pipelines. The paper and project README emphasize smaller quantization error during initialization compared with QLoRA-style baselines. For teams adapting 70B-class models, that can be the difference between a quantized run that tracks full-precision quality and one that spends much of training recovering from quantization damage.
Computing SVD for every training run is wasteful. The PiSSA workflow supports pre-decomposed residual models plus adapter initialization artifacts. A team can generate or download these once for a base model and rank, then reuse the same starting point across many datasets and experiments.
Full SVD over every target matrix is easy to describe but expensive on large models. PiSSA implementations expose fast SVD options such as pissa_niter_4, where the iteration count controls the tradeoff between initialization quality and runtime. Use exact SVD for small experiments, but prefer fast SVD for production-scale models and repeated runs.
The PiSSA README recommends setting LoRA dropout to zero because the adapter factors are no longer random disposable capacity. They represent the principal singular component of the pretrained matrix. Randomly dropping that branch during training changes the meaning of the decomposition and can undercut the intended initialization.
If the serving target expects standard LoRA, preserve the initial PiSSA adapter state. Conversion needs both the fine-tuned factors and the initial factors so it can compute the adapter delta relative to the original base model.
# Sketch of conversion bookkeeping.
# Save the initialized PiSSA adapter before training.
peft_model.save_pretrained("run/pissa_init")
# ...train the adapter...
trainer.train()
# Save a LoRA-compatible adapter by subtracting the initial PiSSA branch.
peft_model.save_pretrained(
"run/pissa_as_lora",
path_initial_model_for_weight_conversion="run/pissa_init",
)