Multimodal embedding models map text, images, video frames, or other media into a shared vector space. A general-purpose model can support cross-modal retrieval out of the box, but domain-specific retrieval often depends on visual conventions, product layouts, document templates, chart styles, or vocabulary that the base model has not optimized for.
Fine-tuning adapts the embedding space to those retrieval judgments. The Hugging Face source article demonstrates a Sentence Transformers workflow for adapting a Qwen3-VL embedding model with multimodal training pairs, hard negatives, task-specific evaluation, and Matryoshka Representation Learning so the resulting embeddings can be truncated at deployment time.
This topic focuses on the trainable embedding model, not the vector database. The output is still a bi-encoder that can precompute document vectors and answer nearest-neighbor queries quickly. The difference is that the geometry has been shaped by your domain’s positive and negative examples instead of relying entirely on a generic pretrained space.
A multimodal bi-encoder independently encodes each side of a pair, such as a text query and an image document. Training pulls matching pairs together and pushes mismatches apart. Because documents can be embedded offline, this architecture remains practical for large retrieval collections.
Training data usually contains a query, a relevant multimodal item, and one or more irrelevant items. Random negatives teach broad separation, but hard negatives are more valuable: they look plausible to the base model yet are wrong for the task. In visual document retrieval, hard negatives might share layout, topic, or object categories while failing to answer the query.
Matryoshka training asks the model to perform well not only with the full embedding vector, but also with prefixes of that vector. The result is an embedding where earlier dimensions carry high-value signal. At deployment time, the index can truncate vectors to reduce storage and memory bandwidth while preserving much of the retrieval quality.
Embedding fine-tuning changes which items become neighbors. Evaluation must therefore measure retrieval quality on the target query/document distribution, not just loss curves or generic benchmark scores.
Sentence Transformers can accept multimodal fields such as image paths, PIL images, URLs, and text strings when the underlying model processor supports those modalities. A practical dataset normalizes each example into a query, a relevant document, and optionally mined negatives. Keep metadata with each row so failed retrievals can be inspected after evaluation.
from datasets import Dataset
from sentence_transformers import SentenceTransformer
from sentence_transformers.losses import CachedMultipleNegativesRankingLoss
from sentence_transformers.trainer import SentenceTransformerTrainer
from sentence_transformers.training_args import SentenceTransformerTrainingArguments
model = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")
train = Dataset.from_list([
{
"anchor": {"text": "find the page showing quarterly cloud revenue"},
"positive": {"image": "reports/q3/page_12.png"},
},
{
"anchor": {"text": "diagram of the ingestion architecture"},
"positive": {"image": "docs/architecture/page_03.png"},
},
])
loss = CachedMultipleNegativesRankingLoss(model)
args = SentenceTransformerTrainingArguments(
output_dir="runs/vl-embedding-domain",
per_device_train_batch_size=8,
gradient_accumulation_steps=4,
learning_rate=2e-5,
bf16=True,
)
trainer = SentenceTransformerTrainer(
model=model,
args=args,
train_dataset=train,
loss=loss,
)
trainer.train()Matryoshka training wraps a base retrieval loss and evaluates it at multiple embedding dimensions. The model learns to place useful signal early in the vector, allowing a production service to choose a smaller dimension without retraining.
from sentence_transformers.losses import MatryoshkaLoss
base_loss = CachedMultipleNegativesRankingLoss(model)
loss = MatryoshkaLoss(
model=model,
loss=base_loss,
matryoshka_dims=[2048, 1024, 512, 256, 128, 64],
)
trainer = SentenceTransformerTrainer(
model=model,
args=args,
train_dataset=train,
loss=loss,
)
trainer.train()
model.save_pretrained("vl-embedding-domain", truncate_dim=1024)After training, evaluate with retrieval metrics such as NDCG@10, MRR, and recall@k over a held-out corpus. The source article reports comparing full-size vectors against truncated dimensions; that is the right deployment-oriented test because vector size affects index RAM, disk footprint, cache pressure, and network transfer.
Slides, scanned PDFs, dashboards, forms, and product catalogs often contain meaning in layout and imagery. Fine-tuning a multimodal embedder on real query-page pairs can make first-stage retrieval find pages that OCR-only or generic image embeddings miss.
Retail, design, and media applications often need text-to-image and image-to-image retrieval that reflects domain labels rather than generic visual similarity. Fine-tuning can teach the model that fabric type, room style, product variant, or brand-specific taxonomy matters for ranking.
Multimodal RAG systems need recall before reranking or answer generation can work. A fine-tuned embedding model can supply a stronger candidate set for downstream OCR, visual rerankers, or vision-language answer generation.
Run your existing embedder against training queries, collect high-scoring wrong results, and use them as hard negatives. This aligns training with the actual mistakes your deployed system makes.
def mine_hard_negatives(model, index, labeled_queries, k=20):
rows = []
for item in labeled_queries:
query_vec = model.encode(item["query"])
candidates = index.search(query_vec, top_k=k)
negatives = [
hit.id for hit in candidates
if hit.id not in item["relevant_ids"]
][:5]
rows.append({
"anchor": {"text": item["query"]},
"positive": {"image": item["positive_image"]},
"negative": [{"image": image_path(n)} for n in negatives],
})
return rowsDo not evaluate only on fresh mined negatives from the same run. Maintain a fixed held-out set with stable queries, candidate documents, and relevance labels so changes in training data or model checkpoints can be compared reliably.
If Matryoshka training shows that 1024 dimensions are nearly as strong as 2048 for your task, save the default truncation in the model configuration and record the chosen dimension in index metadata. Query and document vectors must use the same truncation.