The Client-Side CLIP Visual Search Bridge is an advanced search pattern that enables users to find visually similar documents by uploading a query image. Rather than sending the query image to a dedicated vector search backend, this approach generates a CLIP embedding for the query image on the client side, then scans through stored clip_embedding vectors directly, computing cosine similarity without relying on the Approximate K-Nearest Neighbor (AKNN) index.
This architecture is particularly useful in scenarios where the AKNN index is unavailable, not yet provisioned, or where the corpus is small enough that brute-force similarity search is acceptable. By performing the comparison client-side, the system avoids round-trips to a vector database and can provide results even when the backend search infrastructure is partially offline.
A key design choice is the fusion of visual and textual search results. After identifying visually similar documents via CLIP cosine similarity, the bridge augments these results with OCR-derived semantic search hits. If the query image contains text (e.g., a screenshot of code, a scanned document), the OCR text is extracted and used as a secondary text query against the standard search index. The source image is always excluded from results to avoid trivial self-matches.
The query image is passed through a CLIP model (typically loaded via ONNX Runtime or TensorFlow.js in the browser, or via a lightweight local API) to produce a fixed-dimensional embedding vector. This vector represents the semantic content of the image in the same latent space as the stored document embeddings, enabling direct comparison without server-side inference.
Instead of querying an AKNN index (like HNSW or IVF), the bridge iterates over all stored clip_embedding vectors and computes cosine similarity against the query embedding. This O(n) scan is feasible for corpora under ~100K documents where each comparison is a simple dot product on normalized vectors. The results are sorted by similarity score and the top-K are returned.
When a user searches using an image that already exists in the corpus, the system must exclude that exact document from the results. The bridge identifies the source file by its document ID or content hash and filters it out before ranking, preventing the trivial “best match is itself” result that would otherwise dominate.
For images containing text (screenshots, scanned pages, whiteboard photos), the bridge extracts OCR text from the query image and runs a parallel text search against the standard semantic index. The text-based results are merged with the visual similarity results using a weighted score fusion, improving recall for documents that are textually related but visually different from the query.
The decision to bypass AKNN is not just a fallback—it’s a deliberate architectural choice for certain use cases. AKNN indices trade recall for speed; brute-force search guarantees perfect recall. For applications where finding the true nearest neighbors matters more than latency (e.g., deduplication, compliance checks), the bypass produces more accurate results.
The visual search flow has four stages: embedding generation, similarity scan, source exclusion, and OCR augmentation. Each stage is designed to be independent so that OCR augmentation can be disabled without affecting the core visual search path.
import numpy as np
from typing import List, Tuple, Optional
def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
"""Compute cosine similarity between two vectors."""
dot = np.dot(a, b)
norm_a = np.linalg.norm(a)
norm_b = np.linalg.norm(b)
if norm_a == 0 or norm_b == 0:
return 0.0
return float(dot / (norm_a * norm_b))
def visual_search(
query_embedding: np.ndarray,
stored_embeddings: List[Tuple[str, np.ndarray]],
source_doc_id: Optional[str] = None,
top_k: int = 10,
) -> List[Tuple[str, float]]:
"""
Brute-force visual search over stored CLIP embeddings.
Args:
query_embedding: CLIP vector for the query image.
stored_embeddings: List of (doc_id, embedding) pairs.
source_doc_id: Document ID to exclude (the query image's source).
top_k: Number of results to return.
Returns:
List of (doc_id, similarity_score) sorted by descending similarity.
"""
scores = []
for doc_id, emb in stored_embeddings:
if doc_id == source_doc_id:
continue
sim = cosine_similarity(query_embedding, emb)
scores.append((doc_id, sim))
scores.sort(key=lambda x: x[1], reverse=True)
return scores[:top_k]from dataclasses import dataclass
from typing import Dict
@dataclass
class SearchResult:
doc_id: str
visual_score: float
text_score: float
combined_score: float
def hybrid_visual_search(
query_embedding: np.ndarray,
stored_embeddings: List[Tuple[str, np.ndarray]],
ocr_text: Optional[str],
text_search_fn,
source_doc_id: Optional[str] = None,
top_k: int = 10,
visual_weight: float = 0.7,
text_weight: float = 0.3,
) -> List[SearchResult]:
"""
Combine visual similarity with OCR-based text search.
If the query image contains text, OCR results augment visual
matches to improve recall for textually-related documents.
"""
# Stage 1: Visual similarity scan
visual_results = visual_search(
query_embedding, stored_embeddings,
source_doc_id=source_doc_id, top_k=top_k * 3
)
visual_map: Dict[str, float] = {
doc_id: score for doc_id, score in visual_results
}
# Stage 2: Text search using OCR content (if available)
text_map: Dict[str, float] = {}
if ocr_text and len(ocr_text.strip()) > 10:
text_results = text_search_fn(ocr_text, top_k=top_k * 3)
text_map = {
doc_id: score for doc_id, score in text_results
if doc_id != source_doc_id
}
# Stage 3: Score fusion
all_doc_ids = set(visual_map.keys()) | set(text_map.keys())
fused = []
for doc_id in all_doc_ids:
v_score = visual_map.get(doc_id, 0.0)
t_score = text_map.get(doc_id, 0.0)
combined = (visual_weight * v_score) + (text_weight * t_score)
fused.append(SearchResult(
doc_id=doc_id,
visual_score=v_score,
text_score=t_score,
combined_score=combined,
))
fused.sort(key=lambda r: r.combined_score, reverse=True)
return fused[:top_k]import asyncio
from typing import Any, Dict, List
class VisualSearchBridge:
"""
Orchestrates the client-side visual search flow:
1. Generate CLIP embedding for query image
2. Load stored embeddings from the search index
3. Run hybrid search with OCR augmentation
"""
def __init__(self, clip_client, search_client, ocr_client=None):
self.clip = clip_client
self.search = search_client
self.ocr = ocr_client
async def search_by_image(
self, image_bytes: bytes,
source_doc_id: str = None,
top_k: int = 10,
) -> List[Dict[str, Any]]:
# Generate query embedding
query_emb = await self.clip.embed_image(image_bytes)
query_vec = np.array(query_emb)
# Load stored embeddings from index
stored = await self.search.get_all_clip_embeddings()
stored_pairs = [
(doc["id"], np.array(doc["clip_embedding"]))
for doc in stored
if doc.get("clip_embedding") is not None
]
# Extract OCR text if client is available
ocr_text = None
if self.ocr:
ocr_text = await self.ocr.extract_text(image_bytes)
# Run hybrid search
results = hybrid_visual_search(
query_embedding=query_vec,
stored_embeddings=stored_pairs,
ocr_text=ocr_text,
text_search_fn=self.search.semantic_search,
source_doc_id=source_doc_id,
top_k=top_k,
)
# Hydrate results with document metadata
doc_ids = [r.doc_id for r in results]
docs = await self.search.get_documents(doc_ids)
doc_map = {d["id"]: d for d in docs}
return [
{
"document": doc_map.get(r.doc_id, {}),
"visual_score": r.visual_score,
"text_score": r.text_score,
"combined_score": r.combined_score,
}
for r in results
if r.doc_id in doc_map
]In the search platform, the visual search bridge powers the “Find Similar” feature. When a user clicks on an image in search results, the UI sends the image to the bridge which generates a CLIP embedding and scans the corpus for visually similar documents. Because the scan is brute-force, the results have perfect recall—every similar document is found, not just those that happen to be near the query in an approximate index.
The OCR augmentation is critical for the code screenshot use case. Developers frequently search for error messages by uploading a screenshot of their terminal. The bridge extracts the error text via OCR and runs it as a parallel text query, finding documentation and past incidents that mention the same error string even if the visual layout differs completely.
For deduplication workflows, the bridge identifies near-duplicate images across the corpus. Setting a high similarity threshold (e.g., 0.95) and scanning the entire embedding space catches duplicates that AKNN might miss due to its approximate nature. This is used during batch ingestion to flag potential duplicates before committing them to the index.
Loading all stored embeddings on every query is expensive. The bridge caches the embedding matrix in memory and refreshes it on a timer or when the index version changes. For corpora under 50K documents, the cache fits in a few hundred MB (50K × 768 floats × 4 bytes = ~150MB). The cache is stored as a contiguous NumPy array for vectorized cosine similarity computation.
Rather than computing cosine similarity one pair at a time, the bridge normalizes all stored embeddings to unit vectors at cache-warm time and then computes similarity as a single matrix-vector multiply: scores = embeddings @ query_vec. This reduces the O(n) loop to a single BLAS call, which is orders of magnitude faster for large corpora.
For large corpora where even the vectorized scan takes noticeable time, the bridge can stream top-K results incrementally. A min-heap of size K is maintained during the scan; each time a new candidate exceeds the heap’s minimum, it replaces that entry. The UI receives updates as better matches are found, providing perceived responsiveness.