Visualizing high-dimensional vector embeddings is a fundamental debugging and exploration tool for search and AI systems. When you have hundreds of thousands of document vectors in a search index, you need to understand how they cluster, where query vectors land relative to document clusters, and whether your embedding model is producing meaningful spatial relationships. As an example of this approach, consider an interactive scatterplot that renders PCA-reduced embedding vectors as points in a 2D orthographic projection, streaming point data from a Go backend over WebSockets so the visualization updates in real time as new documents are indexed.
The visualization is built on deck.gl, a WebGL-powered framework for rendering large-scale data overlays. Unlike chart libraries designed for hundreds of data points, deck.gl is optimized for rendering millions of points at 60fps by batching geometry into GPU buffers and minimizing per-frame JavaScript work. The scatterplot layer renders each embedding as a colored circle, with color derived from a hash of the document’s source collection, size indicating relevance score, and position determined by the first two principal components of the embedding vector.
The system operates in two modes: a “preview” mode that shows a uniformly sampled subset of the index (10,000 points) for quick orientation, and a “full” mode that streams all points over WebSocket in chunked batches. Query vectors are overlaid as larger, distinctly colored markers so users can see where a search query lands in the embedding space relative to the document clusters. This visual feedback helps diagnose problems like embedding drift (where newly indexed documents cluster differently from older ones), query-document mismatch (where queries land far from relevant documents), and index pollution (where junk documents contaminate otherwise clean clusters).
Principal Component Analysis projects high-dimensional vectors (768 or 1536 dimensions for typical embedding models) down to 2 or 3 dimensions while preserving as much variance as possible. The Go backend computes the top-2 principal components from a sample of the index and projects all vectors onto these axes. This is done server-side because the raw embedding vectors are too large to send to the browser — a 100K-document index with 768-dim embeddings would be ~300MB of float data, but the 2D projections are only ~1.6MB.
Deck.gl’s default view is a map-style Mercator projection designed for geospatial data. For embedding visualizations, you use an OrthographicView instead — a flat 2D coordinate system with no perspective distortion, where the x/y axes correspond directly to PCA component values. This makes spatial distances meaningful: if two points are twice as far apart on screen, their PCA projections are twice as far apart in the reduced space. Pan and zoom operate on the orthographic coordinates rather than lat/lng.
Sending 100K+ points in a single HTTP response would cause a multi-second loading delay and a memory spike on the client. Instead, the Go backend streams point batches over a WebSocket connection — typically 1,000 points per message. The frontend accumulates batches into a growing buffer and triggers a deck.gl layer update on each batch. This produces a progressive rendering effect where the scatterplot fills in over 2-3 seconds, giving the user visual feedback immediately rather than waiting for all data to arrive.
Each document in the index belongs to a source collection (a corpus, a crawl batch, or a manual upload). The visualization assigns each collection a deterministic color by hashing the collection name to HSL values — specifically, the hash determines the hue (0-360), while saturation and lightness are fixed to ensure all colors are visually distinguishable against the dark background. This means the same collection always renders with the same color across sessions and users, making the visualization a reliable communication tool.
The preview mode sends a uniformly random sample of 10,000 points in a single WebSocket message — fast enough to render instantly and sufficient to show the overall cluster structure. Full mode streams all points, which can take several seconds for large indices. The UI starts in preview mode and offers a toggle to “Load All Points.” Switching modes sends a control message over the same WebSocket connection, and the backend begins streaming full data while the frontend clears and rebuilds its point buffer.
The frontend maintains a single WebSocket connection to the Go backend. Messages are typed JSON objects — either point batches or control messages (mode switches, query vector overlays).
interface VectorPoint {
x: number; // PCA component 1
y: number; // PCA component 2
id: string; // document ID
collection: string; // source collection name
score?: number; // relevance score (if from a query)
}
interface WSMessage {
type: 'points' | 'query_overlay' | 'clear' | 'stats';
data: VectorPoint[] | QueryOverlay | IndexStats;
}
function useVectorStream(wsUrl: string) {
const [points, setPoints] = React.useState<VectorPoint[]>([]);
const [queryPoints, setQueryPoints] = React.useState<VectorPoint[]>([]);
const bufferRef = React.useRef<VectorPoint[]>([]);
const wsRef = React.useRef<WebSocket | null>(null);
const rafRef = React.useRef<number | null>(null);
const flush = React.useCallback(() => {
if (bufferRef.current.length > 0) {
const batch = bufferRef.current;
bufferRef.current = [];
setPoints(prev => {
const next = new Array(prev.length + batch.length);
for (let i = 0; i < prev.length; i++) next[i] = prev[i];
for (let i = 0; i < batch.length; i++) next[prev.length + i] = batch[i];
return next;
});
}
rafRef.current = null;
}, []);
React.useEffect(() => {
const ws = new WebSocket(wsUrl);
wsRef.current = ws;
ws.onmessage = (event) => {
const msg: WSMessage = JSON.parse(event.data);
switch (msg.type) {
case 'points':
bufferRef.current.push(...(msg.data as VectorPoint[]));
if (!rafRef.current) {
rafRef.current = requestAnimationFrame(flush);
}
break;
case 'query_overlay':
setQueryPoints((msg.data as QueryOverlay).points);
break;
case 'clear':
bufferRef.current = [];
setPoints([]);
setQueryPoints([]);
break;
}
};
return () => {
ws.close();
if (rafRef.current) cancelAnimationFrame(rafRef.current);
};
}, [wsUrl, flush]);
const sendQuery = React.useCallback((query: string) => {
wsRef.current?.send(JSON.stringify({ type: 'query', query }));
}, []);
const setMode = React.useCallback((mode: 'preview' | 'full') => {
wsRef.current?.send(JSON.stringify({ type: 'mode', mode }));
}, []);
return { points, queryPoints, sendQuery, setMode };
}The deck.gl DeckGL component renders layers onto a WebGL canvas. The scatterplot uses two layers: one for document points and one for query overlay points with a different visual treatment.
import DeckGL from '@deck.gl/react';
import { OrthographicView } from '@deck.gl/core';
import { ScatterplotLayer } from '@deck.gl/layers';
function collectionToColor(collection: string): [number, number, number] {
let hash = 0;
for (let i = 0; i < collection.length; i++) {
hash = ((hash << 5) - hash + collection.charCodeAt(i)) | 0;
}
const hue = Math.abs(hash) % 360;
// Convert HSL(hue, 70%, 55%) to RGB
return hslToRgb(hue, 0.7, 0.55);
}
function VectorScatterplot({ points, queryPoints }: {
points: VectorPoint[];
queryPoints: VectorPoint[];
}) {
const layers = [
new ScatterplotLayer({
id: 'documents',
data: points,
getPosition: (d: VectorPoint) => [d.x, d.y],
getFillColor: (d: VectorPoint) => collectionToColor(d.collection),
getRadius: 3,
opacity: 0.6,
radiusMinPixels: 1,
radiusMaxPixels: 8,
// Enable GPU aggregation for large point counts
parameters: { depthTest: false },
updateTriggers: {
getPosition: points.length,
getFillColor: points.length,
},
}),
new ScatterplotLayer({
id: 'queries',
data: queryPoints,
getPosition: (d: VectorPoint) => [d.x, d.y],
getFillColor: [255, 107, 138], // --danger color
getRadius: 8,
opacity: 0.9,
radiusMinPixels: 4,
radiusMaxPixels: 16,
stroked: true,
getLineColor: [255, 255, 255],
getLineWidth: 2,
parameters: { depthTest: false },
}),
];
return (
<DeckGL
views={new OrthographicView({ flipY: false })}
initialViewState={{
target: [0, 0],
zoom: 1,
minZoom: -2,
maxZoom: 10,
}}
controller={true}
layers={layers}
style={{ background: '#0f1117' }}
/>
);
}The Go backend computes PCA on a sample of vectors and streams the projected 2D coordinates to connected clients.
// Server-side PCA projection and WebSocket streaming (Go)
func (s *VizServer) handleWS(w http.ResponseWriter, r *http.Request) {
conn, err := s.upgrader.Upgrade(w, r, nil)
if err != nil {
return
}
defer conn.Close()
// Send preview (sampled 10K points) immediately
sample := s.index.UniformSample(10000)
projected := s.pca.Project(sample)
batch := make([]VectorPoint, len(projected))
for i, p := range projected {
batch[i] = VectorPoint{
X: p[0],
Y: p[1],
ID: sample[i].ID,
Collection: sample[i].Collection,
}
}
conn.WriteJSON(WSMessage{Type: "points", Data: batch})
// Listen for control messages
for {
var msg ClientMessage
if err := conn.ReadJSON(&msg); err != nil {
return
}
switch msg.Type {
case "mode":
if msg.Mode == "full" {
s.streamFullIndex(conn)
}
case "query":
s.overlayQuery(conn, msg.Query)
}
}
}
func (s *VizServer) streamFullIndex(conn *websocket.Conn) {
conn.WriteJSON(WSMessage{Type: "clear"})
const batchSize = 1000
allVectors := s.index.AllVectors()
for i := 0; i < len(allVectors); i += batchSize {
end := min(i+batchSize, len(allVectors))
chunk := allVectors[i:end]
projected := s.pca.Project(chunk)
batch := make([]VectorPoint, len(projected))
for j, p := range projected {
batch[j] = VectorPoint{
X: p[0],
Y: p[1],
ID: chunk[j].ID,
Collection: chunk[j].Collection,
}
}
if err := conn.WriteJSON(WSMessage{Type: "points", Data: batch}); err != nil {
return
}
}
}This type of embedding scatterplot is used during index development to visually verify that embedding quality is consistent across document collections. When a new crawl batch is indexed, engineers open the visualization and check whether the new points (shown in the batch’s deterministic color) cluster near related existing documents or form an isolated island — an isolated island suggests the embedding model is handling the new content differently, which could indicate a domain shift or preprocessing problem.
The query overlay feature is used to debug search quality issues. When a user reports that a query returns irrelevant results, an engineer enters the query in the visualization and observes where the query vector lands. If it lands far from the expected document cluster, the problem is in the embedding model or query preprocessing. If it lands near the right cluster but the wrong documents are returned, the problem is in the retrieval ranking algorithm.
The preview-to-full mode switch was designed for production indices with 500K+ documents. Loading all points takes 3-5 seconds over WebSocket, during which the scatterplot progressively fills in. The preview mode gives engineers immediate orientation — they can identify which cluster to investigate, then switch to full mode to see the fine-grained structure within that cluster. Pan and zoom work during streaming, so engineers can navigate to the region of interest while data is still loading.
WebSocket messages arrive faster than React should re-render. The buffer-and-flush pattern using requestAnimationFrame accumulates incoming point batches in a mutable ref and flushes to React state at most once per frame. This ensures the deck.gl layer updates at 60fps regardless of WebSocket message rate.
When accumulating points, naive [...prev, ...batch] creates a new array and copies all elements on every flush — O(n) per batch on a growing dataset. Pre-allocating with new Array(prev.length + batch.length) and copying with indexed assignment avoids the spread operator’s overhead and reduces GC pressure on large datasets.
Deck.gl layers compare props by reference to decide whether to update GPU buffers. Since the getPosition and getFillColor accessors are the same function reference across renders, deck.gl won’t re-upload data when points change. The updateTriggers prop solves this: setting it to points.length tells deck.gl to re-evaluate the accessors whenever the array grows.
new ScatterplotLayer({...}) inside a render function without memoization, deck.gl creates a new WebGL program and re-uploads all data to the GPU on every React render. Use useMemo or ensure layer IDs are stable so deck.gl can diff props and update incrementally.depthTest: false in layer parameters ensures all points render in the expected order.