Multi-Modal Embeddings (Text + Image)

Overview

Multi-modal embeddings map inputs from different modalities—text, images, audio, video—into a shared vector space where semantic similarity can be measured across modality boundaries. A text query like “sunset over the ocean” should produce a vector close to an image of an ocean sunset, enabling cross-modal retrieval: searching images with text, finding text descriptions that match a photo, or clustering mixed-media content by meaning rather than format.

The search platform in a production AI platform leverages Google’s multimodalembedding model via Vertex AI to generate unified vector representations from both text and image inputs. This allows the system to index a heterogeneous corpus—documentation pages, screenshots, diagrams, photos—into a single vector index and retrieve relevant results regardless of the query or document modality.

Building a production multi-modal embedding pipeline introduces challenges beyond text-only systems: images require preprocessing (resizing, format conversion, byte encoding), the embedding space geometry differs from text-only models (cosine similarity distributions shift), and latency profiles change dramatically when processing image payloads. Understanding these differences is critical for building reliable cross-modal search.

Core Concepts

How It Works

The search platform processes multi-modal content through a pipeline that normalizes inputs, calls the Vertex AI embedding endpoint, and stores the resulting vectors alongside metadata in the vector index. Text and image inputs follow different preprocessing paths but converge at the embedding API call.

Embedding Text and Images via Vertex AI

package multimodal

import (
    "context"
    "encoding/base64"
    "fmt"
    "os"

    aiplatform "cloud.google.com/go/aiplatform/apiv1"
    "cloud.google.com/go/aiplatform/apiv1/aiplatformpb"
    "google.golang.org/protobuf/types/known/structpb"
)

// MultiModalEmbedder generates embeddings for text and images using Vertex AI.
type MultiModalEmbedder struct {
    client    *aiplatform.PredictionClient
    endpoint  string // projects/{proj}/locations/{loc}/publishers/google/models/multimodalembedding@001
    dimension int
}

// EmbedText generates an embedding vector for a text input.
func (m *MultiModalEmbedder) EmbedText(ctx context.Context, text string) ([]float32, error) {
    instance, _ := structpb.NewValue(map[string]any{
        "text": text,
    })

    params, _ := structpb.NewValue(map[string]any{
        "dimension": m.dimension,
    })

    resp, err := m.client.Predict(ctx, &aiplatformpb.PredictRequest{
        Endpoint:   m.endpoint,
        Instances:  []*structpb.Value{instance},
        Parameters: params,
    })
    if err != nil {
        return nil, fmt.Errorf("vertex predict (text): %w", err)
    }

    return extractEmbedding(resp.Predictions[0], "textEmbedding")
}

// EmbedImage generates an embedding vector for an image file.
func (m *MultiModalEmbedder) EmbedImage(ctx context.Context, imagePath string) ([]float32, error) {
    imageBytes, err := os.ReadFile(imagePath)
    if err != nil {
        return nil, fmt.Errorf("reading image: %w", err)
    }

    encoded := base64.StdEncoding.EncodeToString(imageBytes)

    instance, _ := structpb.NewValue(map[string]any{
        "image": map[string]any{
            "bytesBase64Encoded": encoded,
        },
    })

    params, _ := structpb.NewValue(map[string]any{
        "dimension": m.dimension,
    })

    resp, err := m.client.Predict(ctx, &aiplatformpb.PredictRequest{
        Endpoint:   m.endpoint,
        Instances:  []*structpb.Value{instance},
        Parameters: params,
    })
    if err != nil {
        return nil, fmt.Errorf("vertex predict (image): %w", err)
    }

    return extractEmbedding(resp.Predictions[0], "imageEmbedding")
}

// extractEmbedding pulls the float32 vector from the prediction response.
func extractEmbedding(prediction *structpb.Value, key string) ([]float32, error) {
    fields := prediction.GetStructValue().GetFields()
    embeddingVal, ok := fields[key]
    if !ok {
        return nil, fmt.Errorf("missing %s in response", key)
    }
    values := embeddingVal.GetListValue().GetValues()
    vec := make([]float32, len(values))
    for i, v := range values {
        vec[i] = float32(v.GetNumberValue())
    }
    return vec, nil
}

Cross-Modal Search Pipeline

package multimodal

import (
    "context"
    "fmt"
    "sort"
)

// SearchResult holds a matched document with its similarity score.
type SearchResult struct {
    ID       string
    Score    float32
    Modality string // "text" or "image"
    Content  string // text content or image path
}

// CrossModalSearch queries a vector index using either text or image input.
func CrossModalSearch(ctx context.Context, embedder *MultiModalEmbedder, index VectorIndex, query string, topK int) ([]SearchResult, error) {
    // Embed the text query
    queryVec, err := embedder.EmbedText(ctx, query)
    if err != nil {
        return nil, fmt.Errorf("embedding query: %w", err)
    }

    // Search the unified index (contains both text and image embeddings)
    matches, err := index.NearestNeighbors(ctx, queryVec, topK)
    if err != nil {
        return nil, fmt.Errorf("nearest neighbor search: %w", err)
    }

    results := make([]SearchResult, len(matches))
    for i, match := range matches {
        results[i] = SearchResult{
            ID:       match.ID,
            Score:    match.Score,
            Modality: match.Metadata["modality"].(string),
            Content:  match.Metadata["content"].(string),
        }
    }

    sort.Slice(results, func(i, j int) bool {
        return results[i].Score > results[j].Score
    })

    return results, nil
}

Practical Applications

The search platform uses multi-modal embeddings to power visual search across a documentation and asset library. Users can type a query like “architecture diagram showing the ingestion pipeline” and retrieve both text descriptions of the pipeline and actual architecture diagrams, ranked by semantic relevance in a single result list. This eliminates the need for separate text search and image search interfaces.

The system also supports reverse image search: uploading a screenshot or diagram and finding related documentation. When a developer encounters an unfamiliar system diagram, they can embed the image and find text documents that describe the depicted architecture. This cross-modal capability is particularly valuable for onboarding, where new team members encounter visual artifacts without textual context.

Multi-modal embeddings also enable automated content organization. By embedding all assets (documentation pages, images, diagrams, code snippets) into a shared space, the system can automatically cluster related content, detect near-duplicate images, and suggest missing documentation for visual assets that lack text descriptions.

Common Pitfalls

• Modality gap: Even well-trained multi-modal models exhibit a “modality gap”—text embeddings and image embeddings occupy slightly different regions of the shared space. This means text-to-text similarity scores are not directly comparable to text-to-image scores. Normalize scores per-modality or use separate thresholds when mixing modalities in ranked results.
• Image preprocessing inconsistency: If your client-side image preprocessing (resizing, cropping, color space conversion) differs from what the model expects, embedding quality degrades silently. Rely on the API’s built-in preprocessing when possible, and validate that your preprocessing pipeline produces consistent results.
• Ignoring dimension reduction trade-offs: Lower-dimensional embeddings (128 or 256) reduce storage and speed up search but sacrifice fine-grained discrimination. For cross-modal retrieval where subtle semantic differences matter, prefer the full 1408-dimension output. Only reduce dimensions if you have empirically validated that recall remains acceptable.
• Rate limiting on image payloads: Image embedding requests carry significantly more data than text requests. Vertex AI rate limits apply per-request and per-byte. Batch image embedding requests carefully and implement exponential backoff. A burst of 100 image embeddings will likely hit rate limits that 100 text embeddings would not.
• Mixing embedding model versions: If you index documents with multimodalembedding@001 and later switch to a newer version, existing vectors become incompatible. The shared embedding space is specific to each model version. Reindex the entire corpus when upgrading models.
• Not validating cross-modal quality: Standard embedding evaluation metrics (recall@k on text-to-text) do not capture cross-modal quality. Build evaluation sets with known text-image pairs and measure cross-modal recall separately. A model may excel at text-to-text but perform poorly on text-to-image for your specific domain.