When migrating from a managed vector database like Pinecone to a self-hosted solution, you need a way to export every vector along with its metadata. Pinecone does not provide a bulk-export API; the only retrieval mechanism is the query endpoint, which returns vectors ranked by similarity to a query vector. This creates a bootstrapping problem: how do you retrieve all vectors when you need a query vector to start?
The zero-vector query technique solves this by querying with an all-zeros vector and a large top_k value. Because cosine similarity between a zero vector and any non-zero vector is undefined (and Pinecone treats it as returning arbitrary results), this effectively pages through the namespace. Combined with namespace enumeration from the describe_index_stats endpoint, migration scripts can systematically drain every namespace and stream matches—IDs, embedding values, and metadata—into a new system.
This pattern is commonly used when migrating legacy Pinecone spaces into a self-hosted vector storage system, preserving the full embedding geometry and all associated metadata for downstream search and retrieval.
Querying Pinecone with a vector of all zeros and include_values=True, include_metadata=True returns vectors in an implementation-defined order. Since the zero vector has no meaningful direction, the similarity scores are degenerate, but the results still contain the full vector data. By combining this with pagination via ID cursors or repeated queries with exclusion filters, you can iterate through the entire namespace.
Pinecone’s describe_index_stats() endpoint returns per-namespace vector counts and the embedding dimensionality. Migration scripts use this to discover all namespaces in an index, infer the correct vector dimension for constructing the zero vector, and estimate progress by comparing exported counts against the reported totals.
Rather than loading all vectors into memory, export scripts stream matches in batches. Each batch query returns up to top_k results (Pinecone caps this at 10,000). The script tracks seen IDs, applies exclusion filters or cursor-based pagination to avoid duplicates, and writes each batch to the target system incrementally. This keeps memory usage bounded regardless of namespace size.
The zero vector must match the index’s embedding dimensionality exactly. Rather than hardcoding this, migration scripts query describe_index_stats() to read the dimension field dynamically. This makes the export script generic across indexes with different embedding models (e.g., 768-dim for sentence-transformers, 1536-dim for OpenAI text-embedding-ada-002).
The migration pipeline operates in three phases: discovery, export, and verification. Discovery queries Pinecone stats to enumerate namespaces and dimensions. Export iterates through each namespace using zero-vector queries. Verification compares exported counts against the stats to ensure completeness.
import pinecone
from typing import Dict, Tuple
def discover_namespaces(index_name: str) -> Tuple[Dict[str, int], int]:
"""
Discover all namespaces and their vector counts,
plus the embedding dimension for the index.
Returns:
(namespace_counts, dimension) where namespace_counts maps
namespace name to vector count.
"""
index = pinecone.Index(index_name)
stats = index.describe_index_stats()
dimension = stats.dimension
namespace_counts = {}
for ns_name, ns_stats in stats.namespaces.items():
namespace_counts[ns_name] = ns_stats.vector_count
print(f"Index '{index_name}': {len(namespace_counts)} namespaces, "
f"dimension={dimension}")
for ns, count in namespace_counts.items():
print(f" namespace='{ns}': {count} vectors")
return namespace_counts, dimensionimport numpy as np
from typing import List, Dict, Any, Generator
def export_namespace(
index_name: str,
namespace: str,
dimension: int,
batch_size: int = 10000,
) -> Generator[Dict[str, Any], None, None]:
"""
Export all vectors from a Pinecone namespace using zero-vector queries.
Yields dicts with keys: id, values (list[float]), metadata (dict).
Uses ID-based pagination to avoid duplicates.
"""
index = pinecone.Index(index_name)
zero_vector = np.zeros(dimension).tolist()
seen_ids = set()
total_exported = 0
while True:
# Build filter to exclude already-seen IDs
# For large sets, use cursor-based approach with ID ordering
results = index.query(
vector=zero_vector,
top_k=batch_size,
namespace=namespace,
include_values=True,
include_metadata=True,
)
new_matches = []
for match in results.matches:
if match.id not in seen_ids:
seen_ids.add(match.id)
new_matches.append(match)
if not new_matches:
break # No new vectors found, namespace fully exported
for match in new_matches:
yield {
"id": match.id,
"values": match.values,
"metadata": match.metadata or {},
}
total_exported += 1
print(f" Exported {total_exported} vectors from '{namespace}'...")
# If we got fewer new matches than batch_size, we're likely done
if len(new_matches) < batch_size // 2:
break
print(f" Completed: {total_exported} vectors from '{namespace}'")import json
from pathlib import Path
def migrate_pinecone_to_target(
pinecone_index: str,
output_dir: str,
) -> Dict[str, int]:
"""
Export all namespaces from a Pinecone index to JSONL files
that can be bulk-imported into a self-hosted vector database.
Returns a dict mapping namespace to exported vector count.
"""
output_path = Path(output_dir)
output_path.mkdir(parents=True, exist_ok=True)
namespace_counts, dimension = discover_namespaces(pinecone_index)
export_summary = {}
for namespace, expected_count in namespace_counts.items():
safe_name = namespace.replace("/", "_") or "default"
out_file = output_path / f"{safe_name}.jsonl"
count = 0
with open(out_file, "w") as f:
for vector_data in export_namespace(
pinecone_index, namespace, dimension
):
f.write(json.dumps(vector_data) + "\n")
count += 1
export_summary[namespace] = count
# Verification
if count != expected_count:
print(f" WARNING: Expected {expected_count} vectors "
f"but exported {count} for '{namespace}'")
else:
print(f" Verified: {count}/{expected_count} vectors "
f"for '{namespace}'")
return export_summaryIn a typical migration workflow, this export technique is the first step in moving from Pinecone to a self-hosted vector storage solution. Migration scripts use this pattern to drain legacy namespaces, transform the exported JSONL into the target system’s batch upsert format, and re-index the vectors with the destination database’s semantic search capabilities.
The zero-vector technique is also useful for auditing. Before a migration, teams run the export in dry-run mode to count vectors per namespace and compare against billing records. Discrepancies between describe_index_stats() counts and actual exported vectors can reveal deleted-but-not-yet-garbage-collected entries or replication lag in Pinecone’s infrastructure.
Beyond migration, the same pattern enables backup workflows. Organizations running Pinecone in production can schedule periodic exports to cold storage (S3, GCS) as insurance against vendor lock-in or accidental index deletion. The streaming export keeps memory usage constant regardless of index size, making it practical for indexes with millions of vectors.
seen_ids set and filter duplicates. For very large namespaces, consider using Pinecone’s list endpoint (if available) to get IDs first, then fetch in batches.429 responses. Space queries with a small delay (100–500ms) to stay within limits.include_metadata=True flag significantly increases response payload size. Monitor memory usage and consider streaming to disk rather than accumulating in memory."") is Pinecone’s default namespace but is easy to overlook. Always check for it in the stats response. Vectors inserted without an explicit namespace land here.