Vector databases rarely agree on their control-plane APIs, index names, namespace semantics, metadata filters, or bulk import behavior. A team that wants to move from one engine to another can often export raw embeddings, but loses the context needed to rebuild a useful search corpus: dimensions, distance metric, model name, vector column names, per-index layout, and the non-vector payload fields that make search results understandable.
A portable vector dataset format treats embeddings as data, not as an opaque side effect of one managed database. The vector-io project calls this shape VDF: a directory containing a metadata manifest plus Parquet files for vectors and payload fields. Export adapters pull batched records from source databases, normalize them into this directory layout, and import adapters read the same layout to upsert records into a target database.
The transferable design pattern is a two-layer interchange boundary. The stable layer is the dataset package: schema, files, IDs, model metadata, dimensions, metrics, and paths. The changing layer is a family of adapters for Pinecone, Qdrant, Milvus, LanceDB, Chroma, Turbopuffer, and similar systems. Keeping those layers separate makes migrations, offline benchmarks, re-embedding jobs, and reproducible vector-search experiments much easier to run.
A vector interchange package needs both metadata and bulk data. The manifest records what the files mean: exported source, version, author, indexes, namespaces, dimensions, metric, vector columns, model information, data path, and optionally index configuration. The Parquet files store the actual IDs, vectors, and payload fields in a columnar format that is efficient for large scans and selective reads.
Each database adapter owns the vendor-specific details: listing indexes, fetching points in batches, mapping namespace concepts, creating target indexes, and upserting records. The interchange layer should not know how Qdrant scrolls points or how Pinecone names indexes. It should only receive normalized rows and enough metadata to reconstruct a compatible target.
Embeddings from different models do not share a vector space just because they have the same numeric type. A useful interchange format records the model name, vector column names, dimensions, and text-column mapping when available. Import tools can then refuse impossible loads, warn about missing dimensions, or trigger a re-embedding workflow before building a new index.
A dump of float arrays is not enough to recreate a vector search system. The package must preserve IDs, payload fields, dimensions, metric, model provenance, and index grouping so a target database can rebuild both search behavior and result meaning.
The VDF pattern uses a root directory with a metadata file and one or more data directories. Each index or namespace points at Parquet files containing vector rows. The manifest’s file_structure field lists generated files, while each namespace entry stores the relative data_path to the files that belong to that partition of the corpus.
{
"version": "0.0.0",
"exported_from": "qdrant",
"exported_at": "2026-04-23T12:00:00-07:00",
"id_column": "id",
"file_structure": ["products/1.parquet", "products/2.parquet"],
"indexes": {
"products": [
{
"namespace": "",
"index_name": "products",
"total_vector_count": 2500000,
"exported_vector_count": 2500000,
"dimensions": 768,
"metric": "cosine",
"model_name": "sentence-transformers/all-mpnet-base-v2",
"vector_columns": ["vector"],
"data_path": "products",
"index_config": {"source": "captured-from-export"}
}
]
}
}An exporter starts by discovering source indexes and creating an output directory. It then fetches vectors and metadata in batches, converts them into rows keyed by a stable ID column, writes those rows to Parquet, and accumulates a schema. When the export completes, it writes namespace metadata with dimensions, count, metric, model name, vector columns, schema text, and data paths.
def export_index(source, package, index_name, batch_size):
vectors = {}
metadata = {}
for point in source.iter_points(index_name, batch_size=batch_size):
vectors[point.id] = point.vector
metadata[point.id] = point.payload
if estimated_size_mb(vectors, metadata) >= package.max_file_size_mb:
package.write_parquet(index_name, vectors, metadata)
vectors, metadata = {}, {}
if vectors:
package.write_parquet(index_name, vectors, metadata)
package.add_namespace_meta(
index_name=index_name,
dimensions=source.dimensions(index_name),
metric=source.metric(index_name),
model_name=source.model_name(index_name),
total_vector_count=source.count(index_name)
)An importer reads the manifest first. It validates that the package has indexes, resolves data paths, determines the ID column, checks or infers dimensions, and loads Parquet files in batches. For each target database, the adapter maps rows into that database’s upsert format and optionally creates a new index name when the user does not want to overwrite an existing one.
Teams often start with a convenient managed vector store and later need different cost, latency, filtering, or deployment properties. A portable package lets them export an index once, inspect the dataset offline, import a subset into candidate databases, and run recall and latency comparisons before choosing a migration target.
ANN benchmarks are only meaningful when every system receives the same vectors, IDs, metadata, and query corpus. A columnar interchange package becomes the fixture for repeatable tests. The same package can feed exact-search baselines, HNSW builds, partitioned indexes, filtered query tests, and re-ranking experiments.
When an embedding model changes, existing vectors are no longer compatible with new query vectors. A portable dataset can preserve the source text or payload columns next to old vectors, run a re-embedding job into a new vector column, and import the refreshed corpus into a shadow index for evaluation before cutover.
The source project uses abstract base classes for importers and exporters. Each concrete adapter declares a database slug and implements parser construction plus the database-specific import or export method. This is the right shape for extensibility: adding a database means adding two adapters, not modifying one giant migration function.
class ExportAdapter:
DB_NAME_SLUG = "exampledb"
def get_index_names(self):
raise NotImplementedError
def fetch_batches(self, index_name):
raise NotImplementedError
def export(self, package):
for index_name in self.get_index_names():
for batch in self.fetch_batches(index_name):
package.write_rows(index_name, batch)
package.write_manifest_entry(index_name, self.describe_index(index_name))
class ImportAdapter:
DB_NAME_SLUG = "exampledb"
def ensure_index(self, namespace_meta):
raise NotImplementedError
def upsert_batch(self, index_name, rows):
raise NotImplementedErrorJSON is convenient for manifests, but it is a poor container for millions of dense vectors. Parquet gives typed columns, compression, schema introspection, and efficient batch reads. It also keeps metadata columns alongside vectors without forcing the importer to parse one giant document into memory.
Interchange tools often run on proprietary corpora. They should make external publishing and telemetry behavior explicit, documented, and configurable. The source repository includes a flag for pushing packages to a dataset hub and an environment variable for disabling telemetry; production teams should apply the same principle to any internal migration tool.