Columnar file formats are a hidden bottleneck in many AI data pipelines. Parquet, ORC, Arrow IPC, and related formats were optimized for analytics workloads where humans eventually consume aggregates, dashboards, or SQL result sets. Modern multimodal and training workloads are different: they need repeated scans, selective point reads, compressed tensor-adjacent arrays, small-object access from object storage, and fast conversion into memory layouts that query engines or accelerators can consume.
Vortex is a newer columnar format that separates the logical file container from the physical layout tree used to read data. Instead of baking one row-group/page hierarchy into the file specification, a Vortex file stores serialized arrays, a serialized layout tree, file-level statistics, and a compact postscript/footer that lets readers find only the segments they need. The result is a format family: writers can compose chunked, struct, dictionary, zoned, and custom layouts while readers interact through a common scan model.
The important engineering idea is not simply “a faster Parquet.” It is the decoupling of logical data type, physical encoding, out-of-memory layout, and scan execution. That decoupling lets a file remain self-describing while still evolving toward new compression kernels, object-store access patterns, pruning metadata, and eventually device-aware buffers. For AI engineers, the pattern is useful whenever a dataset has to serve both analytical engines and machine consumers such as training loops, retrieval systems, feature pipelines, or multimodal batch processors.
A Vortex array is a tree node with a logical data type, length, physical encoding, child arrays, buffers, statistics, and a vtable for behavior. That means an encoded array can carry enough information to operate on compressed data directly when a specialized kernel exists. If no specialized kernel exists, the array can execute toward a canonical representation and then run ordinary operations.
This is a useful mental model for data systems: a compressed array should not be treated as inert bytes that must be fully inflated before every operation. It can be a small execution plan for decompression, filtering, projection, and statistics-aware pruning. The same idea shows up in vectorized databases, GPU dataframes, and embedding stores that want to delay materialization until the last responsible moment.
Layouts are the persistent, out-of-memory counterpart to arrays. A layout tree describes how serialized arrays are partitioned and where lazy segments live. During deserialization, the layout is bound to a segment source such as a local file, object store, remote cache, or database block store. A scan can then fetch only the required columns, row chunks, or extents instead of loading the full file.
Built-in layouts include flat layouts for a single array, struct layouts for named columns, chunked layouts for row-wise partitions, dictionary layouts for shared value dictionaries, and zoned layouts with statistics for pruning. By composing these layouts, a writer can mimic Parquet-style row groups, create smaller random-access blocks, or design a file that favors object-store range reads.
The Vortex file specification is intentionally small. Most of the read behavior lives in layouts and encodings, while the file container records how to find those structures. This keeps the stable on-disk envelope separate from the parts that need to evolve for new workloads.
A Vortex file starts and ends with a magic number and stores its critical discovery metadata near the tail. The postscript points to the top-level data type, root layout, file statistics, and footer. The postscript is size-bounded so an initial tail read can find the information needed to locate the rest of the metadata without a chain of extra object-store requests.
// Simplified Vortex-like file discovery flow.
tail = read_last_64kb(object_uri)
assert tail.ends_with("VTXF")
postscript = parse_postscript(tail)
layout_bytes = range_read(object_uri, postscript.layout.offset, postscript.layout.length)
footer_bytes = range_read(object_uri, postscript.footer.offset, postscript.footer.length)
stats_bytes = range_read(object_uri, postscript.statistics.offset, postscript.statistics.length)
layout = decode_layout(layout_bytes, footer_bytes)
reader = bind_layout_to_segment_source(layout, object_uri)
scan = reader.scan(columns=["embedding", "image_ref"], predicate="tenant_id = 42")The footer dictionary-encodes segment, layout, array, compression, and encryption specifications. A segment entry records offset, length, alignment, and reserved hooks for compression or encryption. This lets the format represent many small physical buffers without repeating the same metadata over and over.
When a query asks for a few columns or a filtered row range, the layout tree determines which segments must be fetched. A struct layout can drop unrelated columns, a chunked layout can expose row partitions, and a zoned layout can prune chunks using min/max or other statistics. The reader can then produce Arrow-compatible batches for engines that expect columnar memory.
One practical integration lesson from the Iceberg work is that byte-range splittability is not the only useful notion of scan parallelism. A format with flexible layouts may need row-splittability or planner-level metadata that tells Spark or another engine how to create many balanced tasks even when the relevant physical byte ranges do not match Parquet’s contiguous row-group assumptions.
Training datasets often contain vectors, captions, image references, model features, and medium-sized binary payloads. Storing each item as an object causes request-latency overhead, while stuffing everything into analytics-oriented files can make selective reads inefficient. A layout-tree columnar file can batch segments for object-store throughput while preserving enough metadata to retrieve a narrow slice quickly.
This is especially useful for active training loops where a sampler may need random-ish access to many records, not just sequential SQL scans. The file can keep metadata, labels, embeddings, and object references in columnar layouts, while larger media payloads are batched or externalized according to the workload’s access pattern.
Table formats such as Iceberg manage snapshots, schema evolution, manifests, and transaction metadata, while data files hold the actual records. A Vortex data-file reader can plug into this model by exposing projection, predicate pushdown, and Arrow batch output to engines such as Spark. The value comes from making the file format pluggable without forcing the table format to understand every encoding detail.
Feature stores, embedding refresh jobs, and offline retrieval benchmarks repeatedly read wide columns and large numeric arrays. If statistics can skip irrelevant extents and compressed kernels can operate without full materialization, the pipeline spends less time converting Parquet pages into Arrow and then into application buffers. The same pattern applies to multimodal search corpora where text, vectors, and image references live in the same logical table.
Do not let the logical schema dictate one physical representation. A logical UTF-8 column might be stored as dictionary values plus integer codes, run-end encoded codes, FSST-compressed strings, or a canonical variable-binary view. A logical integer column might use frame-of-reference, bit packing, delta encoding, or sparse patches. The reader should ask the encoding vtable how to execute or canonicalize the data.
// Pseudocode for an encoding-aware scan operator.
func scanColumn(layout Layout, expr Predicate) Iterator[RecordBatch] {
for segment in layout.segments() {
stats := segment.statistics()
if stats.canProveNoMatch(expr) {
continue
}
arr := segment.loadArrayLazy()
if arr.encoding().hasKernel("filter") {
yield arr.encoding().filterCompressed(arr, expr)
continue
}
canonical := arr.executeTowardCanonical()
yield filterCanonical(canonical, expr)
}
}Object stores make metadata placement matter. Put enough information at the end of the file for a reader to discover the root layout and footer with one tail read, then fetch only the metadata segments it needs. Keep large or shareable schema metadata in a separate segment so it does not force every narrow scan to load the full type graph.
The Spark/Iceberg integration uses a Rust implementation exposed through Java native bindings. That pattern avoids rewriting the file reader in every host language, but it moves memory safety across an FFI boundary. Use opaque handles, explicit close methods, and host-language lifecycle types such as AutoCloseable so native allocations, sockets, and file handles are released exactly once.