Memory mapping is elegant: map a file, treat it like an array, and let the operating system page data in on demand. For random reads, immutable index files, and simple readers, that elegance is often worth it. But for large sequential scans, mmap() can expose a surprising bottleneck: every first touch of a page requires the kernel to establish a mapping into the process page tables, even when the file data is already warm in the page cache.
Direct streaming takes the opposite approach. Instead of faulting through a virtual address range page by page, the program issues explicit reads into reusable buffers and pipelines I/O with computation. With asynchronous APIs such as io_uring, worker threads, or batched pread calls, a scanner can keep storage, DMA, caches, and CPU execution overlapped.
The training lesson is not that disk is literally faster than memory. The lesson is that access pattern matters more than slogans. Sequential scan throughput is bounded by the slowest stage in a pipeline: page-fault latency, memory bandwidth, NUMA locality, copy cost, syscall overhead, storage bandwidth, or branchy compute. A good systems engineer chooses the I/O interface that keeps that pipeline full.
The Linux page cache may already contain file data, but an mmap() reader still has to fault virtual pages into the process address space. Those minor faults do not read from disk, yet they still require kernel work: locate the cached page, update page tables, invalidate or coordinate TLB state as needed, and return to user code.
A streaming design allocates a bounded set of buffers and cycles them through states such as free, submitted, filled, processing, and reusable. The dataset can be much larger than memory because the memory footprint is tied to pipeline depth and block size, not file size.
Storage latency is large, but asynchronous streaming can hide it by having many requests in flight. Page faults are different: they interrupt the thread at the point of demand. If the scanner crosses a 4 KiB boundary and must wait for kernel mapping work, the compute pipeline stalls exactly where it wants to run continuously.
If the access pattern is predictable, request data before the CPU needs it. Demand paging is a convenient cache abstraction, but explicit streaming gives the program control over queue depth, block size, buffer placement, and overlap between I/O and computation.
The memory-mapped version is compact. It opens a file, maps the whole range, and loops over typed values. The code is easy to read and can be fast enough for many workloads, but it leaves paging and readahead policy mostly to the kernel.
int fd = open(path, O_RDONLY);
int *data = mmap(NULL, size_bytes, PROT_READ, MAP_SHARED, fd, 0);
size_t total = size_bytes / sizeof(int);
uint64_t count = 0;
for (size_t i = 0; i < total; i++) {
count += (data[i] == 10);
}A streaming scanner separates the queueing phase from the processing phase. It submits reads for future blocks, processes completed buffers in order or out of order depending on the algorithm, then returns buffers to the free pool. The core loop is a backpressure controller.
while (blocks_done < total_blocks) {
while (free_buffers_available() && blocks_submitted < total_blocks) {
buffer *b = acquire_buffer();
submit_read(ring, fd, b->data, block_size, blocks_submitted * block_size, b);
blocks_submitted++;
}
completion c = wait_for_completion(ring);
buffer *b = c.user_data;
count += count_matches_branchless((int *)b->data, b->bytes / sizeof(int));
release_buffer(b);
blocks_done++;
}The compute stage should not become the bottleneck. The source uses loop unrolling and branchless comparisons so compilers can schedule vector-friendly code. Once I/O is pipelined, inefficient counting logic can dominate the benchmark.
Columnar scans, compaction jobs, analytics filters, and backup verification often read large contiguous ranges. If the engine can predict the next blocks, explicit streaming can outperform a page-fault-driven path and gives the database control over admission, cancellation, and buffer reuse.
Index rebuilds, vector quantization passes, posting-list merges, and checksum scans all operate over large files with simple per-record computation. These jobs benefit from block sizing and queue depth tuned to the storage device rather than relying on default virtual-memory behavior.
ETL and ML data preparation frequently scan terabytes to filter, tokenize, validate, or count records. A streaming architecture lets the pipeline bound memory while saturating disks and CPUs, making it easier to run on commodity machines without loading full datasets into RAM.
Each buffer should have an explicit state. This avoids double-use bugs where a buffer is resubmitted while the compute stage is still reading it, or where completed I/O overwrites data that has not been consumed.
enum buffer_state {
BUFFER_FREE,
BUFFER_SUBMITTED,
BUFFER_FILLED,
BUFFER_PROCESSING
};
struct scan_buffer {
void *data;
size_t bytes;
enum buffer_state state;
uint64_t block_index;
};Small blocks increase completion overhead and metadata churn. Huge blocks reduce scheduling flexibility and may harm cache locality. Tune block size with queue depth: the product determines how much work is in flight and how much memory the scanner reserves.
On NUMA machines, place buffers close to the CPU that processes them or keep processing on the NUMA node where DMA lands. Otherwise, the benchmark may silently measure cross-socket memory traffic instead of the I/O strategy.
read() path may copy data from the page cache into user buffers. Direct I/O can avoid the page cache but adds alignment and cache-management complexity.O_DIRECT often requires aligned buffers and block-sized reads. It can bypass useful cache behavior and make small random reads worse.