Large-model training is usually GPU-centric: parameters, gradients, optimizer states, activation metadata, and execution graphs are arranged around the assumption that accelerator memory is the scarce but primary home for training state. That assumption breaks down when the model is larger than available VRAM, when an organization has one high-end GPU instead of a cluster, or when a post-training job needs full-precision updates without accepting low-rank adaptation or heavy quantization.
RAM-centric layer streaming reverses the default ownership model. Host memory stores the durable model parameters and optimizer state, while the GPU becomes a transient compute engine that receives one layer or block at a time. The runtime streams weights to the device, computes the forward and backward work for the active block, streams gradients or updates back to host memory, then releases device memory for the next block.
MegaTrain is a concrete example of this pattern. Its paper describes a memory-centric system that trains 100B+ parameter language models at full precision on a single GPU by storing parameters and optimizer states in CPU RAM, streaming layer state through the GPU, overlapping transfer and compute with double buffering, and replacing persistent autograd graphs with stateless layer templates. The reported result is not that host memory is faster than VRAM; it is that careful scheduling can trade bandwidth and latency for a much larger trainable model envelope.
The durable copy of each parameter tensor, optimizer slot, and often the master precision state lives in CPU RAM. GPU memory only needs the active layer weights, the local activations needed for the current step, temporary buffers, and any colocated inference or rollout engine. This makes capacity proportional to host RAM rather than VRAM, at the cost of making PCIe, NVLink-C2C, or unified-memory bandwidth part of the critical path.
The GPU processes the model as a stream of executable blocks. A block may be a transformer layer, an attention-plus-MLP group, or an architecture-specific unit such as a MoE or hybrid attention block. Once the block finishes its scheduled forward or backward work, its parameter buffers can be evicted from device memory and reused for the next block.
Traditional PyTorch training tends to bind module parameters to a persistent autograd graph. A RAM-centric runtime instead treats each layer as a template: the layer code is stable, but the concrete weights are bound dynamically as they stream in. That separation reduces persistent graph metadata and gives the scheduler explicit control over when parameters, gradients, and temporary buffers exist on the GPU.
Layer streaming does not make large-model training free. It exchanges persistent device residency for a scheduling problem: keep the GPU fed with the next layer while offloading completed work quickly enough that compute does not stall.
The runtime keeps a small device window: current block, next block, and completed-gradient buffers. The current block is computing; the next block is being prefetched; completed gradients are copied back to host memory or handed to a CPU-side optimizer. A minimal scheduler has explicit states for host-resident, prefetching, resident, computing, offloading, and released.
for microbatch in train_loader:
activations = microbatch.tokens
for layer_id in model.layers:
prefetch(layer_id + 1, stream="h2d")
weights = await_resident(layer_id)
activations = run_forward_template(layer_id, weights, activations)
release_if_checkpointed(layer_id, weights)
loss = compute_loss(activations, microbatch.labels)
for layer_id in reversed(model.layers):
prefetch(layer_id - 1, stream="h2d")
weights = await_resident(layer_id)
grads = run_backward_template(layer_id, weights)
offload(grads, destination="host_optimizer", stream="d2h")
release_device_buffers(layer_id)The central optimization is double buffering. While one buffer is being consumed by GPU kernels, another buffer is filled from host memory with the next layer’s weights. During backward, gradient buffers can be copied back to host memory on a separate stream while compute continues on the previous layer. Correct overlap requires pinned host memory, independent CUDA streams, dependency events, and predictable buffer lifetimes.
Optimizer states can be several times larger than the model weights. Keeping Adam moments and master parameters on the CPU avoids exhausting VRAM, but it also makes optimizer throughput important. Systems in this family often rely on optimized CPU Adam implementations, slab pools for gradient transfer, and asynchronous workers so the optimizer does not serialize the GPU pipeline.
When the goal is to update all weights of a large model, parameter-efficient methods may be too restrictive. RAM-centric streaming can make full-precision supervised fine-tuning possible on a single workstation-class GPU, provided the host has enough RAM and the workload can tolerate lower throughput than a GPU cluster.
Long contexts increase activation and KV-related memory pressure. If parameters and optimizer state no longer occupy most of VRAM, more device memory can be reserved for sequence length, activation checkpointing windows, and attention kernels. MegaTrain’s paper reports a 7B model training setup with a 512k token context on a GH200-class system, illustrating why parameter residency is only one part of the memory budget.
For GRPO or PPO-style post-training, an inference engine may need to stay resident for rollout generation while the actor or reference model trains. A layer-streamed training backend can coexist with a quantized rollout model by using transient GPU memory for training layers and leaving a predictable fraction of VRAM for inference KV cache and generation throughput.
A production runtime needs a manifest of layer tensors, byte sizes, dtype, host location, device buffers, and architecture tags. Hybrid models complicate this because not every block has the same structure: dense attention, linear attention, MoE routing, and vision encoders may require different templates and scratch-space budgets.
type LayerState struct {
ID int
Kind string
HostWeights []HostTensorRef
HostOptimizer []HostTensorRef
DeviceWeights []DeviceBuffer
ScratchBytes int64
CheckpointEvery int
}
func (s *Scheduler) EnsureResident(layer *LayerState) error {
if layer.DeviceWeights != nil {
return nil
}
buf := s.bufferPool.Acquire(layer.TotalWeightBytes())
event := s.copyEngine.CopyHostToDeviceAsync(layer.HostWeights, buf)
s.residency[layer.ID] = Residency{Buffer: buf, Ready: event}
return nil
}Allocating transfer buffers per layer will fragment memory and add synchronization. A slab pool preallocates a bounded number of host and device buffers sized for common gradient groups. Backward kernels write into slabs, copy engines offload slabs asynchronously, and CPU optimizer workers recycle them after updates complete.
The optimal batch size is a function of model size, sequence length, layer scratch space, checkpoint interval, host bandwidth, optimizer speed, and any colocated rollout engine. A calculator or dry-run planner should estimate peak transient memory before training starts; guessing often leads to either OOM failures or poor GPU utilization.