Traditional object storage is optimized for immutable blobs: write the whole object, read ranges from it, and replace it with a new object when the contents change. That model works well for model weights, database snapshots, and archived files, but it is awkward for workloads that continuously produce a single logical stream. Log shippers, media transcoders, sensor collectors, and checkpoint writers often want to publish bytes as they arrive while readers consume the same growing object.
Appendable object storage adds a constrained mutation primitive to that model. Instead of rewriting the object, a client sends a normal object write request with an explicit byte offset that must equal the current object size. If the offset matches, the service attaches the new bytes to the end of the object and immediately exposes the longer object to readers. If the offset is stale, the write fails instead of silently corrupting the stream.
Amazon S3 Express One Zone implements this pattern for directory buckets by using PutObject with a write-offset header. Each successful append becomes a new internal part, with the same 5 GiB maximum request size as a normal upload and a 10,000-part ceiling per object. Those constraints make appendable objects powerful but not free: production designs must manage writer ownership, offset discovery, part-count compaction, checksums, and cost-per-append.
This topic teaches the transferable design pattern behind appendable objects: how offset-checked appends differ from multipart upload, where they fit in streaming systems, and how to build a safe client around part limits and concurrent writers.
An append operation is only valid when the client proves where the end of the object currently is. In S3 Express One Zone, the client sets x-amz-write-offset-bytes or the SDK’s equivalent field to the existing object size. An offset of 0 can create a new object; later appends must use the precise byte length returned by metadata reads or tracked from prior successful writes.
This turns object growth into an optimistic concurrency protocol. The object store does not need a complex file-locking API, and the client does not need to rewrite previous bytes. A stale writer that believes the object is 4096 bytes long will fail if another writer has already extended it to 8192 bytes.
Appendable objects are still represented as a sequence of parts. Each successful append consumes one part slot, and each object has a maximum of 10,000 parts. An object originally created through multipart upload starts with its existing multipart part count; append operations consume the remaining slots. Once the limit is reached, the service returns a too-many-parts error until the object is compacted or rolled over.
The main reason to append rather than stage locally is reader visibility. A log analysis job can read the object while a producer continues writing new entries. A media delivery workflow can publish segments into a growing object while consumers fetch ranges. This is different from multipart upload, where parallel parts are not visible as the final object until completion.
Appendable object storage only adds bytes at the current end. It does not support editing bytes in the middle, truncating the object, or reordering parts. Treat it as a durable streaming sink, not as a replacement for a mutable filesystem or database log.
The client first determines the current object length, then issues a write whose offset equals that length. In AWS SDKs this is exposed as WriteOffsetBytes; in raw HTTP it is the x-amz-write-offset-bytes header. The service validates the offset, stores the appended bytes as a new part, and returns metadata that the client can use to advance local state.
import boto3
from botocore.exceptions import ClientError
s3 = boto3.client("s3")
def object_size(bucket: str, key: str) -> int:
try:
head = s3.head_object(Bucket=bucket, Key=key)
return head["ContentLength"]
except ClientError as exc:
if exc.response["Error"]["Code"] in {"404", "NoSuchKey", "NotFound"}:
return 0
raise
def append_bytes(bucket: str, key: str, payload: bytes) -> int:
offset = object_size(bucket, key)
s3.put_object(
Bucket=bucket,
Key=key,
Body=payload,
WriteOffsetBytes=offset,
)
return offset + len(payload)A robust client treats the offset as a compare-and-append condition. If the append fails because the offset no longer matches, the client should refresh the object size, decide whether the payload is still valid, and retry only if duplicate data cannot be introduced.
Multipart upload is best when a client has one large object to assemble and readers do not need the final object until all parts finish. Parts can be uploaded in parallel, retried independently, and completed in a final assembly step. Appendable objects are best when the logical object should become readable incrementally, but appends are serialized at the end offset.
The distinction matters for architecture. A backup writer that already has a complete 200 GiB snapshot should use multipart upload. A stream processor that emits records every second and wants downstream readers to tail the object can use append operations, provided it manages the part limit and request cost.
Many systems still write logs locally, rotate files, compress them, and upload final artifacts to object storage. Appendable objects allow a log writer to push small batches directly into a durable cloud object while consumers read the same object through range requests. This reduces local buffering and can simplify failure recovery because acknowledged bytes are already remote.
Video transcoding pipelines often produce output as a sequence of segments. Appending segments into an object lets downstream services read the growing file during production instead of waiting for a final copy. The pattern is especially useful when the producer and consumer are colocated with low-latency object storage.
Database export jobs can write checkpoints or audit streams as ordered byte ranges. If a worker fails, a replacement can inspect object length, resume at the next offset, and avoid re-uploading the entire artifact. This design works best when each chunk is self-delimiting and replay can detect incomplete records at the tail.
The safest production pattern assigns one active writer per object. A coordination service, database row, or queue partition grants a lease for a stream key. The writer tracks the next offset locally after each successful append and periodically confirms it with HeadObject. If the lease changes owners, the new writer starts by reading the object size rather than trusting inherited process state.
class AppendWriter:
def __init__(self, bucket: str, key: str, part_budget: int = 9500):
self.bucket = bucket
self.key = key
self.offset = object_size(bucket, key)
self.parts_written = 0
self.part_budget = part_budget
def write_batch(self, records: list[bytes]) -> None:
payload = b"".join(record + b"\\n" for record in records)
if self.parts_written >= self.part_budget:
raise RuntimeError("roll over before exhausting object part slots")
s3.put_object(
Bucket=self.bucket,
Key=self.key,
Body=payload,
WriteOffsetBytes=self.offset,
ChecksumAlgorithm="CRC32C",
)
self.offset += len(payload)
self.parts_written += 1Do not wait for the 10,000-part ceiling in the hot path. Roll over to a new object name before exhaustion, such as logs/2026/04/22/stream-00017.log to stream-00018.log. A manifest object or database row can point readers at the current segment and preserve ordering across segments.
When a long-lived object accumulates many small appended parts, a server-side copy can rewrite it as a fresh object and reset the internal part count. This is similar to compaction in log-structured storage: it trades background I/O for a healthier future write path. Schedule compaction when readers can tolerate an object generation change and when the object size is still practical to copy.
Use application-level framing inside the object. Length-prefixed records, newline-delimited JSON, or chunk headers with sequence numbers let readers recover from partial tails and detect gaps. Storage checksums verify bytes in transit and at rest, but application framing verifies semantic completeness.