S3-Compatible Object Storage (MinIO)

Overview

Object storage has become the de facto standard for storing large binary blobs — database snapshots, backups, model weights, and log archives — in distributed systems. Amazon S3 established the API contract that virtually every cloud provider and self-hosted solution now implements: a flat namespace of buckets containing objects, accessed via HTTP with presigned URLs, multipart uploads, and versioning. For distributed database systems that need to persist snapshots and backups to remote storage, coding directly against the S3 REST API (or any single provider’s SDK) creates a hard dependency on a specific cloud vendor.

S3-compatible object storage solutions like MinIO solve this problem by implementing the full S3 API surface as a standalone binary that can run locally, in CI pipelines, or in production on bare metal. By targeting the S3 API contract rather than a specific provider, a database snapshot and backup subsystem can transparently switch between AWS S3, Google Cloud Storage (via its S3-compatible endpoint), and MinIO without changing a single line of application code. The only difference is the endpoint URL and credentials.

This topic covers the architecture of S3-compatible object storage, how a distributed database integrates with it for snapshot persistence and recovery, and the patterns for using MinIO as a local development and testing substitute for cloud object storage. Understanding this integration is essential for anyone working on distributed database durability and disaster recovery subsystems.

Core Concepts

How It Works

A typical distributed database snapshot subsystem uses a storage abstraction layer that separates the snapshot creation logic (serializing in-memory state to a binary format) from the persistence logic (writing that binary to durable storage). The persistence layer implements a SnapshotStore interface with methods for Put, Get, List, and Delete. The S3 implementation of this interface uses the AWS SDK’s S3 client configured with a custom endpoint URL, allowing it to target MinIO or any S3-compatible service.

S3 Client Configuration

The key to S3 compatibility is endpoint injection. Rather than hardcoding AWS endpoints, the client accepts an endpoint URL from configuration. For AWS S3 this is left empty (the SDK auto-discovers the regional endpoint). For MinIO or other compatible services, it points to the custom server.

// storage/s3store.go
package storage

import (
    "context"
    "io"
    "fmt"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/credentials"
    "github.com/aws/aws-sdk-go-v2/service/s3"
)

// S3Store implements SnapshotStore using any S3-compatible backend.
type S3Store struct {
    client *s3.Client
    bucket string
}

// NewS3Store creates an S3Store pointed at the given endpoint.
// Pass "" for endpoint to use the default AWS S3 endpoint.
func NewS3Store(ctx context.Context, endpoint, bucket, accessKey, secretKey, region string) (*S3Store, error) {
    cfg, err := config.LoadDefaultConfig(ctx,
        config.WithRegion(region),
        config.WithCredentialsProvider(
            credentials.NewStaticCredentialsProvider(accessKey, secretKey, ""),
        ),
    )
    if err != nil {
        return nil, fmt.Errorf("loading AWS config: %w", err)
    }

    client := s3.NewFromConfig(cfg, func(o *s3.Options) {
        if endpoint != "" {
            o.BaseEndpoint = aws.String(endpoint)
            o.UsePathStyle = true // MinIO requires path-style addressing
        }
    })

    return &S3Store{client: client, bucket: bucket}, nil
}

Snapshot Upload with Multipart Support

For large snapshots, the store uses the S3 manager’s upload function which automatically switches to multipart upload when the object exceeds a configurable threshold. This handles chunking, parallel part uploads, and retry logic transparently.

// storage/s3store.go (continued)
import (
    "github.com/aws/aws-sdk-go-v2/feature/s3/manager"
    "github.com/aws/aws-sdk-go-v2/service/s3/types"
)

// PutSnapshot uploads a snapshot to S3 with multipart support.
func (s *S3Store) PutSnapshot(ctx context.Context, key string, r io.Reader, size int64) error {
    uploader := manager.NewUploader(s.client, func(u *manager.Uploader) {
        u.PartSize = 64 * 1024 * 1024 // 64 MB parts
        u.Concurrency = 4              // Upload 4 parts in parallel
    })

    _, err := uploader.Upload(ctx, &s3.PutObjectInput{
        Bucket:               aws.String(s.bucket),
        Key:                  aws.String(key),
        Body:                 r,
        ServerSideEncryption: types.ServerSideEncryptionAes256,
    })
    if err != nil {
        return fmt.Errorf("uploading snapshot %s: %w", key, err)
    }
    return nil
}

// GetSnapshot downloads a snapshot from S3.
func (s *S3Store) GetSnapshot(ctx context.Context, key string) (io.ReadCloser, error) {
    result, err := s.client.GetObject(ctx, &s3.GetObjectInput{
        Bucket: aws.String(s.bucket),
        Key:    aws.String(key),
    })
    if err != nil {
        return nil, fmt.Errorf("downloading snapshot %s: %w", key, err)
    }
    return result.Body, nil
}

// ListSnapshots returns all snapshot keys with a given prefix.
func (s *S3Store) ListSnapshots(ctx context.Context, prefix string) ([]string, error) {
    paginator := s3.NewListObjectsV2Paginator(s.client, &s3.ListObjectsV2Input{
        Bucket: aws.String(s.bucket),
        Prefix: aws.String(prefix),
    })

    var keys []string
    for paginator.HasMorePages() {
        page, err := paginator.NextPage(ctx)
        if err != nil {
            return nil, fmt.Errorf("listing snapshots: %w", err)
        }
        for _, obj := range page.Contents {
            keys = append(keys, *obj.Key)
        }
    }
    return keys, nil
}

Integration Testing with MinIO

Tests spin up a MinIO container, create a test bucket, run the snapshot lifecycle, and verify data integrity by comparing checksums. The test helper manages the container lifecycle so each test gets an isolated environment.

// storage/s3store_test.go
package storage_test

import (
    "bytes"
    "context"
    "crypto/sha256"
    "io"
    "testing"

    "github.com/example/mydb-private/storage"
    "github.com/testcontainers/testcontainers-go"
    "github.com/testcontainers/testcontainers-go/modules/minio"
)

func TestS3StoreRoundTrip(t *testing.T) {
    ctx := context.Background()

    // Start a MinIO container.
    minioC, err := minio.Run(ctx, "minio/minio:latest",
        minio.WithUsername("minioadmin"),
        minio.WithPassword("minioadmin"),
    )
    if err != nil {
        t.Fatalf("starting MinIO: %v", err)
    }
    defer testcontainers.CleanupContainer(t, minioC)

    endpoint, _ := minioC.ConnectionString(ctx)

    store, err := storage.NewS3Store(ctx,
        "http://"+endpoint, "test-snapshots",
        "minioadmin", "minioadmin", "us-east-1",
    )
    if err != nil {
        t.Fatalf("creating S3Store: %v", err)
    }

    // Create the test bucket.
    store.CreateBucket(ctx, "test-snapshots")

    // Write a snapshot.
    data := bytes.Repeat([]byte("snapshot-data-"), 10000) // ~140 KB
    expectedHash := sha256.Sum256(data)

    err = store.PutSnapshot(ctx, "node-1/snap-00001.bin", bytes.NewReader(data), int64(len(data)))
    if err != nil {
        t.Fatalf("PutSnapshot: %v", err)
    }

    // Read it back and verify.
    rc, err := store.GetSnapshot(ctx, "node-1/snap-00001.bin")
    if err != nil {
        t.Fatalf("GetSnapshot: %v", err)
    }
    defer rc.Close()

    downloaded, _ := io.ReadAll(rc)
    actualHash := sha256.Sum256(downloaded)
    if expectedHash != actualHash {
        t.Fatal("snapshot checksum mismatch after round-trip")
    }
}

Practical Applications

In a distributed database, S3-compatible object storage serves as the durability backbone for the Raft snapshot subsystem. When a node’s Raft log grows beyond a configured threshold, the state machine takes a point-in-time snapshot of the database, serializes it to a binary format, and uploads it to S3. The Raft log is then truncated up to the snapshot index. On recovery, a node downloads the latest snapshot from S3, applies it to rebuild the state machine, and then replays any log entries after the snapshot index. This combination of local Raft log for recent durability and remote S3 for long-term snapshot storage balances latency (log entries are applied locally) with cost (old state is archived to cheap object storage).

A second application is cross-region backup. By replicating snapshots to S3 buckets in different regions (or to different MinIO clusters in different data centers), a database achieves geographic redundancy without implementing custom cross-region replication protocols. S3’s built-in cross-region replication feature handles the data transfer, and a backup scheduler simply verifies that replicas appear in the target bucket within the SLA window.

During development and CI, MinIO eliminates the need for cloud credentials and network access. Developers run docker run -p 9000:9000 minio/minio server /data locally and point the database configuration at http://localhost:9000. CI pipelines use a MinIO service container that starts in under a second, enabling full integration test coverage of the snapshot subsystem on every commit.

Typical Integration Points

• Raft snapshot persistence — uploading state machine snapshots after log compaction and downloading them during node recovery or cluster bootstrapping.
• Scheduled backup jobs — CronJob controllers that trigger snapshot creation and upload on a configurable schedule, with retention policies that delete old snapshots.
• CI/CD integration testing — running the full snapshot upload/download/restore cycle against a MinIO container to catch serialization bugs and API regressions.
• Multi-cloud portability — switching between AWS S3, GCS (via S3 compatibility), and on-prem MinIO by changing only the endpoint URL in configuration.

Implementation Patterns

Storage Interface Abstraction

The SnapshotStore interface abstracts the storage backend behind Put, Get, List, and Delete methods. The S3 implementation is one of several — others include a local filesystem store (for development without Docker) and an in-memory store (for unit tests that don’t need persistence). The interface is kept deliberately narrow to avoid leaking S3-specific concepts (like multipart uploads or presigned URLs) into the snapshot engine.

Path-Style vs Virtual-Hosted-Style Addressing

AWS S3 defaults to virtual-hosted-style URLs (bucket.s3.amazonaws.com/key), while MinIO and most self-hosted solutions require path-style URLs (s3.endpoint/bucket/key). The client must set UsePathStyle = true when targeting non-AWS endpoints. Forgetting this is one of the most common integration failures — the client constructs a virtual-hosted URL that resolves to a nonexistent DNS name, producing cryptic connection errors rather than a clear 404.

Retry and Backoff for Transient Failures

S3 operations can fail transiently due to rate limiting (HTTP 503 SlowDown), network hiccups, or temporary server errors. The AWS SDK includes built-in retry logic with exponential backoff, but the default retry count (3) may be insufficient for large multipart uploads over unreliable networks. For example, a production S3Store can override the retry configuration to use 5 retries with a maximum backoff of 30 seconds, and wraps the upload in an application-level retry that re-initiates the entire multipart upload if too many parts fail.

Object Key Naming Convention

Snapshot objects are keyed by a hierarchical path: {cluster-id}/{node-id}/snap-{index}.bin. This convention enables efficient listing by cluster or node using the S3 prefix parameter, supports retention policies that operate on per-node snapshot histories, and makes it easy to identify which node produced which snapshot during incident investigation. The index is zero-padded to 10 digits to ensure lexicographic ordering matches chronological ordering.

Common Pitfalls

Forgetting Path-Style Addressing for MinIO

The most frequent MinIO integration bug is leaving UsePathStyle as false (the default). The SDK constructs a virtual-hosted URL like mybucket.localhost:9000/key, which fails DNS resolution. The fix is a single flag, but the error message — a connection refused or DNS lookup failure — doesn’t mention S3 path style, making this hard to diagnose without prior experience.

Ignoring Incomplete Multipart Uploads

If a multipart upload is initiated but never completed (due to a crash or timeout), the uploaded parts consume storage on the S3 server indefinitely. MinIO does not garbage-collect incomplete uploads by default. Production deployments should configure a bucket lifecycle rule with AbortIncompleteMultipartUpload set to expire incomplete uploads after a reasonable period (e.g., 24 hours).

Assuming Atomic Visibility

S3 provides read-after-write consistency for new objects (since December 2020), but overwriting an existing key may still show the old version for a brief window on some S3-compatible implementations. Snapshot keys in a well-designed system are immutable (each snapshot gets a new index-based key), avoiding this issue entirely. Code that overwrites existing keys should not assume the new version is immediately visible to concurrent readers.

Hardcoding Credentials

Embedding S3 access keys in source code or configuration files is a security risk. The AWS SDK supports credential resolution from environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), IAM instance profiles, and OIDC federation. Production deployments should use IAM roles (on EC2/EKS) or workload identity (on GKE) to avoid managing static credentials. Only local MinIO instances should use static access keys, and even then the credentials should come from environment variables, not checked-in configuration.

Not Setting Content-Length for Uploads

Some S3-compatible servers reject PutObject requests that omit the Content-Length header, even though AWS S3 supports chunked transfer encoding. When uploading from a stream of unknown size (e.g., a compressed snapshot being generated on-the-fly), use the multipart upload API instead, which requires knowing only the size of each part. Alternatively, buffer the entire object to disk first and use the file size as the content length.

Further Reading