Secrets Manager-Synced Session Keys & Autocert Cache

Overview

Web applications that use cookie-based sessions need cryptographic keys to sign and encrypt session tokens. These keys must be consistent across server restarts (so existing sessions survive deployments) and across multiple instances (so any server can validate any session). Similarly, applications that serve HTTPS using Let’s Encrypt autocert need to persist certificate material so that certificates aren’t re-issued on every restart, avoiding rate limits and unnecessary latency on first requests.

The pattern explored here solves both problems by using AWS Secrets Manager as the canonical source of truth for session signing keys and TLS certificate material. On startup, the application fetches keys from Secrets Manager. If no keys exist (first deployment), it generates fresh cryptographic material, stores it in Secrets Manager, and proceeds. For autocert, a custom cache implementation mirrors Let’s Encrypt certificates between a local DirCache (for fast reads) and Secrets Manager (for durability across deployments).

In a typical web application, this pattern bootstraps secure cookie keys from AWS Secrets Manager on each deploy, backfills missing keys when a new environment is provisioned, and mirrors Let’s Encrypt autocert material into Secrets Manager with a local DirCache fallback. This ensures that rolling deployments, container restarts, and new instances all serve consistent sessions and valid HTTPS certificates without manual key management.

Core Concepts

Session Key Management

Cookie-based session frameworks (Gorilla sessions, Express sessions, etc.) use cryptographic keys for two operations: signing (HMAC) to detect tampering, and optionally encrypting (AES) to hide session contents. These keys are typically 32 or 64 bytes of cryptographically random data. The security of every active session depends on these keys—if they’re lost, all sessions are invalidated; if they’re leaked, all sessions can be forged.

AWS Secrets Manager as Key Store

Secrets Manager provides encrypted storage with IAM-based access control, automatic versioning, and audit trails via CloudTrail. Compared to environment variables or config files, it offers:

• Encryption at rest: Secrets are encrypted with KMS keys you control
• Access control: IAM policies restrict which services and roles can read which secrets
• Versioning: Previous secret values are preserved, enabling rollback
• Audit trail: Every read and write is logged in CloudTrail
• Cross-region replication: Secrets can be replicated for multi-region deployments

Autocert and the ACME Protocol

Let’s Encrypt issues free TLS certificates via the ACME protocol. Go’s golang.org/x/crypto/acme/autocert package automates the entire flow: when a TLS handshake arrives for a hostname, autocert checks its cache for a valid certificate, and if none exists, performs the ACME challenge (typically HTTP-01 or TLS-ALPN-01), obtains the certificate, and caches it. The cache interface is pluggable—the default DirCache writes to the local filesystem.

Backfill Strategy

When a new environment is provisioned or a secret is accidentally deleted, the application must generate new keys and store them rather than failing to start. This “backfill” pattern checks Secrets Manager first, falls back to generating new material if the secret doesn’t exist, stores the new material in Secrets Manager, and then proceeds with normal operation. This makes deployments self-healing without requiring manual secret provisioning as a prerequisite.

How It Works

Startup Sequence

1. Application starts and calls bootstrapSessionKeys()
2. Attempt to fetch session keys from Secrets Manager (app/session-keys)
3. If found: decode the JSON payload containing signing and encryption keys
4. If not found (ResourceNotFoundException): generate new random keys, store them in Secrets Manager, and proceed
5. Configure the session store with the fetched or generated keys
6. Initialize autocert with a dual-layer cache (local DirCache + Secrets Manager mirror)
7. Start the HTTPS listener—autocert handles certificate provisioning transparently

Session Key Bootstrap in Go

package auth

import (
	"context"
	"crypto/rand"
	"encoding/json"
	"fmt"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/secretsmanager"
	"github.com/aws/aws-sdk-go-v2/service/secretsmanager/types"
	"github.com/gorilla/sessions"
)

type SessionKeys struct {
	SigningKey    []byte `json:"signing_key"`
	EncryptionKey []byte `json:"encryption_key"`
	// Previous keys for rotation (validation only)
	PrevSigningKey    []byte `json:"prev_signing_key,omitempty"`
	PrevEncryptionKey []byte `json:"prev_encryption_key,omitempty"`
}

func BootstrapSessionKeys(
	ctx context.Context,
	client *secretsmanager.Client,
	secretName string,
) (*sessions.CookieStore, error) {
	keys, err := fetchOrCreateKeys(ctx, client, secretName)
	if err != nil {
		return nil, fmt.Errorf("bootstrap session keys: %w", err)
	}

	// Configure store with current + previous key pairs
	store := sessions.NewCookieStore(
		keys.SigningKey, keys.EncryptionKey,       // current (sign + encrypt)
		keys.PrevSigningKey, keys.PrevEncryptionKey, // previous (validate only)
	)
	store.Options.HttpOnly = true
	store.Options.Secure = true
	store.Options.SameSite = 3 // SameSiteStrictMode
	store.Options.MaxAge = 86400 * 7 // 7 days

	return store, nil
}

func fetchOrCreateKeys(
	ctx context.Context,
	client *secretsmanager.Client,
	secretName string,
) (*SessionKeys, error) {
	// Try to fetch existing keys
	result, err := client.GetSecretValue(ctx, &secretsmanager.GetSecretValueInput{
		SecretId: aws.String(secretName),
	})
	if err == nil {
		var keys SessionKeys
		if err := json.Unmarshal([]byte(*result.SecretString), &keys); err != nil {
			return nil, fmt.Errorf("decode session keys: %w", err)
		}
		return &keys, nil
	}

	// Check if secret simply doesn't exist (new environment)
	var notFound *types.ResourceNotFoundException
	if !errors.As(err, &notFound) {
		return nil, fmt.Errorf("fetch session keys: %w", err)
	}

	// Generate new keys and store them
	keys := &SessionKeys{
		SigningKey:    generateKey(64),
		EncryptionKey: generateKey(32),
	}

	payload, _ := json.Marshal(keys)
	_, err = client.CreateSecret(ctx, &secretsmanager.CreateSecretInput{
		Name:         aws.String(secretName),
		SecretString: aws.String(string(payload)),
		Description:  aws.String("Auto-generated session signing/encryption keys"),
	})
	if err != nil {
		return nil, fmt.Errorf("store new session keys: %w", err)
	}

	return keys, nil
}

func generateKey(size int) []byte {
	key := make([]byte, size)
	if _, err := rand.Read(key); err != nil {
		panic("crypto/rand failed: " + err.Error())
	}
	return key
}

Dual-Layer Autocert Cache

package auth

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/secretsmanager"
	"github.com/aws/aws-sdk-go-v2/service/secretsmanager/types"
	"golang.org/x/crypto/acme/autocert"
)

// SecretsManagerCache implements autocert.Cache with a local DirCache
// read-through layer and Secrets Manager as the durable backend.
type SecretsManagerCache struct {
	sm        *secretsmanager.Client
	prefix    string // e.g., "app/autocert/"
	localDir  string // local DirCache path for fast reads
}

func NewSecretsManagerCache(
	client *secretsmanager.Client,
	prefix string,
	localDir string,
) *SecretsManagerCache {
	os.MkdirAll(localDir, 0700)
	return &SecretsManagerCache{
		sm:       client,
		prefix:   prefix,
		localDir: localDir,
	}
}

func (c *SecretsManagerCache) Get(ctx context.Context, key string) ([]byte, error) {
	// Try local cache first (fast path)
	localPath := filepath.Join(c.localDir, key)
	if data, err := os.ReadFile(localPath); err == nil {
		return data, nil
	}

	// Fall back to Secrets Manager
	result, err := c.sm.GetSecretValue(ctx, &secretsmanager.GetSecretValueInput{
		SecretId: aws.String(c.prefix + key),
	})
	if err != nil {
		var notFound *types.ResourceNotFoundException
		if errors.As(err, &notFound) {
			return nil, autocert.ErrCacheMiss
		}
		return nil, fmt.Errorf("autocert cache get: %w", err)
	}

	data := result.SecretBinary
	// Populate local cache for future reads
	os.WriteFile(localPath, data, 0600)

	return data, nil
}

func (c *SecretsManagerCache) Put(ctx context.Context, key string, data []byte) error {
	// Write to local cache
	localPath := filepath.Join(c.localDir, key)
	if err := os.WriteFile(localPath, data, 0600); err != nil {
		return fmt.Errorf("local cache write: %w", err)
	}

	// Mirror to Secrets Manager
	secretName := c.prefix + key
	_, err := c.sm.PutSecretValue(ctx, &secretsmanager.PutSecretValueInput{
		SecretId:    aws.String(secretName),
		SecretBinary: data,
	})
	if err != nil {
		// Secret might not exist yet; create it
		var notFound *types.ResourceNotFoundException
		if errors.As(err, &notFound) {
			_, err = c.sm.CreateSecret(ctx, &secretsmanager.CreateSecretInput{
				Name:         aws.String(secretName),
				SecretBinary: data,
				Description:  aws.String("Autocert certificate: " + key),
			})
		}
		if err != nil {
			return fmt.Errorf("secrets manager write: %w", err)
		}
	}

	return nil
}

func (c *SecretsManagerCache) Delete(ctx context.Context, key string) error {
	// Remove local cache
	os.Remove(filepath.Join(c.localDir, key))

	// Remove from Secrets Manager
	_, err := c.sm.DeleteSecret(ctx, &secretsmanager.DeleteSecretInput{
		SecretId:                   aws.String(c.prefix + key),
		ForceDeleteWithoutRecovery: aws.Bool(true),
	})
	if err != nil {
		var notFound *types.ResourceNotFoundException
		if errors.As(err, &notFound) {
			return nil // Already gone
		}
		return fmt.Errorf("secrets manager delete: %w", err)
	}

	return nil
}

Implementation Patterns

Wiring It Together: HTTPS Server with Autocert

package main

import (
	"context"
	"crypto/tls"
	"log"
	"net/http"

	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/secretsmanager"
	"golang.org/x/crypto/acme/autocert"

	"webapp/auth"
)

func main() {
	ctx := context.Background()

	// Initialize AWS client
	cfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		log.Fatalf("AWS config: %v", err)
	}
	smClient := secretsmanager.NewFromConfig(cfg)

	// Bootstrap session keys from Secrets Manager
	sessionStore, err := auth.BootstrapSessionKeys(ctx, smClient, "webapp/session-keys")
	if err != nil {
		log.Fatalf("Session keys: %v", err)
	}

	// Configure autocert with dual-layer cache
	cache := auth.NewSecretsManagerCache(smClient, "webapp/certs/", "/var/cache/autocert")
	certManager := &autocert.Manager{
		Prompt:     autocert.AcceptTOS,
		HostPolicy: autocert.HostWhitelist("app.example.io", "api.example.io"),
		Cache:      cache,
	}

	// Build application handler
	mux := http.NewServeMux()
	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		session, _ := sessionStore.Get(r, "app-session")
		// ... handle request with session
		_ = session
		w.Write([]byte("OK"))
	})

	// HTTPS server with autocert TLS
	server := &http.Server{
		Addr:    ":443",
		Handler: mux,
		TLSConfig: &tls.Config{
			GetCertificate: certManager.GetCertificate,
			MinVersion:     tls.VersionTLS12,
		},
	}

	// HTTP server for ACME challenges + redirect
	go http.ListenAndServe(":80", certManager.HTTPHandler(nil))

	log.Println("Starting HTTPS server on :443")
	log.Fatal(server.ListenAndServeTLS("", ""))
}

Key Rotation Procedure

package auth

import (
	"context"
	"encoding/json"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/service/secretsmanager"
)

// RotateSessionKeys generates a new key pair and demotes the current
// keys to the "previous" slot for graceful session migration.
func RotateSessionKeys(
	ctx context.Context,
	client *secretsmanager.Client,
	secretName string,
) error {
	// Fetch current keys
	result, err := client.GetSecretValue(ctx, &secretsmanager.GetSecretValueInput{
		SecretId: aws.String(secretName),
	})
	if err != nil {
		return err
	}

	var current SessionKeys
	json.Unmarshal([]byte(*result.SecretString), &current)

	// Rotate: current becomes previous, generate new current
	rotated := SessionKeys{
		SigningKey:        generateKey(64),
		EncryptionKey:     generateKey(32),
		PrevSigningKey:    current.SigningKey,
		PrevEncryptionKey: current.EncryptionKey,
	}

	payload, _ := json.Marshal(rotated)
	_, err = client.PutSecretValue(ctx, &secretsmanager.PutSecretValueInput{
		SecretId:     aws.String(secretName),
		SecretString: aws.String(string(payload)),
	})
	return err
}

Practical Applications

Zero-Downtime Deployments

When deploying new container instances, each instance fetches the same session keys from Secrets Manager. Users are never logged out during deployments because all instances share identical keys. Without centralized key storage, each container would generate its own keys, and a user whose request hits a different container would appear unauthenticated.

Multi-Region Deployments

Secrets Manager supports cross-region replication. For applications deployed across regions with global load balancing, replicating session keys ensures that a user authenticated in us-east-1 has a valid session when their next request routes to eu-west-1. The same applies to autocert certificates—a certificate obtained in one region is available in all regions without re-issuing.

Disaster Recovery

If a server’s local DirCache is lost (disk failure, container restart), the autocert cache falls through to Secrets Manager and recovers the certificate without hitting Let’s Encrypt again. This prevents rate limit exhaustion during recovery scenarios and eliminates the brief period of TLS unavailability that would occur while waiting for a new certificate issuance.

Common Pitfalls

• Hardcoding keys in environment variables: Storing session keys as plain-text environment variables in deployment configs or Docker Compose files makes them visible in process listings, container inspect output, and CI logs. Use Secrets Manager or a similar vault, and fetch keys at runtime.
• Forgetting the backfill path: If the application only reads from Secrets Manager and fails hard when the secret doesn’t exist, new environment provisioning requires a manual step to create the secret. Always include backfill logic that generates and stores keys when the secret is missing.
• Not rotating keys: Session keys should be rotated periodically (e.g., monthly) to limit the blast radius of a key compromise. Without the dual-key pattern (current + previous), rotation invalidates all active sessions and forces every user to re-login.
• Autocert rate limits: Let’s Encrypt enforces rate limits (50 certificates per registered domain per week). Without durable certificate caching, rapid container cycling (auto-scaling, frequent deploys) can exhaust the rate limit and leave new instances unable to obtain certificates. The Secrets Manager mirror prevents this.
• Missing error handling on Secrets Manager calls: Network issues, IAM permission errors, or Secrets Manager outages should not crash the application. Implement fallback behavior: if Secrets Manager is unreachable on startup, use the local DirCache for autocert and generate ephemeral session keys with a health check that reports degraded status.