Every API that faces the internet needs rate limiting. Without it, a single misbehaving client—whether a buggy integration, a scraper, or an attacker—can consume enough resources to degrade service for everyone. Rate limiting protects backend systems by bounding the number of requests any given client can make within a time window, and it does so without requiring the client to be well-behaved.
The platform implements rate limiting at the API gateway layer, applying configurable thresholds per user, per project, and per IP address. Different endpoint categories (read-heavy queries vs. write-heavy mutations vs. expensive search operations) carry different limits, reflecting the actual cost each request imposes on backend infrastructure. The system uses token bucket algorithms backed by Redis for distributed enforcement across multiple gateway instances.
Getting rate limiting right means choosing the correct algorithm for your traffic pattern, deciding where to enforce limits (edge vs. application), and providing clear feedback to clients when they hit a limit. A rate limiter that silently drops requests or returns confusing errors is worse than no rate limiter at all, because it makes debugging impossible for the clients you’re trying to protect.
The token bucket is the most widely used rate limiting algorithm. A bucket holds up to N tokens and refills at a steady rate (e.g., 10 tokens per second). Each request consumes one token. If the bucket is empty, the request is rejected. This naturally allows short bursts (up to the bucket capacity) while enforcing a sustained average rate, making it ideal for APIs where bursty traffic is normal.
Sliding window rate limiting divides time into fixed windows (e.g., 1-minute intervals) and counts requests in each window. The “sliding” variant interpolates between the current and previous window based on the elapsed time, avoiding the boundary spike problem where a client could send 2x the limit by timing requests around a window boundary.
Real-world rate limiting rarely uses a single key. The platform applies limits along multiple dimensions simultaneously: per-user (authenticated identity), per-project (organizational scope), and per-IP (unauthenticated fallback). A request must pass all applicable limits. This prevents a single user from consuming an entire project’s quota, and prevents a single IP from overwhelming the system before authentication.
When a request is rate-limited, the response must include a Retry-After header telling the client how many seconds to wait before retrying. Combined with an HTTP 429 status code, this gives well-behaved clients everything they need to back off gracefully. Without this header, clients either retry immediately (making things worse) or give up entirely (degrading user experience).
The rate limiter sits in the HTTP middleware chain, executing before the request reaches the handler. On each incoming request, the middleware extracts the rate key (user ID, project ID, or IP), queries the current token count from Redis, and either allows or rejects the request. The entire check-and-decrement operation is atomic, implemented as a Lua script running inside Redis to avoid race conditions under concurrent access.
The token bucket state is stored in Redis as two fields: the current token count and the timestamp of the last refill. When a request arrives, the Lua script calculates how many tokens should have been added since the last refill, caps at the bucket maximum, then attempts to consume one token. If successful, the request proceeds. If the bucket is empty, the script returns the time until the next token is available, which becomes the Retry-After value.
-- Redis Lua script for token bucket rate limiting
local key = KEYS[1]
local max_tokens = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2]) -- tokens per second
local now = tonumber(ARGV[3])
local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
local tokens = tonumber(bucket[1]) or max_tokens
local last_refill = tonumber(bucket[2]) or now
-- Calculate tokens to add since last refill
local elapsed = now - last_refill
local new_tokens = math.min(max_tokens, tokens + (elapsed * refill_rate))
if new_tokens >= 1 then
-- Consume one token
redis.call('HMSET', key, 'tokens', new_tokens - 1, 'last_refill', now)
redis.call('EXPIRE', key, math.ceil(max_tokens / refill_rate) + 1)
return {1, new_tokens - 1} -- allowed, remaining tokens
else
-- Calculate wait time for next token
local wait = (1 - new_tokens) / refill_rate
return {0, wait} -- denied, retry after seconds
endThe rate limiting middleware wraps any http.Handler and injects itself into the middleware chain. It extracts the rate key from the request context (populated by earlier auth middleware), calls the Redis-backed limiter, and either passes the request through or returns a 429 response with appropriate headers.
package middleware
import (
"net/http"
"strconv"
"time"
)
type RateLimiter struct {
store RateLimitStore
config RateLimitConfig
}
type RateLimitConfig struct {
MaxTokens int
RefillRate float64 // tokens per second
KeyFunc func(r *http.Request) string
}
func (rl *RateLimiter) Middleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
key := rl.config.KeyFunc(r)
result, err := rl.store.Allow(r.Context(), key, rl.config.MaxTokens, rl.config.RefillRate)
if err != nil {
// If Redis is down, fail open to avoid blocking all traffic
next.ServeHTTP(w, r)
return
}
w.Header().Set("X-RateLimit-Limit", strconv.Itoa(rl.config.MaxTokens))
w.Header().Set("X-RateLimit-Remaining", strconv.Itoa(result.Remaining))
if !result.Allowed {
w.Header().Set("Retry-After", strconv.Itoa(int(result.RetryAfter.Seconds())+1))
w.Header().Set("X-RateLimit-Reset", strconv.FormatInt(time.Now().Add(result.RetryAfter).Unix(), 10))
http.Error(w, "rate limit exceeded", http.StatusTooManyRequests)
return
}
next.ServeHTTP(w, r)
})
}Not all endpoints cost the same. A simple key-value lookup is orders of magnitude cheaper than a full-text search query. The platform defines rate limit tiers by endpoint category, applying stricter limits to expensive operations:
In the platform, rate limiting serves multiple purposes beyond simple abuse prevention. The search endpoint limiter protects the underlying vector index from overload, since each search query triggers embedding generation and approximate nearest neighbor lookups. The auth endpoint limiter prevents brute-force credential stuffing. The write endpoint limiter prevents accidental bulk operations from overwhelming the ingestion pipeline.
The per-project rate limit is particularly important for multi-tenant SaaS. Each project (tenant) gets a quota based on their billing tier. The rate limiter enforces these quotas in real time, and usage data feeds into the metered billing system. When a project approaches its limit, the API returns warning headers so the client can throttle proactively before hitting the hard limit.
When the rate limit backend (Redis) is unavailable, you must choose: reject all requests (fail-closed) or allow all requests (fail-open). For most APIs, fail-open is the right default. A Redis outage is temporary, and blocking all traffic during that window causes more damage than allowing a brief period of unlimited requests. Reserve fail-closed for security-critical endpoints like authentication, where allowing unlimited attempts is dangerous.
When running multiple API gateway instances, rate limits must be enforced centrally. Each instance queries the same Redis cluster, and the Lua script’s atomicity ensures that concurrent requests from different instances don’t overshoot the limit. For extreme throughput, you can shard rate limit keys across Redis nodes using consistent hashing on the rate key.
type DistributedLimiter struct {
ring *redis.Ring // Consistent hash ring across Redis nodes
}
func (dl *DistributedLimiter) Allow(ctx context.Context, key string, max int, rate float64) (*Result, error) {
// The ring automatically routes to the correct shard based on the key
client := dl.ring.ForKey(ctx, key)
result, err := client.EvalSha(ctx, tokenBucketSHA, []string{key},
max, rate, float64(time.Now().UnixMicro())/1e6,
).Int64Slice()
if err != nil {
return nil, fmt.Errorf("rate limit check failed: %w", err)
}
return &Result{
Allowed: result[0] == 1,
Remaining: int(result[1]),
RetryAfter: time.Duration(result[1]) * time.Millisecond,
}, nil
}Always include rate limit headers on every response, not just on 429 responses. This lets clients monitor their remaining quota and adjust their request rate before hitting the limit. The standard headers are X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset.
Retry-After forces clients to guess when to retry. Most will retry too aggressively, creating a thundering herd when the window resets. Always include the header.