Modern AI-powered applications rarely rely on a single model provider. Production systems need to integrate with OpenAI for GPT models, Anthropic for Claude, Google for Gemini, and local inference engines like Ollama—often simultaneously. Multi-provider AI configuration encompasses the architecture patterns, credential management, and runtime selection logic that allow an application to seamlessly switch between providers, models, and deployment targets without code changes.
The challenge goes beyond simple HTTP client wrappers. Each provider has a different API shape, authentication mechanism, token counting approach, rate limiting behavior, and streaming protocol. A well-designed multi-provider system abstracts these differences behind a unified interface while preserving provider-specific capabilities like function calling, vision inputs, or structured output modes. API keys must be stored securely—encrypted at rest—and the system must handle provider failures gracefully with fallback chains.
In a typical multi-provider AI application, users configure their preferred providers through a settings interface. API keys are encrypted before storage using AES-GCM encryption, and the provider selection logic routes requests to the appropriate backend based on model availability, cost, and user preference. This pattern enables the application to work with whichever AI services the user has access to, from enterprise OpenAI deployments to fully local Ollama instances.
The provider abstraction defines a common interface that all AI backends must implement. This typically includes methods for text completion, chat completion, embedding generation, and streaming responses. Each provider adapter translates between the unified interface and the provider-specific API format, handling differences in request/response schemas, error codes, and authentication headers.
A provider interface decouples application logic from specific AI services. The application code calls provider.Complete(prompt) without knowing whether the request goes to OpenAI, Anthropic, or a local model. This enables runtime provider switching, A/B testing between models, and graceful fallback when a provider is unavailable.
AI provider API keys are high-value secrets—a leaked OpenAI key can generate thousands of dollars in charges within minutes. Multi-provider systems must encrypt keys at rest using authenticated encryption (AES-256-GCM), derive encryption keys from user credentials or hardware-bound secrets, and never log or transmit keys in plaintext. The encryption boundary should be as tight as possible: decrypt only when making an API call, and clear the plaintext from memory immediately after.
Each provider offers multiple models with different capabilities, context windows, pricing tiers, and feature sets. A model registry maps model identifiers to their capabilities: maximum token count, support for vision/images, function calling, JSON mode, and streaming. This registry drives the UI (showing only compatible models for a given task) and the runtime (rejecting requests that exceed a model’s capabilities before making an API call).
Instead of hardcoding model names, select models by required capabilities: “I need a model that supports function calling with at least 32K context.” The registry returns the best available model matching those constraints, automatically adapting as providers release new models or deprecate old ones.
Provider outages are common—OpenAI rate limits, Anthropic has capacity issues, local models run out of GPU memory. A fallback chain defines an ordered list of providers to try for a given request. If the primary provider fails (timeout, rate limit, 5xx error), the system automatically retries with the next provider in the chain. Fallback decisions must account for latency budgets, cost differences, and capability compatibility.
The core abstraction defines what every AI provider must support:
// Provider defines the interface all AI backends must implement.
type Provider interface {
// Name returns the provider identifier (e.g., "openai", "anthropic", "ollama").
Name() string
// Models returns the list of available models for this provider.
Models() []ModelInfo
// Complete sends a completion request and returns the full response.
Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error)
// Stream sends a completion request and returns a stream of response chunks.
Stream(ctx context.Context, req *CompletionRequest) (ResponseStream, error)
// Embed generates embeddings for the given texts.
Embed(ctx context.Context, texts []string, model string) ([][]float32, error)
}
type ModelInfo struct {
ID string `json:"id"`
Name string `json:"name"`
Provider string `json:"provider"`
MaxTokens int `json:"max_tokens"`
ContextWindow int `json:"context_window"`
Capabilities []string `json:"capabilities"` // "chat", "vision", "function_calling", "json_mode"
CostPer1KIn float64 `json:"cost_per_1k_input"`
CostPer1KOut float64 `json:"cost_per_1k_output"`
}
type CompletionRequest struct {
Model string `json:"model"`
Messages []Message `json:"messages"`
Temperature float64 `json:"temperature,omitempty"`
MaxTokens int `json:"max_tokens,omitempty"`
Tools []ToolDefinition `json:"tools,omitempty"`
Stream bool `json:"stream,omitempty"`
}
type CompletionResponse struct {
Content string `json:"content"`
Model string `json:"model"`
Usage UsageStats `json:"usage"`
ToolCalls []ToolCall `json:"tool_calls,omitempty"`
FinishReason string `json:"finish_reason"`
}Keys are encrypted before being persisted to configuration files or databases:
import (
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"fmt"
"io"
)
// KeyEncryptor handles encryption and decryption of API keys.
type KeyEncryptor struct {
gcm cipher.AEAD
}
// NewKeyEncryptor creates an encryptor from a passphrase.
// The passphrase is hashed with SHA-256 to produce a 256-bit AES key.
func NewKeyEncryptor(passphrase string) (*KeyEncryptor, error) {
hash := sha256.Sum256([]byte(passphrase))
block, err := aes.NewCipher(hash[:])
if err != nil {
return nil, fmt.Errorf("creating cipher: %w", err)
}
gcm, err := cipher.NewGCM(block)
if err != nil {
return nil, fmt.Errorf("creating GCM: %w", err)
}
return &KeyEncryptor{gcm: gcm}, nil
}
// Encrypt encrypts an API key, returning a base64-encoded ciphertext.
func (e *KeyEncryptor) Encrypt(apiKey string) (string, error) {
nonce := make([]byte, e.gcm.NonceSize())
if _, err := io.ReadFull(rand.Reader, nonce); err != nil {
return "", fmt.Errorf("generating nonce: %w", err)
}
ciphertext := e.gcm.Seal(nonce, nonce, []byte(apiKey), nil)
return base64.StdEncoding.EncodeToString(ciphertext), nil
}
// Decrypt decrypts a base64-encoded ciphertext back to the original API key.
func (e *KeyEncryptor) Decrypt(encrypted string) (string, error) {
data, err := base64.StdEncoding.DecodeString(encrypted)
if err != nil {
return "", fmt.Errorf("decoding base64: %w", err)
}
nonceSize := e.gcm.NonceSize()
if len(data) < nonceSize {
return "", fmt.Errorf("ciphertext too short")
}
nonce, ciphertext := data[:nonceSize], data[nonceSize:]
plaintext, err := e.gcm.Open(nil, nonce, ciphertext, nil)
if err != nil {
return "", fmt.Errorf("decryption failed: %w", err)
}
return string(plaintext), nil
}AES-GCM provides both confidentiality and integrity. The authentication tag ensures that tampered ciphertext is rejected during decryption, preventing an attacker from modifying encrypted keys without detection. Always use authenticated encryption modes (GCM, ChaCha20-Poly1305) rather than raw AES-CBC or AES-CTR, which provide confidentiality but not integrity.
A registry maps provider names to their constructors, enabling runtime instantiation:
type ProviderConfig struct {
Name string `json:"name"`
APIKey string `json:"api_key"` // encrypted
BaseURL string `json:"base_url"` // custom endpoints (Azure, proxies)
OrgID string `json:"org_id"` // optional org/project scoping
MaxRetries int `json:"max_retries"`
TimeoutSecs int `json:"timeout_secs"`
}
type ProviderFactory func(cfg ProviderConfig, enc *KeyEncryptor) (Provider, error)
type ProviderRegistry struct {
mu sync.RWMutex
factories map[string]ProviderFactory
instances map[string]Provider
encryptor *KeyEncryptor
}
func NewProviderRegistry(enc *KeyEncryptor) *ProviderRegistry {
r := &ProviderRegistry{
factories: make(map[string]ProviderFactory),
instances: make(map[string]Provider),
encryptor: enc,
}
// Register built-in providers
r.Register("openai", NewOpenAIProvider)
r.Register("anthropic", NewAnthropicProvider)
r.Register("gemini", NewGeminiProvider)
r.Register("ollama", NewOllamaProvider)
return r
}
func (r *ProviderRegistry) Register(name string, factory ProviderFactory) {
r.mu.Lock()
defer r.mu.Unlock()
r.factories[name] = factory
}
func (r *ProviderRegistry) Get(cfg ProviderConfig) (Provider, error) {
r.mu.RLock()
if p, ok := r.instances[cfg.Name]; ok {
r.mu.RUnlock()
return p, nil
}
r.mu.RUnlock()
r.mu.Lock()
defer r.mu.Unlock()
// Double-check after acquiring write lock
if p, ok := r.instances[cfg.Name]; ok {
return p, nil
}
factory, ok := r.factories[cfg.Name]
if !ok {
return nil, fmt.Errorf("unknown provider: %s", cfg.Name)
}
p, err := factory(cfg, r.encryptor)
if err != nil {
return nil, fmt.Errorf("initializing provider %s: %w", cfg.Name, err)
}
r.instances[cfg.Name] = p
return p, nil
}In the platform, users configure their AI providers through a settings panel. Each provider entry includes the API key (encrypted client-side before storage), preferred model, and optional custom endpoint URL. The settings UI queries each provider’s model list to populate dropdowns, only showing models the user’s API key has access to. This approach lets enterprise users point to Azure OpenAI endpoints while personal users connect directly to OpenAI’s API.
Different tasks have different quality and cost requirements. A summarization task might use a cheaper model (GPT-4o-mini, Haiku) while a complex reasoning task routes to a more capable model (GPT-4o, Opus). The multi-provider system enables cost-aware routing: attach a cost budget to each request, and the router selects the cheapest model that meets the capability requirements. Usage tracking aggregates costs across providers to enforce organizational spending limits.
Some workloads benefit from a hybrid approach: use local Ollama models for high-volume, latency-insensitive tasks (bulk document summarization, classification) and cloud providers for low-volume, quality-critical tasks (user-facing chat, complex analysis). The multi-provider architecture makes this transparent—the routing layer selects local or cloud based on the request parameters without the calling code needing to know the difference.
Track latency, error rates, and availability per provider. When a provider’s error rate exceeds a threshold, automatically route traffic to healthy alternatives. Expose provider health metrics to operators via dashboards, enabling proactive capacity planning and vendor negotiations based on real reliability data.
Combining fallback chains with circuit breakers prevents cascading failures:
type FallbackRouter struct {
providers []Provider
breakers map[string]*CircuitBreaker
}
type CircuitBreaker struct {
mu sync.Mutex
failures int
threshold int
resetAfter time.Duration
lastFailure time.Time
state string // "closed", "open", "half-open"
}
func (cb *CircuitBreaker) Allow() bool {
cb.mu.Lock()
defer cb.mu.Unlock()
switch cb.state {
case "closed":
return true
case "open":
if time.Since(cb.lastFailure) > cb.resetAfter {
cb.state = "half-open"
return true
}
return false
case "half-open":
return true
}
return false
}
func (cb *CircuitBreaker) RecordSuccess() {
cb.mu.Lock()
defer cb.mu.Unlock()
cb.failures = 0
cb.state = "closed"
}
func (cb *CircuitBreaker) RecordFailure() {
cb.mu.Lock()
defer cb.mu.Unlock()
cb.failures++
cb.lastFailure = time.Now()
if cb.failures >= cb.threshold {
cb.state = "open"
}
}
func (r *FallbackRouter) Complete(ctx context.Context, req *CompletionRequest) (*CompletionResponse, error) {
var lastErr error
for _, p := range r.providers {
cb := r.breakers[p.Name()]
if !cb.Allow() {
continue // skip providers with open circuit breakers
}
resp, err := p.Complete(ctx, req)
if err == nil {
cb.RecordSuccess()
return resp, nil
}
cb.RecordFailure()
lastErr = fmt.Errorf("provider %s: %w", p.Name(), err)
}
return nil, fmt.Errorf("all providers failed, last error: %w", lastErr)
}Provider configurations are serialized to JSON with encrypted API keys:
type AISettings struct {
Providers []ProviderConfig `json:"providers"`
DefaultModel string `json:"default_model"`
FallbackOrder []string `json:"fallback_order"`
MaxCostPerDay float64 `json:"max_cost_per_day"`
}
func SaveSettings(path string, settings *AISettings, enc *KeyEncryptor) error {
// Encrypt all API keys before serialization
for i := range settings.Providers {
if settings.Providers[i].APIKey != "" &&
!strings.HasPrefix(settings.Providers[i].APIKey, "enc:") {
encrypted, err := enc.Encrypt(settings.Providers[i].APIKey)
if err != nil {
return fmt.Errorf("encrypting key for %s: %w",
settings.Providers[i].Name, err)
}
settings.Providers[i].APIKey = "enc:" + encrypted
}
}
data, err := json.MarshalIndent(settings, "", " ")
if err != nil {
return fmt.Errorf("marshaling settings: %w", err)
}
return os.WriteFile(path, data, 0600) // restrictive permissions
}Encrypt API keys at the application layer (AES-GCM), write config files with restrictive permissions (0600), and use OS-level secret storage (macOS Keychain, Windows Credential Manager, Linux Secret Service) when available. No single layer is sufficient—encryption protects against file theft, permissions protect against local privilege escalation, and OS secret stores protect against application-level vulnerabilities.
http.Client with a configured Transport that sets MaxIdleConnsPerHost appropriately for your concurrency level.Retry-After header when available. Cap the maximum retry delay to prevent requests from hanging indefinitely.Authorization header contains the API key in plaintext. Always configure your HTTP client to redact sensitive headers from debug logs, and never log the full request/response in production.context.WithTimeout and configure HTTP client timeouts as a safety net.