Ollama is an open-source tool for running large language models locally. It packages model weights, tokenizer configurations, and runtime settings into a single distributable format, then serves them through an OpenAI-compatible REST API on localhost:11434. This eliminates the need for cloud API keys, removes per-token costs, and keeps all data on the local machine—critical for privacy-sensitive workloads and air-gapped environments.
Under the hood, Ollama uses llama.cpp for inference, a highly optimized C++ inference engine that supports quantized models (4-bit, 5-bit, 8-bit) running on CPU, Apple Metal, NVIDIA CUDA, and AMD ROCm. Ollama wraps this complexity behind a Docker-like CLI: ollama pull llama3 downloads a model, ollama run llama3 starts an interactive chat, and the REST API enables programmatic access from any language. Models are stored in a local registry (~/.ollama/models) with layer-based deduplication similar to container images.
Ollama can serve dual roles in an AI-powered application: as a backend for local LLM inference (chat, summarization, analysis) without cloud API dependencies, and as an embedding endpoint to generate text embeddings with models like nomic-embed-text for local vector search. This dual-purpose usage—both generation and embedding—makes Ollama a single dependency that replaces multiple cloud API integrations for development and offline deployments.
Ollama uses a container-image-like model for distributing LLMs. Each model is defined by a Modelfile (analogous to a Dockerfile) that specifies the base weights, system prompt, template format, and inference parameters. Models are pulled from the Ollama registry, stored locally with layer deduplication, and can be customized by creating derivative Modelfiles. The ollama list command shows all locally available models with their sizes and modification dates.
Ollama models come in multiple quantization levels that trade quality for resource usage. Q4_0 (4-bit) reduces a 7B-parameter model from 14 GB (FP16) to ~4 GB, making it runnable on consumer hardware. Q5_K_M and Q8_0 offer progressively better quality at higher memory cost. For most tasks, Q4_K_M provides the best balance of quality and resource usage. Always benchmark your specific use case—quantization impacts different tasks (code, math, creative writing) differently.
Ollama exposes a REST API on port 11434 with endpoints mirroring OpenAI’s API structure. The key endpoints are: /api/generate for text completion, /api/chat for multi-turn chat with message history, /api/embeddings for generating vector embeddings, /api/tags for listing available models, and /api/show for model metadata. Responses stream by default as newline-delimited JSON, enabling real-time token-by-token display.
Local models have fixed context windows (typically 2K–128K tokens depending on the model). Ollama manages context by maintaining a KV cache in memory for each active conversation. When the context fills up, older messages are evicted. The num_ctx parameter controls the context window size—larger windows use more memory (roughly 1 GB per 4K tokens for a 7B model). Applications must track token usage and manage context truncation to stay within the model’s window.
The first request to a model loads it into memory (GPU VRAM or system RAM), which takes several seconds for large models. Subsequent requests are fast because the model stays resident. Ollama’s keep_alive parameter controls how long an idle model stays loaded (default: 5 minutes). Set keep_alive: -1 to keep models loaded indefinitely for latency-sensitive applications, or keep_alive: 0 to unload immediately after each request for memory-constrained environments.
Beyond text generation, Ollama serves embedding models that convert text into dense vector representations for semantic search. Models like nomic-embed-text (768 dimensions, ~270 MB) and all-minilm (384 dimensions, ~45 MB) generate embeddings locally at speeds comparable to cloud APIs for moderate batch sizes. The embedding endpoint accepts a single text and returns a float array, making it straightforward to integrate with vector databases and search engines.
A typed Go client wrapping Ollama’s REST API for chat and embedding generation:
package ollama
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"time"
)
type Client struct {
baseURL string
httpClient *http.Client
}
func NewClient(baseURL string) *Client {
if baseURL == "" {
baseURL = "http://localhost:11434"
}
return &Client{
baseURL: baseURL,
httpClient: &http.Client{
Timeout: 5 * time.Minute, // LLM responses can be slow
},
}
}
type ChatRequest struct {
Model string `json:"model"`
Messages []Message `json:"messages"`
Stream bool `json:"stream"`
Options *Options `json:"options,omitempty"`
}
type Message struct {
Role string `json:"role"` // "system", "user", "assistant"
Content string `json:"content"`
}
type Options struct {
Temperature float64 `json:"temperature,omitempty"`
NumCtx int `json:"num_ctx,omitempty"`
NumPredict int `json:"num_predict,omitempty"`
TopP float64 `json:"top_p,omitempty"`
Seed int `json:"seed,omitempty"`
}
type ChatResponse struct {
Model string `json:"model"`
Message Message `json:"message"`
Done bool `json:"done"`
TotalDuration int64 `json:"total_duration"`
EvalCount int `json:"eval_count"`
EvalDuration int64 `json:"eval_duration"`
}
// Chat sends a chat completion request and returns the full response.
func (c *Client) Chat(ctx context.Context, req ChatRequest) (*ChatResponse, error) {
req.Stream = false
body, err := json.Marshal(req)
if err != nil {
return nil, fmt.Errorf("marshaling request: %w", err)
}
httpReq, err := http.NewRequestWithContext(ctx, "POST",
c.baseURL+"/api/chat", bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("creating request: %w", err)
}
httpReq.Header.Set("Content-Type", "application/json")
resp, err := c.httpClient.Do(httpReq)
if err != nil {
return nil, fmt.Errorf("sending request: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
errBody, _ := io.ReadAll(resp.Body)
return nil, fmt.Errorf("ollama error (status %d): %s",
resp.StatusCode, string(errBody))
}
var chatResp ChatResponse
if err := json.NewDecoder(resp.Body).Decode(&chatResp); err != nil {
return nil, fmt.Errorf("decoding response: %w", err)
}
return &chatResp, nil
}For interactive applications, stream tokens as they are generated:
// StreamChat sends a streaming chat request, calling onChunk for each token.
func (c *Client) StreamChat(ctx context.Context, req ChatRequest,
onChunk func(ChatResponse)) error {
req.Stream = true
body, err := json.Marshal(req)
if err != nil {
return fmt.Errorf("marshaling request: %w", err)
}
httpReq, err := http.NewRequestWithContext(ctx, "POST",
c.baseURL+"/api/chat", bytes.NewReader(body))
if err != nil {
return fmt.Errorf("creating request: %w", err)
}
httpReq.Header.Set("Content-Type", "application/json")
resp, err := c.httpClient.Do(httpReq)
if err != nil {
return fmt.Errorf("sending request: %w", err)
}
defer resp.Body.Close()
decoder := json.NewDecoder(resp.Body)
for {
var chunk ChatResponse
if err := decoder.Decode(&chunk); err != nil {
if err == io.EOF {
return nil
}
return fmt.Errorf("decoding chunk: %w", err)
}
onChunk(chunk)
if chunk.Done {
return nil
}
}
}
// Usage example:
// var fullResponse strings.Builder
// err := client.StreamChat(ctx, req, func(chunk ChatResponse) {
// fmt.Print(chunk.Message.Content) // print tokens as they arrive
// fullResponse.WriteString(chunk.Message.Content)
// })Ollama streams responses as newline-delimited JSON (NDJSON): each line is a complete JSON object representing one token or a metadata payload. The final chunk has "done": true and includes aggregate statistics (total duration, eval count, tokens per second). This format is simpler than Server-Sent Events (SSE) and works naturally with Go’s json.Decoder, which reads one JSON value at a time from a stream.
Generate vector embeddings for semantic search:
type EmbedRequest struct {
Model string `json:"model"`
Prompt string `json:"prompt"`
}
type EmbedResponse struct {
Embedding []float64 `json:"embedding"`
}
// Embed generates an embedding vector for the given text.
func (c *Client) Embed(ctx context.Context, model, text string) ([]float64, error) {
body, err := json.Marshal(EmbedRequest{
Model: model,
Prompt: text,
})
if err != nil {
return nil, fmt.Errorf("marshaling request: %w", err)
}
httpReq, err := http.NewRequestWithContext(ctx, "POST",
c.baseURL+"/api/embeddings", bytes.NewReader(body))
if err != nil {
return nil, fmt.Errorf("creating request: %w", err)
}
httpReq.Header.Set("Content-Type", "application/json")
resp, err := c.httpClient.Do(httpReq)
if err != nil {
return nil, fmt.Errorf("sending request: %w", err)
}
defer resp.Body.Close()
var embedResp EmbedResponse
if err := json.NewDecoder(resp.Body).Decode(&embedResp); err != nil {
return nil, fmt.Errorf("decoding response: %w", err)
}
return embedResp.Embedding, nil
}
// EmbedBatch generates embeddings for multiple texts sequentially.
// Ollama does not natively support batch embedding, so we iterate.
func (c *Client) EmbedBatch(ctx context.Context, model string,
texts []string) ([][]float64, error) {
results := make([][]float64, len(texts))
for i, text := range texts {
emb, err := c.Embed(ctx, model, text)
if err != nil {
return nil, fmt.Errorf("embedding text %d: %w", i, err)
}
results[i] = emb
}
return results, nil
}Ollama enables developers to build and test AI-powered features without provisioning API keys or incurring cloud costs. A dashboard prototype application uses Ollama as the default development backend—developers run ollama pull llama3 once and have a fully functional AI backend. This removes onboarding friction (no API key setup) and eliminates the risk of accidentally burning through API credits during development and testing.
A local semantic search platform uses Ollama’s embedding endpoint with nomic-embed-text to build a fully offline semantic search system. Documents are embedded at index time and stored in a local vector database. At query time, the search query is embedded using the same Ollama model and compared against stored vectors. The entire pipeline runs without internet connectivity, making it suitable for air-gapped enterprise deployments.
Organizations handling PII, medical records, financial data, or classified information cannot send this data to third-party APIs. Ollama runs entirely on-premises, ensuring data never leaves the machine. This satisfies compliance requirements (HIPAA, SOC 2, GDPR) without sacrificing AI capabilities. The trade-off is reduced model quality compared to frontier cloud models, but for many tasks (classification, extraction, summarization), local models are sufficient.
AI-powered applications need integration tests that verify LLM interactions without depending on cloud API availability or rate limits. Running Ollama in CI (via Docker: docker run -d ollama/ollama) provides a deterministic, cost-free test backend. Use small models (TinyLlama, Phi-2) and fixed random seeds for reproducible test outputs.
Verify Ollama is running and required models are available at application startup:
type ModelRequirement struct {
Name string
MinSize int64 // minimum size in bytes to verify correct variant
}
// EnsureModels checks that all required models are available locally.
// Pulls missing models automatically.
func (c *Client) EnsureModels(ctx context.Context,
required []ModelRequirement) error {
// First, check if Ollama is reachable
healthReq, _ := http.NewRequestWithContext(ctx, "GET",
c.baseURL+"/api/tags", nil)
resp, err := c.httpClient.Do(healthReq)
if err != nil {
return fmt.Errorf("ollama not reachable at %s: %w", c.baseURL, err)
}
defer resp.Body.Close()
var tagsResp struct {
Models []struct {
Name string `json:"name"`
Size int64 `json:"size"`
} `json:"models"`
}
json.NewDecoder(resp.Body).Decode(&tagsResp)
available := make(map[string]int64)
for _, m := range tagsResp.Models {
available[m.Name] = m.Size
}
for _, req := range required {
if _, ok := available[req.Name]; !ok {
fmt.Printf("Pulling model %s...\n", req.Name)
if err := c.pullModel(ctx, req.Name); err != nil {
return fmt.Errorf("pulling model %s: %w", req.Name, err)
}
}
}
return nil
}
func (c *Client) pullModel(ctx context.Context, name string) error {
body, _ := json.Marshal(map[string]any{
"name": name,
"stream": false,
})
req, _ := http.NewRequestWithContext(ctx, "POST",
c.baseURL+"/api/pull", bytes.NewReader(body))
req.Header.Set("Content-Type", "application/json")
resp, err := c.httpClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
errBody, _ := io.ReadAll(resp.Body)
return fmt.Errorf("pull failed (status %d): %s",
resp.StatusCode, string(errBody))
}
return nil
}Create specialized models with custom system prompts and parameters:
# Modelfile for a code review assistant
FROM llama3
SYSTEM """
You are a senior Go developer performing code reviews. Focus on:
- Concurrency safety (goroutine leaks, race conditions, proper sync)
- Error handling (wrapped errors, sentinel values, error types)
- Performance (unnecessary allocations, buffer reuse, slice pre-allocation)
- API design (interface segregation, option patterns, context propagation)
Be specific and reference line numbers. Suggest fixes, not just problems.
"""
PARAMETER temperature 0.3
PARAMETER num_ctx 8192
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1# Create the custom model:
# ollama create code-reviewer -f Modelfile
# Use it:
# ollama run code-reviewer "Review this Go code: ..."Modelfiles work like Dockerfiles: FROM llama3 inherits the base model’s weights, then SYSTEM, PARAMETER, and TEMPLATE directives add customization layers. Creating a derivative model with ollama create only stores the delta (system prompt and parameters), not a copy of the weights. This means you can create dozens of specialized variants of the same base model with negligible additional disk usage.
ollama ps to verify models are loaded on the GPU, not CPU. A model that fits entirely in VRAM runs 5–20x faster than CPU inference.OLLAMA_NUM_PARALLEL or deploy behind a load balancer with multiple Ollama instances. Note that each concurrent slot doubles the KV cache memory requirement.num_ctx: 32768 on a 7B model requires ~8 GB just for context, plus ~4 GB for weights. Monitor actual memory usage, not just model file size.nomic-embed-text can embed ~100 texts per second on a modern CPU, or ~500/second on GPU. For bulk indexing, batch requests to amortize HTTP overhead./api/tags and provide a clear error message directing the user to install or start Ollama./api/chat endpoint, but the /api/generate endpoint sends raw text. Using the wrong format with /api/generate produces garbled output. Prefer the chat endpoint unless you need raw prompt control.nomic-embed-text and later switch to all-minilm, all existing embeddings become incompatible (different dimensions, different vector spaces). Track which embedding model was used for each index and re-embed everything when changing models.keep_alive: 0 on infrequently used models to free memory between requests.