lebsral/dspy-ollama
skills/dspy-ollama/SKILL.md
Run DSPy with local models using Ollama — no API key needed. Use when you want to run DSPy locally, use Ollama, set up a local LLM, run offline, or configure local model parameters. Also used for ollama, local model, run LLM locally, llama local, self-hosted LLM, ollama serve, ollama_chat, local inference, run DSPy offline, no API key needed, ollama pull, ollama list, ollama rm, num_ctx, ollama context window, ollama GPU, OLLAMA_NUM_GPU, OLLAMA_HOST, ollama remote, ollama embeddings, nomic-embed-text, dspy.Embedder ollama, which local model, best model for ollama, ollama too slow, ollama vs vllm, develop locally deploy remotely, ollama environment variables, ollama systemd, ollama background service.
$ install --global
skillsauthnpx skillsauth add lebsral/dspy-programming-not-prompting-lms-skills dspy-ollamaInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: May 7, 2026, 7:00 AM129.7s4 files scanned
SKILL.md
- name:
- dspy-ollama
- description:
- Run DSPy with local models using Ollama — no API key needed. Use when you want to run DSPy locally, use Ollama, set up a local LLM, run offline, or configure local model parameters. Also used for ollama, local model, run LLM locally, llama local, self-hosted LLM, ollama serve, ollama_chat, local inference, run DSPy offline, no API key needed, ollama pull, ollama list, ollama rm, num_ctx, ollama context window, ollama GPU, OLLAMA_NUM_GPU, OLLAMA_HOST, ollama remote, ollama embeddings, nomic-embed-text, dspy.Embedder ollama, which local model, best model for ollama, ollama too slow, ollama vs vllm, develop locally deploy remotely, ollama environment variables, ollama systemd, ollama background service.
Ollama — Run DSPy with Local Models
Guide the user through running DSPy with local models via Ollama. No API keys, no cloud costs, full privacy.
What is Ollama
Ollama is a local LLM runner (166k+ GitHub stars) that wraps llama.cpp. It downloads, manages, and serves models locally with a simple CLI. DSPy connects to it through LiteLLM's ollama_chat/ provider.
Setup
Install Ollama
# macOS
brew install ollama
# Linux
curl -fsSL https://ollama.com/install.sh | sh
# Windows: download from ollama.com
Start the server and pull a model
# Start the Ollama server (runs in background)
ollama serve
# Pull a model (one-time download)
ollama pull llama3.1
# Quick test
ollama run llama3.1 "What is DSPy?"
Connect DSPy to Ollama
import dspy
lm = dspy.LM(
"ollama_chat/llama3.1",
api_base="http://localhost:11434",
api_key="", # required but ignored
temperature=0.7,
num_ctx=8192, # IMPORTANT: set context window explicitly
)
dspy.configure(lm=lm)
# Test it
classify = dspy.Predict("text -> sentiment")
result = classify(text="DSPy makes AI development easier")
print(result.sentiment)
Note: dspy.OllamaLocal is deprecated. Use dspy.LM("ollama_chat/...") instead.
Model selection guide
| Model | Sizes | Context | Good for | Notes | |-------|-------|---------|----------|-------| | Llama 3.1 | 8B, 70B | 128K | General purpose, instruction following | Best all-rounder | | Llama 3.2 | 1B, 3B | 128K | Edge, mobile, lightweight tasks | Very fast, less capable | | Qwen 2.5 | 0.5B–72B | 128K | Multilingual, coding, math | Strong on benchmarks | | Qwen 3 | 0.6B–32B | 128K | Reasoning, multilingual | Latest, thinking mode | | Mistral | 7B | 32K | Fast general purpose | Good speed/quality tradeoff | | Phi-4 | 14B | 16K | Reasoning, STEM, code | Small but capable | | Gemma 2 | 2B, 9B, 27B | 8K | Lightweight, fast | Google, good quality/size ratio | | DeepSeek-R1 | 1.5B–70B | 128K | Complex reasoning | Distilled reasoning chains | | CodeLlama | 7B, 13B, 34B | 16K | Code generation | Fine-tuned for code |
Quick recommendations
Prototyping (fast iteration, good quality):
→ llama3.1:8b or qwen2.5:7b
Best quality on consumer hardware (16GB+ RAM):
→ llama3.1:8b or phi4:14b
Complex reasoning:
→ deepseek-r1:14b or qwen3:14b
Coding tasks:
→ qwen2.5-coder:7b or codellama:13b
Minimal resources (8GB RAM):
→ llama3.2:3b or gemma2:2b or qwen2.5:3b
Context window gotcha (critical)
Ollama defaults to 4096 tokens regardless of the model's actual capacity. This is the #1 source of issues when running DSPy with Ollama. DSPy prompts with few-shot demos can easily exceed 4096 tokens.
Always set num_ctx explicitly:
# BAD — defaults to 4096 tokens, will silently truncate
lm = dspy.LM("ollama_chat/llama3.1", api_base="http://localhost:11434", api_key="")
# GOOD — set context window to match model capability
lm = dspy.LM(
"ollama_chat/llama3.1",
api_base="http://localhost:11434",
api_key="",
num_ctx=8192, # 8K is a safe default for most tasks
)
Larger context = more VRAM. If you get OOM errors, reduce num_ctx:
| num_ctx | VRAM overhead (approx) | When to use | |---------|----------------------:|-------------| | 4096 | Baseline | Simple classification, short prompts | | 8192 | +2-4 GB | Most DSPy tasks, few-shot demos | | 16384 | +4-8 GB | RAG with long contexts | | 32768 | +8-16 GB | Long document processing |
Environment variables
| Variable | Default | Description |
|----------|---------|-------------|
| OLLAMA_HOST | 127.0.0.1:11434 | Bind address. Set to 0.0.0.0:11434 to allow remote access (e.g., GPU server serving DSPy clients on other machines) |
| OLLAMA_NUM_GPU | auto | Number of GPU layers. 999 = all GPU, 0 = CPU only |
| OLLAMA_NUM_PARALLEL | 1 | Concurrent requests. Increase for DSPy batch/optimization runs |
| OLLAMA_MAX_LOADED_MODELS | 1 | Models kept in memory simultaneously. Increase for multi-model pipelines |
| OLLAMA_MODELS | ~/.ollama/models | Model storage directory. Change if disk space is limited |
Setting env vars per platform:
- macOS (app):
launchctl setenv OLLAMA_HOST "0.0.0.0:11434"then restart Ollama - Linux (systemd):
sudo systemctl edit ollama.service, addEnvironment=lines under[Service], thensystemctl daemon-reload && systemctl restart ollama - Everywhere else:
export OLLAMA_HOST="0.0.0.0:11434"beforeollama serve
Remote Ollama (GPU server)
If Ollama runs on a different machine (e.g., a GPU server), point DSPy to it:
lm = dspy.LM(
"ollama_chat/llama3.1:8b",
api_base="http://gpu-server:11434", # remote Ollama host
api_key="",
num_ctx=8192,
)
Make sure the server has OLLAMA_HOST=0.0.0.0:11434 set.
Performance tuning
GPU acceleration
Ollama automatically uses GPU if available. Check with:
ollama ps # shows which models are loaded and GPU/CPU split
Apple Silicon optimization
Ollama runs natively on Apple Silicon using Metal. Performance tips:
- M1/M2 (8GB): 8B models work well with
num_ctx=4096 - M1/M2 Pro (16GB): 8B models with
num_ctx=8192, or 14B withnum_ctx=4096 - M1/M2 Max (32GB+): 70B quantized models with
num_ctx=4096 - M3/M4 Max (64GB+): 70B models with
num_ctx=8192
Embedding models (for retrieval pipelines)
Ollama can serve embedding models for dspy.Embedder:
embedder = dspy.Embedder(
"ollama/nomic-embed-text",
api_base="http://localhost:11434",
api_key="",
batch_size=200, # default; reduce if Ollama OOMs on large batches
)
| Model | Dimensions | Size | Notes |
|-------|-----------|------|-------|
| nomic-embed-text | 768 | 274 MB | Good default, fast |
| mxbai-embed-large | 1024 | 670 MB | Higher accuracy |
| bge-m3 | 1024 | 1.2 GB | Multilingual |
Pull embedding models the same way: ollama pull nomic-embed-text
Model management
ollama list # show installed models and sizes
ollama pull llama3.1:8b # download or update a model
ollama rm codellama:7b # remove a model to free disk space
ollama ps # show running models and GPU/CPU split
ollama show llama3.1:8b # show model details (parameters, template, license)
curl http://localhost:11434/api/tags # verify Ollama is running (useful in scripts)
Run Ollama as a background service instead of ollama serve &:
- macOS: The Ollama app runs as a menu bar service automatically
- Linux:
sudo systemctl enable --now ollama(installed by the install script)
Per-module model assignment
Use a big model for hard tasks and a small model for simple ones:
import dspy
big = dspy.LM("ollama_chat/llama3.1:8b", api_base="http://localhost:11434",
api_key="", num_ctx=8192)
small = dspy.LM("ollama_chat/llama3.2:3b", api_base="http://localhost:11434",
api_key="", num_ctx=4096)
dspy.configure(lm=small) # default: cheap model
class Pipeline(dspy.Module):
def __init__(self):
self.classify = dspy.Predict("text -> category")
self.analyze = dspy.ChainOfThought("text, category -> analysis")
def forward(self, text):
cat = self.classify(text=text)
return self.analyze(text=text, category=cat.category)
pipeline = Pipeline()
pipeline.classify.set_lm(small) # simple task → small model
pipeline.analyze.set_lm(big) # complex task → big model
Running DSPy optimization with Ollama
Optimization works with local models but is significantly slower than cloud APIs. Tips:
import dspy
lm = dspy.LM("ollama_chat/llama3.1:8b", api_base="http://localhost:11434",
api_key="", num_ctx=8192)
dspy.configure(lm=lm)
# Tip 1: Start with BootstrapFewShot (fastest optimizer)
optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
optimized = optimizer.compile(program, trainset=trainset)
# Tip 2: For MIPROv2, use auto="light" (fewest trials)
optimizer = dspy.MIPROv2(metric=metric, auto="light")
optimized = optimizer.compile(program, trainset=trainset)
# Tip 3: Use a bigger model as teacher, smaller as student
teacher_lm = dspy.LM("ollama_chat/llama3.1:70b", api_base="http://localhost:11434",
api_key="", num_ctx=8192)
optimizer = dspy.BootstrapFewShot(metric=metric, max_bootstrapped_demos=4)
with dspy.context(lm=teacher_lm):
optimized = optimizer.compile(program, trainset=trainset)
# Deploy optimized program with the smaller model
Expect hours, not minutes for optimization with local models. A MIPROv2 auto="medium" run that takes 5 minutes with GPT-4o-mini might take 2-4 hours with a local 8B model.
Ollama vs vLLM
| | Ollama | vLLM |
|---|--------|------|
| Setup | brew install ollama | pip install vllm (NVIDIA only) |
| Platform | macOS, Linux, Windows | Linux (NVIDIA GPU required) |
| Apple Silicon | Yes (Metal) | No |
| Throughput | Single-user | High concurrency (10+ users) |
| Multi-GPU | No | Yes (tensor parallelism) |
| Best for | Development, prototyping | Production serving |
Recommended workflow: Develop with Ollama locally, deploy with vLLM in production. The DSPy code is identical — only the LM config line changes:
# Development (Ollama)
lm = dspy.LM("ollama_chat/llama3.1:8b", api_base="http://localhost:11434", api_key="")
# Production (vLLM)
lm = dspy.LM("openai/meta-llama/Llama-3.1-8B-Instruct", api_base="http://gpu-server:8000/v1", api_key="none")
Gotchas
- Context window defaults to 4096 — always set
num_ctxexplicitly. DSPy optimized prompts with few-shot demos easily exceed 4096 tokens. api_key=""is required — even though Ollama doesn't use it, LiteLLM requires the parameter.- First request is slow — Ollama loads the model into memory on the first call. Subsequent calls are fast.
- OOM errors — reduce
num_ctxor switch to a smaller model. Check VRAM withollama ps. dspy.OllamaLocalis deprecated — usedspy.LM("ollama_chat/...")instead.
Cross-references
Install any skill:
npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>
- LM configuration basics (providers, parameters, caching) —
/dspy-lm - Production serving with vLLM —
/dspy-vllm - Reducing costs (model routing, caching) —
/ai-cutting-costs - Switching models without breaking things —
/ai-switching-models - Install
/ai-doif you do not have it — it routes any AI problem to the right skill and is the fastest way to work:npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do
Additional resources
- dspy.LM API docs
- dspy.Embedder API docs
- Ollama model library
- For API details, see reference.md
- For worked examples, see examples.md
Related Skills
lebsral/ai-watching-optimization
tools
VerifiedTrustedCommunity
See what is happening during optimizer.compile() instead of waiting blind. Use when you want to watch optimization progress, see scores as they come in, know if your optimizer is working, check if optimization is stuck, understand why optimization is taking too long, get live progress during compile, monitor convergence, detect overfitting during optimization, interpret optimization results, or pick the right tool for watching optimization. Also used for optimizer progress bar, is my optimizer doing anything, optimization seems stuck, how long will optimization take, watch GEPA run, watch MIPROv2 run, live optimization dashboard, optimizer not improving, scores not going up, optimization taking forever, see what optimizer is doing, debug slow optimization, optimization visibility, optimizer metrics, track compile progress, optimization observability.
6SKILL.mdUpdated May 31, 2026
lebsral/dspy-miprov2
testing
VerifiedTrustedCommunity
Use when you want the highest-quality prompt optimization DSPy offers — jointly optimizes instructions and few-shot demos, with auto=light/medium/heavy presets. Common scenarios - you want the best possible accuracy from prompt optimization, jointly tuning instructions and few-shot demonstrations, using auto presets for different compute budgets, or when COPRO or BootstrapFewShot alone are not reaching your accuracy target. Related - ai-improving-accuracy, dspy-copro, dspy-bootstrap-few-shot. Also used for dspy.MIPROv2, best DSPy optimizer, highest quality optimization, auto=light medium heavy, joint instruction and demo optimization, most powerful prompt optimizer, MIPROv2 vs COPRO vs BootstrapFewShot, which optimizer should I use, state of the art prompt optimization, when to use MIPROv2, optimize both instructions and examples, heavy optimization for production, best optimizer for accuracy.
6SKILL.mdUpdated Apr 27, 2026
lebsral/dspy-langwatch
testing
VerifiedTrustedCommunity
Use LangWatch for DSPy auto-tracing and real-time optimizer progress. Use when you want to set up LangWatch, langwatch.dspy.init, auto-tracing DSPy, real-time optimization dashboard, optimizer progress tracking, app.langwatch.ai, or DSPy optimizer dashboard. Also used for langwatch setup, pip install langwatch, langwatch trace, optimizer progress, real-time optimization, watch optimizer run, LangWatch self-hosted, langwatch docker, langwatch vs langtrace, langwatch autotrack_dspy.
6SKILL.mdUpdated Apr 27, 2026
lebsral/dspy-gepa
data-ai
VerifiedTrustedCommunity
Use when you want to optimize instructions without few-shot examples — a lightweight alternative to COPRO when you do not have or do not want to use demonstrations. Common scenarios - optimizing instructions when you do not have or do not want to use few-shot demonstrations, lightweight instruction search as a first step, tasks where examples in the prompt confuse the model, or when you want fast instruction optimization without the cost of COPRO. Related - ai-improving-accuracy, dspy-copro, dspy-miprov2. Also used for dspy.GEPA, instruction optimization without demos, lightweight prompt optimization, optimize instructions only, no few-shot examples needed, GEPA vs COPRO, quick instruction search, when demonstrations hurt performance, zero-shot optimization, instruction-only optimizer, simplest instruction tuner, fast prompt optimization, skip few-shot and just tune instructions, optimize Pydantic field descriptions, GEPA structured output, GEPA does not optimize field desc.
6SKILL.mdUpdated Apr 27, 2026