Heldinhow/ollama-local-ai

Ollama — Local AI

Installation & Setup

# macOS
brew install ollama

# Start the server (runs on http://localhost:11434)
ollama serve

# Pull a model
ollama pull llama3.2          # 3B — fast, general purpose
ollama pull llama3.2:1b       # 1B — ultra-fast, low RAM
ollama pull mistral           # 7B — strong reasoning
ollama pull codellama         # code-specialized
ollama pull nomic-embed-text  # embeddings
ollama pull phi4-mini         # Microsoft, small and capable

Model Management

ollama list                   # list installed models
ollama show llama3.2          # model info
ollama rm llama3.2            # remove model
ollama run llama3.2           # interactive chat in terminal

REST API

# Chat completion
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [{ "role": "user", "content": "Hello!" }],
  "stream": false
}'

# Generate (single turn)
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Why is the sky blue?",
  "stream": false
}'

# Embeddings
curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": "Text to embed"
}'

Integration with Vercel AI SDK

import { ollama } from 'ollama-ai-provider'
import { generateText, streamText } from 'ai'

// Text generation
const { text } = await generateText({
  model: ollama('llama3.2'),
  prompt: 'Explain quantum computing in one paragraph',
})

// Streaming
const stream = streamText({
  model: ollama('llama3.2'),
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'Write a haiku about TypeScript.' },
  ],
})

for await (const chunk of stream.textStream) {
  process.stdout.write(chunk)
}

// Embeddings
import { embed } from 'ai'

const { embedding } = await embed({
  model: ollama.embedding('nomic-embed-text'),
  value: 'Some text to embed',
})

bun add ollama-ai-provider ai

Direct Ollama SDK

import { Ollama } from 'ollama'

const ollama = new Ollama({ host: 'http://localhost:11434' })

// Chat
const response = await ollama.chat({
  model: 'llama3.2',
  messages: [{ role: 'user', content: 'Hello' }],
})
console.log(response.message.content)

// Streaming
const stream = await ollama.chat({
  model: 'llama3.2',
  messages: [{ role: 'user', content: 'Tell me a story' }],
  stream: true,
})
for await (const chunk of stream) {
  process.stdout.write(chunk.message.content)
}

// Embeddings
const { embedding } = await ollama.embed({
  model: 'nomic-embed-text',
  input: 'text to embed',
})

Modelfile (Custom Models)

# Modelfile
FROM llama3.2

SYSTEM """
You are a senior TypeScript developer.
Always return typed, idiomatic TypeScript.
Prefer functional patterns and avoid classes.
"""

PARAMETER temperature 0.3
PARAMETER num_ctx 8192

ollama create my-ts-assistant -f Modelfile
ollama run my-ts-assistant

Model Selection Guide

| Use case | Recommended model | |---|---| | General chat | llama3.2 (3B) or mistral (7B) | | Fast/low RAM | llama3.2:1b or phi4-mini | | Code generation | codellama or qwen2.5-coder | | Embeddings | nomic-embed-text or mxbai-embed-large | | Long context | mistral with num_ctx 32768 | | Math/reasoning | phi4 or qwen2.5 |

Privacy & Performance Notes

• All inference runs locally — no data leaves your machine
• GPU acceleration: Ollama auto-detects Metal (macOS), CUDA, ROCm
• RAM requirements: 4GB for 3B models, 8GB for 7B, 16GB+ for 13B+
• Set OLLAMA_NUM_PARALLEL env var to serve multiple requests