microsoft/image-vision

Image Vision Analysis

Overview

Analyze images using state-of-the-art LLM vision models. Use the provided scripts for most tasks - custom code only needed for advanced scenarios.

Workflow Decision Tree

First time using this skill?

→ Read setup.md for one-time environment and API key setup

Simple image analysis (most common)

→ Use "Quick Start" canned scripts below

Batch processing or multi-turn conversations

→ Read patterns.md for advanced patterns

Something failing?

→ Check setup.md for troubleshooting

Quick Start (Use Wrapper Scripts)

ALWAYS use the wrapper scripts - they handle venv setup automatically:

# Simple analysis (auto-creates venv on first use)
./vision-analyze.sh <provider> <image_path> <prompt>

# Robust analysis (auto-fallback if provider times out)
./vision-analyze-robust.sh <image_path> <prompt> [timeout_seconds]

The wrapper scripts automatically:

• Create venv if it doesn't exist
• Install required SDKs
• Use venv Python (no manual activation needed)
• Handle errors gracefully

Example usage:

# Analyze a UI screenshot (Anthropic Claude)
./vision-analyze.sh anthropic screenshot.png "Describe any UI bugs or issues you see"

# Extract text (Google Gemini - fastest)
./vision-analyze.sh gemini document.jpg "Extract all text from this image"

# Robust analysis with auto-fallback (tries Gemini → Anthropic → OpenAI)
./vision-analyze-robust.sh photo.png "Describe this image in detail"

# With custom timeout (default is 60 seconds)
./vision-analyze-robust.sh large-image.png "Analyze this" 120

Advanced: Direct Script Usage (Not Recommended)

If you need to call the Python scripts directly, you MUST use the venv Python:

# ❌ WRONG - uses system Python, will fail
python examples/anthropic-vision.py image.png "prompt"

# ✅ CORRECT - uses venv Python
./.venv/bin/python examples/anthropic-vision.py image.png "prompt"

For agents: Always use the wrapper scripts to avoid setup issues.

Provider Comparison

| Provider | Model | Best For | Speed | Cost | |----------|-------|----------|-------|------| | Anthropic | claude-sonnet-4-5 | Latest, balanced quality/speed | Fast | $$ | | Anthropic | claude-3-opus | Highest quality (older) | Slow | $$$ | | Anthropic | claude-3-haiku | Fastest, simple tasks | Very Fast | $ | | OpenAI | gpt-5 | Latest flagship model | Fast | $$$ | | OpenAI | gpt-4.1 | High-volume production | Fast | $$ | | Gemini | gemini-2.5-flash | Latest, excellent balance | Very Fast | $ | | Gemini | gemini-2.5-pro | Large images, best quality | Medium | $$ | | Azure | (deployment-based) | Enterprise, compliance | Varies | Varies |

Supported Image Formats

• JPEG/JPG - Most common
• PNG - With transparency
• GIF - Static or animated (only the first frame is analyzed)
• WEBP - Modern format

Max sizes:

• Anthropic: 5MB per image
• OpenAI: 20MB (auto-resizes)
• Gemini: Varies by model (1.5 pro handles very large)

Common Use Cases

# UI/UX Analysis - High-level layout and spacing
./vision-analyze.sh anthropic app-screenshot.png \
  "Analyze this UI for accessibility issues and suggest improvements"

# Bug Identification (use robust for auto-fallback)
./vision-analyze-robust.sh error-state.png \
  "What's wrong with this interface? Describe any visual bugs."

# Content Moderation
./vision-analyze.sh openai user-upload.jpg \
  "Does this image contain inappropriate content? Yes or no, and explain."

# Document Understanding (Gemini is fastest)
./vision-analyze.sh gemini invoice.png \
  "Extract the total amount, date, and vendor name from this invoice"

# Design Review - Layout, color, hierarchy (not typography details)
./vision-analyze-robust.sh mockup.png \
  "Provide design feedback on this mockup. Consider layout, color hierarchy, and spacing."

⚠️ Known Limitations for Web UI Analysis

Typography and Font Detection

Vision models struggle with precise typography at typical screenshot resolutions:

❌ Unreliable for:

• Distinguishing serif vs sans-serif fonts at small sizes (<16px)
• Identifying specific font families (Inter vs Roboto vs Arial)
• Detecting subtle weight differences (400 vs 500)
• Precise alignment measurements (<5px differences)

✅ Reliable for:

• High-level layout issues (spacing, hierarchy, colors)
• Large size differences (14px vs 24px heading sizes)
• Missing elements or obviously broken UI states
• Color contrast and accessibility problems

Best Practice: Multi-Modal Investigation

For Web UI bugs, use this hierarchy:

# 1. Vision for TRIAGE (identify area of concern)
./vision-analyze-robust.sh screenshot.png "Are there any visual inconsistencies in the navigation?"

# 2. Browser inspection for FACTS (if typography/font suspected)
# Use Playwright or DevTools to query computed CSS:
# const styles = await page.evaluate(() => ({
#   fontFamily: getComputedStyle(element).fontFamily
# }));

# 3. Code investigation for ROOT CAUSE
# grep -r ".suspicious-class" src/

# 4. Vision for VERIFICATION (after fix applied)
./vision-analyze-robust.sh fixed.png "Is the navigation font now consistent?"

When to Stop Using Vision

If vision gives contradictory results across 2+ attempts on similar screenshots:

1. Stop asking vision for more detailed analysis
2. Switch to browser DevTools inspection (query computed styles)
3. Use vision only for final verification after fix is applied

This indicates the issue is too subtle for vision models to detect reliably.

Prompt Patterns for Web UI

Font/Typography (with caveats):

# Be explicit about what to look for
./vision-analyze.sh anthropic ui.png \
  "Look at the navigation text. Do any items have decorative 'feet' at letter ends (serif font) 
  while others have clean straight edges (sans-serif)? Point out any font style differences."
  
# Note: Small fonts may be unreliable - verify with browser inspection

Alignment (relative observations):

# Ask for noticeable differences, not pixel precision
./vision-analyze.sh anthropic ui.png \
  "Is the bullet (•) noticeably misaligned with the text baseline? 
  Describe its vertical position relative to the text."

Layout and Spacing:

# Vision is GOOD at this
./vision-analyze.sh anthropic ui.png \
  "Compare the spacing between navigation sections. Is it consistent?"

Output Format

All scripts output to stdout as plain text. The LLM's analysis is printed directly:

$ python examples/anthropic-vision.py screenshot.png "What's in this image?"

This image shows a web application dashboard with a navigation bar at the top,
a sidebar on the left with menu items, and a main content area displaying...

For structured output, modify your prompt:

python examples/openai-vision.py data.png \
  "Extract data as JSON with keys: title, date, amount"

When to Write Custom Scripts

Use the canned scripts for:

• ✅ Single image + single prompt analysis
• ✅ Quick one-off tasks
• ✅ Simple Q&A about images

Write custom scripts when you need:

• ❌ Batch processing (analyze 100 images)
• ❌ Multi-turn conversations (follow-up questions on same image)
• ❌ Custom output formatting (generate markdown reports)
• ❌ Image preprocessing (resize, crop, filter)
• ❌ Provider fallback logic (try Gemini, then Claude)

→ See patterns.md for custom script examples

Anti-Patterns

| ❌ Don't | ✅ Do | |----------|-------| | Write custom script for simple analysis | Use canned scripts | | Use low-quality compressed images | Use clear, high-res images | | Ask vague questions | Be specific in prompts | | Forget to set API keys | Set keys in environment variables | | Mix up provider-specific model names | Check provider comparison table |

Quick Reference

| Task | Command | |------|---------| | Analyze (single provider) | ./vision-analyze.sh anthropic img.png "prompt" | | Analyze (auto-fallback) | ./vision-analyze-robust.sh img.png "prompt" | | Extract text (OCR) | ./vision-analyze.sh gemini img.png "Extract all text" | | Health check | ./health-check.sh | | Compare images | See patterns.md for custom script | | Batch process | See patterns.md for custom script |

⚠️ CRITICAL INSTRUCTIONS FOR AGENTS

READ THIS BEFORE USING THIS SKILL:

1. Always Use the Wrapper Scripts

# For AI agents (recommended) - auto-fallback on timeout
~/.amplifier/skills/image-vision/vision-analyze-robust.sh <image_path> <prompt>

# Single provider (faster if you know which to use)
~/.amplifier/skills/image-vision/vision-analyze.sh <provider> <image_path> <prompt>

Examples:

# Robust analysis (tries multiple providers if timeout)
~/.amplifier/skills/image-vision/vision-analyze-robust.sh screenshot.png "Analyze this UI"

# Specific provider
~/.amplifier/skills/image-vision/vision-analyze.sh anthropic screenshot.png "Describe this"

2. ALWAYS Check Exit Code Before Using Output

# Correct usage pattern
OUTPUT=$(~/.amplifier/skills/image-vision/vision-analyze-robust.sh image.png "Analyze this" 2>&1)
EXIT_CODE=$?

if [ $EXIT_CODE -eq 0 ]; then
    echo "Vision analysis succeeded"
    # Now you can use $OUTPUT
else
    echo "ERROR: Vision analysis failed (exit code: $EXIT_CODE)"
    echo "Error details: $OUTPUT"
    # STOP HERE - do NOT proceed
    exit 1
fi

Exit codes are classified so callers get an honest failure signal instead of a catch-all (this distinction matters: a missing key and a slow provider need different fixes):

| Exit | Meaning | What to do | |------|---------|-----------| | 0 | Success | Use the output | | 1 | Usage error, or all configured providers failed (mixed/other errors) | Read stderr for the underlying error | | 3 | No vision provider configured (no API key present) | Set GOOGLE_API_KEY / ANTHROPIC_API_KEY / OPENAI_API_KEY (or AZURE_OPENAI_API_KEY + AZURE_OPENAI_ENDPOINT) | | 4 | provider_timeout — provider(s) present but every attempt timed out within the bounded timeout | Increase the timeout (3rd arg), use a smaller image, or a faster provider |

vision-analyze-robust.sh fails fast and clearly on exit 3/4 rather than hanging — a missing or slow provider surfaces immediately, not minutes later.

3. NEVER Fabricate Visual Observations

If vision analysis fails, you MUST:

✅ DO:

• Report failure explicitly to user
• Provide error details from stderr
• Ask user how to proceed (retry? different provider? skip visual analysis?)
• Wait for user direction before continuing

❌ NEVER:

• Write analysis documents without successfully seeing images
• Fabricate visual observations based on context/guesswork
• Guess pixel measurements or UI element details
• Pretend you analyzed screenshots you didn't actually see
• Continue with tasks that require visual inspection if vision failed

Example of CORRECT failure handling:

Agent: I attempted to analyze the 3 screenshots using the image-vision skill:
- screenshot-1.png: ✗ Anthropic timed out (60s)
- screenshot-1.png: ✗ Gemini timed out (60s)  
- screenshot-1.png: ✗ OpenAI failed (API error)

I have NOT successfully analyzed any of the screenshots. I cannot provide visual design 
feedback without actually seeing the images.

Options:
1. Retry with different settings
2. Investigate why all providers are failing
3. Defer visual analysis until the issue is resolved

I will NOT write design analysis documents based on guesswork or context alone.

4. Timeout Considerations

Vision API calls typically take 5-60 seconds:

• Gemini Flash: 3-10s (fastest)
• Anthropic Claude: 5-15s
• OpenAI GPT-4: 8-20s

The wrapper scripts handle timeouts with:

• 60-second default timeout (configurable, always bounded)
• Auto-fallback to faster providers (robust script)
• Retry logic on transient failures
• Automatic screenshot downscaling — every image is capped in width (2000px) and bounded in encoded payload size (a real, fail-closed bound) before it is sent (see examples/image_utils.py). This makes the "resize to 2000px max" guidance automatic so an un-capped full-page screenshot can't produce an oversized, interruptible request. Downscaling is conservative: downscale-only (never upscales), aspect preserved, LANCZOS, EXIF-aware. It caps width rather than the longest edge specifically so a tall full-page capture is not squashed — width is the dimension that text legibility depends on. If a pathological image still exceeds the payload bound at the width floor, it is re-encoded to JPEG, and if it still cannot fit the call fails clearly rather than sending an oversized payload.

Capture hygiene still matters. For text-critical verification (small numbers, labels, precise alignment), prefer viewport-sized captures over aggressive full-page captures, and corroborate with browser/DOM facts — see "Known Limitations for Web UI Analysis" above. The cap removes a hang risk; it does not make vision reliable for sub-pixel typography.

If still hitting timeouts:

• Use smaller images (capping is automatic, but viewport captures help most)
• Simplify prompts
• Use faster models (Gemini Flash)

Environment Setup Reminder

For interactive use:

1. Create venv: cd image-vision && uv venv
2. Install SDKs: uv pip install anthropic openai google-genai pillow
3. Set API keys: Export ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY

For agents:

• Just use the wrapper scripts - they auto-setup on first use
• Verify health: ./health-check.sh

→ See setup.md for complete instructions

See Also

• setup.md — One-time environment setup, API keys, troubleshooting
• patterns.md — Advanced patterns: batch processing, multi-turn, custom output