lebsral/ai-understanding-images

AI Understanding Images

Use DSPy's dspy.Image type to pass images into signatures alongside text. Vision LLMs return structured data from photos, screenshots, documents, and charts.

---

Step 1 - Understand the image task

Before writing code, ask:

• What images will you process? (URLs, local files, base64, cloud storage?)
• What do you need to extract? (text, categories, attributes, descriptions?)
• Does the output need to be structured? (typed fields vs. free text?)
• Are you processing images in batch or one at a time?
• Does the task require reasoning about the image, or just direct extraction?

---

Step 2 - Build a basic image analyzer

import dspy

lm = dspy.LM("openai/gpt-4o-mini")  # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)

class AnalyzeImage(dspy.Signature):
    """Analyze the image and answer the question."""
    image: dspy.Image = dspy.InputField(desc="The image to analyze")
    question: str = dspy.InputField(desc="What to extract or analyze")
    answer: str = dspy.OutputField(desc="Analysis result")

analyzer = dspy.Predict(AnalyzeImage)

# From a URL
result = analyzer(
    image=dspy.Image.from_url("https://example.com/photo.jpg"),
    question="What product is shown?"
)

# From a local file
result = analyzer(
    image=dspy.Image.from_file("photo.jpg"),
    question="What product is shown?"
)

print(result.answer)

---

Step 3 - Vision model selection

| Model | Strengths | Notes | |---|---|---| | openai/gpt-4o | Best overall vision quality | Higher cost | | openai/gpt-4o-mini | Fast, cheap, good for simple tasks | Weaker on complex layouts | | anthropic/claude-opus-4-5 | Strong reasoning about images | High cost | | anthropic/claude-sonnet-4-5-20250929 | Balanced quality/cost | Good for production | | google/gemini-pro-vision | Long context, PDF support | Check API availability |

All models listed here support image inputs. Always verify vision support before deploying.

---

Step 4 - Combine image with text context

For richer analysis, pass supplemental text alongside the image:

from typing import Literal
from pydantic import BaseModel

class ProductAttributes(BaseModel):
    category: str
    color: str
    condition: Literal["new", "used", "damaged"]
    description: str

class CategorizeProduct(dspy.Signature):
    """Categorize a product from its photo and any provided context."""
    image: dspy.Image = dspy.InputField(desc="Product photo")
    context: str = dspy.InputField(desc="Additional context such as listing title or seller notes")
    attributes: ProductAttributes = dspy.OutputField(desc="Extracted product attributes")

categorizer = dspy.Predict(CategorizeProduct)
result = categorizer(
    image=dspy.Image.from_url("https://example.com/item.jpg"),
    context="Listed as: Vintage leather jacket, size M"
)
print(result.attributes.category, result.attributes.condition)

---

Step 5 - Common patterns

Alt text generation

class GenerateAltText(dspy.Signature):
    """Generate concise, accurate alt text for accessibility."""
    image: dspy.Image = dspy.InputField(desc="Image to describe")
    context: str = dspy.InputField(desc="Page or article context where the image appears")
    alt_text: str = dspy.OutputField(desc="Alt text under 125 characters, describing the image content")

alt_gen = dspy.Predict(GenerateAltText)
result = alt_gen(
    image=dspy.Image.from_url("https://example.com/team-photo.jpg"),
    context="About page of a SaaS startup"
)

Receipt and invoice OCR

from typing import List

class LineItem(BaseModel):
    description: str
    quantity: int
    unit_price: float
    total: float

class ExtractReceipt(dspy.Signature):
    """Extract all line items and totals from a receipt or invoice photo."""
    image: dspy.Image = dspy.InputField(desc="Photo of receipt or invoice")
    line_items: List[LineItem] = dspy.OutputField(desc="All line items found")
    subtotal: float = dspy.OutputField(desc="Subtotal before tax")
    tax: float = dspy.OutputField(desc="Tax amount")
    total: float = dspy.OutputField(desc="Total amount due")

extractor = dspy.Predict(ExtractReceipt)
result = extractor(image=dspy.Image.from_file("receipt.jpg"))

Chart and graph data extraction

class ExtractChart(dspy.Signature):
    """Extract the data series and labels from a chart or graph image."""
    image: dspy.Image = dspy.InputField(desc="Chart or graph image")
    chart_type: str = dspy.OutputField(desc="Type of chart - bar, line, pie, etc.")
    title: str = dspy.OutputField(desc="Chart title if present")
    data_summary: str = dspy.OutputField(desc="Summary of the data shown, including key values")

chart_reader = dspy.Predict(ExtractChart)

UI screenshot analysis

class AnalyzeUI(dspy.Signature):
    """Analyze a UI screenshot and identify components and issues."""
    image: dspy.Image = dspy.InputField(desc="UI screenshot")
    focus: str = dspy.InputField(desc="What aspect to analyze - layout, accessibility, bugs, etc.")
    findings: str = dspy.OutputField(desc="Detailed findings about the UI")
    suggestions: List[str] = dspy.OutputField(desc="Actionable improvement suggestions")

ui_analyzer = dspy.ChainOfThought(AnalyzeUI)
result = ui_analyzer(
    image=dspy.Image.from_file("screenshot.png"),
    focus="accessibility issues"
)

---

Step 6 - OCR vs vision model tradeoff

| Scenario | Recommended approach | |---|---| | Clean printed text on white background | Tesseract or cloud OCR (faster, cheaper) | | Handwritten text | Vision LLM (GPT-4o, Claude Sonnet) | | Mixed layout with images and text | Vision LLM | | Receipts with varied formatting | Vision LLM | | High-volume document digitization | Dedicated OCR service + vision LLM for exceptions | | Extracting structured fields from forms | Vision LLM with typed output | | Sub-100ms latency requirement | Dedicated OCR only |

---

Step 7 - Image preprocessing

Vision models have token budgets per image. Large images consume more tokens and slow responses.

from PIL import Image as PILImage
import io, base64

def resize_for_vision(image_path: str, max_side: int = 1024) -> dspy.Image:
    """Resize image so the longest side is at most max_side pixels."""
    img = PILImage.open(image_path)
    ratio = min(max_side / img.width, max_side / img.height, 1.0)
    if ratio < 1.0:
        new_size = (int(img.width * ratio), int(img.height * ratio))
        img = img.resize(new_size, PILImage.LANCZOS)
    buf = io.BytesIO()
    img.save(buf, format="JPEG", quality=85)
    b64 = base64.b64encode(buf.getvalue()).decode()
    return dspy.Image.from_base64(b64, media_type="image/jpeg")

image = resize_for_vision("large_photo.jpg")
result = analyzer(image=image, question="What is shown?")

Recommended limits:

• Max 1024px on the longest side for most tasks
• JPEG quality 80-85 for photos; PNG for screenshots with text
• Avoid sending multiple large images in a single call

---

Step 8 - Evaluate visual tasks

For description quality, use an AI judge:

class ImageDescriptionJudge(dspy.Signature):
    """Judge whether an image description is accurate and complete."""
    image: dspy.Image = dspy.InputField(desc="The original image")
    description: str = dspy.InputField(desc="Description to evaluate")
    score: int = dspy.OutputField(desc="Score from 1 to 5, where 5 is fully accurate and complete")
    reasoning: str = dspy.OutputField(desc="Explanation of the score")

judge = dspy.Predict(ImageDescriptionJudge)

For structured extraction (OCR, receipts), use exact match or field-level comparison:

def eval_receipt_extraction(prediction, ground_truth):
    correct_items = sum(
        1 for item in prediction.line_items
        if item.description in [g.description for g in ground_truth.line_items]
    )
    recall = correct_items / max(len(ground_truth.line_items), 1)
    total_match = abs(prediction.total - ground_truth.total) < 0.01
    return {"item_recall": recall, "total_correct": total_match}

---

When NOT to use vision LLMs

• Object detection at scale - use YOLO, Detectron2, or a dedicated CV API
• Simple OCR on clean printed text - Tesseract or cloud OCR is faster and cheaper
• Pixel-level segmentation - use Segment Anything or dedicated segmentation models
• Real-time video processing - vision LLMs have too much latency
• Sub-100ms latency - vision LLMs typically take 1-5 seconds per image
• High-volume identical-format documents - train a specialized model or use template OCR

---

Key patterns

# Pattern 1 - Direct extraction with typed output
class ExtractFields(dspy.Signature):
    """Extract structured fields from the image."""
    image: dspy.Image = dspy.InputField()
    fields: MyDataModel = dspy.OutputField()

extractor = dspy.Predict(ExtractFields)

# Pattern 2 - Reasoning about image content
class ReasonAboutImage(dspy.Signature):
    """Reason step by step about what the image shows."""
    image: dspy.Image = dspy.InputField()
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()

reasoner = dspy.ChainOfThought(ReasonAboutImage)

# Pattern 3 - Batch processing
images = [dspy.Image.from_file(p) for p in image_paths]
results = [extractor(image=img) for img in images]

# Pattern 4 - Iterative refinement when quality is low
refiner = dspy.Refine(dspy.Predict(AnalyzeImage), N=3, reward_fn=my_reward)

---

Gotchas

• Wrap image inputs in dspy.Image - Claude writes raw URL strings as image inputs instead of dspy.Image.from_url(). Always use dspy.Image.from_url() or dspy.Image.from_file(). Raw strings are treated as text, not images.

• Verify the model supports vision - Claude picks a model that does not support image inputs. Not all LLMs handle images. Confirm vision support for your chosen model before deploying (GPT-4o, Claude 3.5+, Gemini Pro Vision all work).

• Use dspy.Refine not dspy.Assert - Claude uses dspy.Assert/dspy.Suggest to validate image outputs. Use dspy.Refine with a reward function for iterative improvement instead.

• Resize before sending - Claude sends full-resolution images without resizing. Large images (4K, RAW photos) consume excessive tokens and can hit context limits. Resize to max 1024px on the longest side before processing.

• Match module to task - Claude applies dspy.ChainOfThought to all image tasks. Use dspy.Predict for direct extraction (OCR, field parsing). Reserve dspy.ChainOfThought for tasks that genuinely benefit from image reasoning, like diagnosing a bug from a screenshot.

---

Cross-references

Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>

• /ai-parsing-data - parse structured data from text; complement with image extraction for mixed inputs
• /ai-stopping-hallucinations - reduce made-up field values in vision extraction pipelines
• /ai-checking-outputs - validate extracted fields after vision model output
• /dspy-refine - iterative refinement when initial image analysis quality is low
• /dspy-modules - understand dspy.Predict vs dspy.ChainOfThought for image tasks
• Install /ai-do if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do

Additional resources

See examples.md for worked examples:

• Product photo categorizer
• Alt text generator
• Receipt/invoice OCR pipeline