szoloth/datadog-analyst

Datadog Product Analyst Skill

This skill transforms Claude into an expert product/data analyst for DreamWorks Animation's product suite, with deep knowledge of Datadog RUM/APM APIs, XmR process control methodology, and per-product metrics.

Products Covered

| Product | Description | Key Metrics | |---------|-------------|-------------| | Lasso | AI-powered search for DWA assets | Unique users, search queries, zero-result rate, latency | | Rico | AI-native Digital Backlot (agentic interface) | Agent interactions, reasoning steps, search accuracy | | Courier | Asset delivery/handoff system | First-time delivery rate, rework rate, preview usage | | Shoppe | Production pipeline dashboard | Asset views, vendor access, pipeline status checks |

Quick Start

Query Metrics

To fetch metrics for any product, use the dd-query.ts script:

npx tsx ~/.claude/skills/datadog-analyst/scripts/dd-query.ts lasso --range 7d

Generate Report

To generate a markdown report with XmR analysis:

npx tsx ~/.claude/skills/datadog-analyst/scripts/dd-report.ts lasso --range 30d --output report.md

Terminal Charts

To display ASCII charts in terminal:

npx tsx ~/.claude/skills/datadog-analyst/scripts/dd-chart.ts lasso users --range 14d

Exclude Internal Users

All scripts support --exclude-internal to filter out team members:

npx tsx ~/.claude/skills/datadog-analyst/scripts/dd-query.ts lasso --exclude-internal
npx tsx ~/.claude/skills/datadog-analyst/scripts/dd-report.ts all --exclude-internal
npx tsx ~/.claude/skills/datadog-analyst/scripts/dd-chart.ts courier --exclude-internal

Internal users currently excluded:

• Charles Adams (Product Owner)
• Jeff Lombardo, Brett Schlank (Leadership)
• Dennis Duong, Dmytro, Charles Billingsley, Srikanth (Engineering)
• Stephen Garver, Jeremy Cytryn (Stakeholders)

To add/remove users, edit references/internal-users.md and the INTERNAL_USER_PATTERNS array in each script.

Environment Setup

The skill requires Datadog API credentials in environment variables:

export DATADOG_API_KEY="your-api-key"
export DATADOG_APP_KEY="your-app-key"
export DD_SITE="datadoghq.com"

If credentials are not set, the skill will guide through setup.

Analysis Approach

XmR Process Behavior Charts

All time series data is automatically analyzed using Donald Wheeler's XmR methodology:

1. Calculate limits: Center line, UNPL (upper), LNPL (lower) using NPL_SCALING = 2.66
2. Detect special variation using 3 rules:
   • Rule 1: Point outside limits (strong signal)
   • Rule 2: 3 of 4 consecutive points near limit (moderate signal)
   • Rule 3: 8 consecutive points on same side of center (subtle shift)
3. Flag actionable signals: Only special variation requires investigation

Interpretation Guidance

• Routine variation: No action needed. Fluctuations are expected noise.
• Special variation: Investigate the cause. Something has actually changed.

Reference references/xmr-methodology.md for detailed interpretation guidance.

Per-Product Analysis

Lasso (AI Search)

Key Questions:

• How many unique users this week vs last week?
• What's the zero-result rate? (Target: <5%)
• What's the search latency? (Target: <500ms)
• Any workflows showing special variation?

Datadog App Name: Lasso

Key Actions:

• search_results_returned - Search completed
• search_query_submitted - User initiated search
• asset_opened - User clicked result

Courier (Asset Delivery)

Key Questions:

• What's the first-time clean delivery rate?
• What's the rework/resend rate?
• Are users using Package Preview before sending?
• Any spike in validation errors?

Datadog App Name: Courier

Key Actions:

• transfer_attempted - User clicked Ship
• transfer_result - Transfer completed (success/fail)
• package_resend_initiated - Correction/resend
• package_preview_opened - Preview feature usage

Shoppe (Pipeline Dashboard)

Key Questions:

• How many asset views per day?
• What's the vendor access pattern?
• Which shows have highest engagement?
• Any changes in usage patterns?

Datadog App Name: Shoppe

Key Actions:

• asset_viewed - User viewed asset details
• export_requested - User exported asset
• vendor_session_started - External vendor access
• catalog_navigation - Cross-tool navigation

Rico (AI Agent)

Key Questions:

• How are users interacting with the agent?
• What's the reasoning step distribution?
• Any errors in agent responses?

Datadog App Name: Rico

Key Actions:

• agent_query_submitted - User asked agent
• agent_response_generated - Agent responded
• reasoning_step_completed - Agent thinking step

Output Formats

Terminal ASCII Charts

For quick visualization in terminal:

Unique Users (Last 14 Days)
        ╭─────────────────────────────────────╮
   45 ┤ ●                           ●        │ UNPL
   40 ┤   ●    ●                              │
   35 ┼───●──●─────────●────────────────────── │ Center
   30 ┤         ●  ●     ●  ●  ●  ●           │
   25 ┤              ●              ●    ●    │
   20 ┤                                       │ LNPL
        ╰─────────────────────────────────────╯
         Jan 14 →→→→→→→→→→→→→→→→→→→→→ Jan 28

Status: ✓ Routine variation (no action needed)

ASCII Tables

For metric breakdowns:

┌──────────────────────────────────────────────────────┐
│ Lasso Metrics Summary (Last 7 Days)                  │
├──────────────────────┬───────────┬──────────┬────────┤
│ Metric               │ Value     │ Change   │ Status │
├──────────────────────┼───────────┼──────────┼────────┤
│ Unique Users         │ 45        │ +12%     │ ●      │
│ Total Searches       │ 1,234     │ +8%      │ ●      │
│ Zero Result Rate     │ 3.2%      │ -0.5%    │ ●      │
│ Avg Latency          │ 342ms     │ -15ms    │ ●      │
└──────────────────────┴───────────┴──────────┴────────┘

● Routine variation  ◐ Moderate signal  ○ Special variation

Markdown Reports

For shareable weekly digests, see assets/report-template.md.

Common Workflows

"How is Lasso doing?"

1. Fetch key metrics for last 7 days
2. Compare to previous 7 days
3. Run XmR analysis on time series
4. Report status (routine vs special variation)
5. Highlight any metrics needing investigation

"Generate weekly report for all products"

1. Query each product's key metrics
2. Build comparison tables
3. Identify special variation signals across products
4. Generate markdown report
5. Include recommendations based on signals

"Why did search latency spike on Monday?"

1. Query hourly latency data for the day
2. Apply XmR analysis to identify exact time of change
3. Query correlated metrics (error rates, user count)
4. Check for deployment events or system changes
5. Provide root cause hypothesis

Bundled Resources

• scripts/dd-query.ts: CLI for Datadog RUM aggregate queries
• scripts/dd-chart.ts: Terminal ASCII chart generator
• scripts/dd-report.ts: Markdown report generator
• references/products.md: Detailed per-product metrics definitions
• references/xmr-methodology.md: XmR interpretation guide
• assets/report-template.md: Weekly report template

Limitations

• Rico may not have production instrumentation yet (prototype stage)
• Courier metrics require specific instrumentation (see references/products.md)
• Real-time data has ~1-2 minute latency from Datadog
• Very long date ranges (>90 days) may timeout on complex queries