frank-luongt/skills/codex/cloudwatch-observability

<!-- AUTO-GENERATED by export-skills.py — DO NOT EDIT -->

---

name: cloudwatch-observability description: Amazon CloudWatch patterns for AI agent observability. Use when monitoring Bedrock agent invocations, tracking token usage, setting up alarms for agent failures, or analyzing agent performance via CloudWatch Logs Insights.

Amazon CloudWatch for AI Agent Observability

Monitor AI agent performance, costs, and reliability using CloudWatch metrics, logs, and alarms.

When to Use

• Monitoring Bedrock agent invocation latency, token usage, and error rates
• Setting up alarms for agent failures or cost spikes
• Analyzing agent reasoning traces via CloudWatch Logs Insights
• Building dashboards for AI operations

Patterns

1. Bedrock Invocation Metrics

Key CloudWatch metrics emitted by Amazon Bedrock:

| Metric | Namespace | Description | |---|---|---| | Invocations | AWS/Bedrock | Number of model invocations | | InvocationLatency | AWS/Bedrock | End-to-end invocation time (ms) | | InvocationClientErrors | AWS/Bedrock | 4xx errors (throttling, validation) | | InvocationServerErrors | AWS/Bedrock | 5xx errors | | InputTokenCount | AWS/Bedrock | Input tokens consumed | | OutputTokenCount | AWS/Bedrock | Output tokens generated | | InvocationThrottles | AWS/Bedrock | Throttled requests |

2. CloudWatch Logs Insights for Agent Traces

-- Find slowest agent invocations in the last 24h
fields @timestamp, @message
| filter @message like /agentId/
| parse @message '"invocationLatencyMs":*,' as latency
| sort latency desc
| limit 20

-- Token usage by model over time
fields @timestamp
| filter @message like /inputTokenCount/
| parse @message '"modelId":"*"' as model
| parse @message '"inputTokenCount":*,' as input_tokens
| parse @message '"outputTokenCount":*,' as output_tokens
| stats sum(input_tokens) as total_input, sum(output_tokens) as total_output by model, bin(1h)

-- Agent errors with reasoning trace
fields @timestamp, @message
| filter @message like /ERROR/ or @message like /ThrottlingException/
| sort @timestamp desc
| limit 50

3. Cost Tracking Alarm

import boto3

cloudwatch = boto3.client("cloudwatch")

# Alarm when daily token usage exceeds threshold
cloudwatch.put_metric_alarm(
    AlarmName="bedrock-daily-token-budget",
    Namespace="AWS/Bedrock",
    MetricName="InputTokenCount",
    Statistic="Sum",
    Period=86400,  # 24 hours
    EvaluationPeriods=1,
    Threshold=10_000_000,  # 10M tokens
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789:ai-ops-alerts"],
    Dimensions=[{"Name": "ModelId", "Value": "anthropic.claude-3-5-sonnet-20241022-v2:0"}],
)

# Alarm for high error rate
cloudwatch.put_metric_alarm(
    AlarmName="bedrock-agent-error-rate",
    Namespace="AWS/Bedrock",
    MetricName="InvocationServerErrors",
    Statistic="Sum",
    Period=300,  # 5 minutes
    EvaluationPeriods=2,
    Threshold=10,
    ComparisonOperator="GreaterThanThreshold",
    AlarmActions=["arn:aws:sns:us-east-1:123456789:ai-ops-alerts"],
)

4. Custom Agent Metrics

import boto3

cloudwatch = boto3.client("cloudwatch")

def publish_agent_metrics(agent_name: str, metrics: dict):
    """Publish custom agent metrics to CloudWatch."""
    cloudwatch.put_metric_data(
        Namespace="FAOS/AgentOps",
        MetricData=[
            {
                "MetricName": "ToolCallCount",
                "Value": metrics["tool_calls"],
                "Unit": "Count",
                "Dimensions": [{"Name": "AgentName", "Value": agent_name}],
            },
            {
                "MetricName": "ResolutionRate",
                "Value": metrics["resolved_pct"],
                "Unit": "Percent",
                "Dimensions": [{"Name": "AgentName", "Value": agent_name}],
            },
            {
                "MetricName": "SessionDuration",
                "Value": metrics["duration_ms"],
                "Unit": "Milliseconds",
                "Dimensions": [{"Name": "AgentName", "Value": agent_name}],
            },
        ],
    )

Anti-Patterns

• Not setting cost alarms -- AI token usage can spike unexpectedly
• Using @message full-text search instead of structured filters -- parse fields first
• Retaining agent logs indefinitely -- set log group retention policies (30-90 days)
• Missing CloudTrail integration -- always enable for Bedrock API audit

References

• Amazon CloudWatch Documentation
• Bedrock CloudWatch Metrics
• CloudWatch Logs Insights Query Syntax
• AWS CloudWatch MCP Server

<!-- Source: .faos/custom/skills/cloud/aws/cloudwatch-observability/SKILL.md -->