codyswanngt/ops-verify-telemetry

Ops: Verify Telemetry

Verify OpenTelemetry traces are being collected and exported to X-Ray.

Argument: $ARGUMENTS — check type (traces, metrics, all; default: all)

Discovery

1. Read Gemfile for OpenTelemetry gem configuration:
   • opentelemetry-sdk
   • opentelemetry-exporter-otlp
   • opentelemetry-instrumentation-all (or individual instrumentation gems)
2. Read config/initializers/opentelemetry.rb (or similar) for SDK configuration
3. Read config/deploy.yml or environment files for OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME
4. Read docker-compose.yml for OpenTelemetry Collector sidecar configuration (if present)

Local Verification

Check OpenTelemetry Gems

bundle list | grep -i opentelemetry

Check OpenTelemetry Configuration

grep -r "OpenTelemetry" config/initializers/ app/

Verify OTLP Endpoint is Set

echo "OTEL_EXPORTER_OTLP_ENDPOINT=${OTEL_EXPORTER_OTLP_ENDPOINT:-not set}"
echo "OTEL_SERVICE_NAME=${OTEL_SERVICE_NAME:-not set}"

Generate a Test Trace (local)

Make a request to the local app and check that traces are produced:

# Make a request that should generate a trace
curl -sf http://localhost:3000/up -w "\nHTTP %{http_code}\n"

# Check Rails logs for OpenTelemetry output (if configured to log)
grep -i "otel\|opentelemetry\|trace_id" log/development.log | tail -10

Check Collector Sidecar (if using Docker Compose)

# Capture the status BEFORE the pipe. A pipeline reports its LAST stage's exit
# code, so `docker compose logs ... | tail -20 || echo "No otel-collector"`
# never prints the fallback: `tail` succeeds even when there is no such
# service, and the absent sidecar reads as a silent pass.
#
# `|| status=$?`, not `; status=$?`: under `set -e` the `;` form exits before
# the assignment, so the branch that reports the failure never runs.
#
# Ask whether the service EXISTS separately from whether its logs could be
# read. `docker compose logs` also exits non-zero for a stopped daemon, a
# permission failure, and an unparseable compose file; answering all of those
# with "No otel-collector service" turns a broken telemetry stack into a
# reassuring sentence — the same silent-measurement failure one level up.
services_status=0
services=$(docker compose config --services 2>&1) || services_status=$?
if [ "$services_status" -ne 0 ]; then
  echo "FAILED ($services_status) — docker compose config could not be read:"
  printf '%s\n' "$services"
elif ! printf '%s\n' "$services" | grep -qx otel-collector; then
  echo "No otel-collector service in Docker Compose"
else
  logs_status=0
  docker compose logs otel-collector >otel.log 2>&1 || logs_status=$?
  if [ "$logs_status" -eq 0 ]; then
    tail -n 20 otel.log
  else
    echo "FAILED ($logs_status) — otel-collector is configured but its logs could not be read:"
    cat otel.log
  fi
fi

Remote Verification (AWS X-Ray)

Check Recent Traces

aws xray get-trace-summaries \
  --region {aws-region} \
  --start-time $(ruby -r time -e 'puts (Time.now.utc - 30 * 60).iso8601') \
  --end-time $(ruby -r time -e 'puts Time.now.utc.iso8601') \
  --query 'TraceSummaries | length(@)' \
  --output text

Check Traces for the Application Service

aws xray get-trace-summaries \
  --region {aws-region} \
  --start-time $(ruby -r time -e 'puts (Time.now.utc - 30 * 60).iso8601') \
  --end-time $(ruby -r time -e 'puts Time.now.utc.iso8601') \
  --filter-expression "service(\"{service-name}\")" \
  --query 'TraceSummaries[:10].{TraceId:Id,Duration:Duration,StatusCode:Http.HttpStatus,URL:Http.HttpURL,ResponseTime:ResponseTime}' \
  --output table

Check for Error Traces

aws xray get-trace-summaries \
  --region {aws-region} \
  --start-time $(ruby -r time -e 'puts (Time.now.utc - 3600).iso8601') \
  --end-time $(ruby -r time -e 'puts Time.now.utc.iso8601') \
  --filter-expression "service(\"{service-name}\") AND fault = true" \
  --query 'TraceSummaries[:10].{TraceId:Id,Duration:Duration,StatusCode:Http.HttpStatus,URL:Http.HttpURL}' \
  --output table

Get Detailed Trace

aws xray batch-get-traces \
  --region {aws-region} \
  --trace-ids "{trace-id}" \
  --query 'Traces[0].Segments[].Document' \
  --output text | jq '.'

Check X-Ray Service Map

aws xray get-service-graph \
  --region {aws-region} \
  --start-time $(ruby -r time -e 'puts (Time.now.utc - 3600).iso8601') \
  --end-time $(ruby -r time -e 'puts Time.now.utc.iso8601') \
  --query 'Services[].{Name:Name,Type:Type,Edges:Edges[].{Ref:ReferenceId,Latency:ResponseTimeHistogram[0].Average}}' \
  --output table

CloudWatch Metrics Verification

Check Custom Metrics Published by the Application

aws cloudwatch list-metrics \
  --region {aws-region} \
  --namespace "{app_name}" \
  --query 'Metrics[].{MetricName:MetricName,Dimensions:Dimensions[].{Name:Name,Value:Value}}' \
  --output table

Check Recent Metric Data Points

aws cloudwatch get-metric-statistics \
  --region {aws-region} \
  --namespace "{app_name}" \
  --metric-name "{metric-name}" \
  --start-time $(ruby -r time -e 'puts (Time.now.utc - 3600).iso8601') \
  --end-time $(ruby -r time -e 'puts Time.now.utc.iso8601') \
  --period 300 \
  --statistics Average Sum \
  --output table

Check CloudWatch Alarms

aws cloudwatch describe-alarms \
  --region {aws-region} \
  --alarm-name-prefix "{app_name}" \
  --query 'MetricAlarms[].{Name:AlarmName,State:StateValue,Metric:MetricName,Threshold:Threshold}' \
  --output table

Remote Verification (via Kamal)

For checking telemetry configuration in deployed environments:

kamal app exec --roles=web "bin/rails runner \"
  puts 'OTEL_EXPORTER_OTLP_ENDPOINT: ' + ENV.fetch('OTEL_EXPORTER_OTLP_ENDPOINT', 'NOT SET')
  puts 'OTEL_SERVICE_NAME: ' + ENV.fetch('OTEL_SERVICE_NAME', 'NOT SET')
  puts 'OTEL_TRACES_EXPORTER: ' + ENV.fetch('OTEL_TRACES_EXPORTER', 'NOT SET')
\"" -d {environment}

Output Format

Trace Summary

| Check | Status | Details | |-------|--------|---------| | OTel gems installed | OK/FAIL | opentelemetry-sdk v1.x.x | | OTel initializer present | OK/FAIL | config/initializers/opentelemetry.rb | | OTLP endpoint configured | OK/FAIL | https://otel-collector:4318 | | Traces in X-Ray (last 30m) | OK/FAIL | 245 traces found | | Error traces (last 1h) | OK/WARN | 3 fault traces | | CloudWatch metrics | OK/FAIL | 12 custom metrics published | | CloudWatch alarms | OK/WARN | 1 alarm in ALARM state |

Recent Traces

| Trace ID | Duration | Status | URL | |----------|----------|--------|-----| | 1-abc123 | 45ms | 200 | GET /up | | 1-def456 | 120ms | 200 | GET /api/v1/users | | 1-ghi789 | 2300ms | 500 | POST /api/v1/orders |

Flag concerns:

• No traces found in the last 30 minutes
• Error rate above 5% of total traces
• Average trace duration above 1 second
• OTLP endpoint not configured
• CloudWatch alarms in ALARM state