athal7/observability

Use this skill to investigate production problems. For query syntax, index patterns, and example queries, consult the log query reference.

Triage process

Start with the symptom, not the tool. Before querying anything:

1. State the hypothesis — what do you think is wrong and why?
2. Bound the time window — when did it start? Is it ongoing or resolved?
3. Identify the scope — one service, one endpoint, one user, or system-wide?

This prevents aimless log-scrolling and makes findings interpretable.

Signal hierarchy

Work top-down — coarser signals first, drill into finer ones only when needed:

| Signal | What it tells you | When to use | |---|---|---| | Error rate / rate spike | Something broke at scale | First check — confirms the problem is real | | APM traces | Which transaction is slow or failing, full call chain | Once you know the scope | | APM errors | Exception type, stack trace, grouping key | When you need the root cause code path | | Logs | Raw context around a specific event | When traces don't have enough detail |

Don't start with logs. Start with traces or error groups, then use trace.id to pull the surrounding log context.

Correlating signals

The trace.id field links all three indices (logs-*, traces-apm*, logs-apm.error-*). Once you have a trace.id from an error or slow trace, use it to pull all logs from that same request:

{"term": {"trace.id": "<trace-id-here>"}}

Asking the right questions

Before querying, write down what a "confirmed" answer looks like. Examples:

• "If query returns 0 errors for service X in the last 1h, the issue has resolved"
• "If the slow trace shows N+1 queries on endpoint Y, the cause is clear"
• "If errors spike at exactly :15 and :45 of every hour, it's likely a cron job"

This prevents misreading absence of evidence as evidence of absence.

When to escalate

Stop investigating and escalate to the team when:

• Error rate is sustained above baseline for > 15 minutes and cause is not identified
• A trace shows calls to an external dependency timing out (not your code)
• Errors reference a data migration or schema change (potential data integrity issue)
• You've ruled out the obvious causes and need production access or context you don't have

Common patterns

| Symptom | Where to look first | |---|---| | Slow page loads | APM traces — sort by transaction.duration.us desc | | 500 errors spiking | APM errors — group by error.grouping_key | | One user affected | Logs — filter by user ID or session ID | | Periodic issue | Logs — look for time pattern in @timestamp | | After a deploy | APM errors — filter by @timestamp after deploy time |