julianobarbosa/prometheus

Prometheus API Skill

Query Prometheus monitoring systems via HTTP API at /api/v1.

Quick Reference

Instant Query

curl 'http://<prometheus>:9090/api/v1/query?query=<promql>&time=<timestamp>'

Range Query

curl 'http://<prometheus>:9090/api/v1/query_range?query=<promql>&start=<ts>&end=<ts>&step=<duration>'

Response Format

All responses return JSON:

{
  "status": "success" | "error",
  "data": <result>,
  "errorType": "<string>",
  "error": "<string>",
  "warnings": ["<string>"]
}

HTTP codes: 400 (bad params), 422 (expression error), 503 (timeout).

Query Endpoints

| Endpoint | Purpose | Key Parameters | |----------|---------|----------------| | /api/v1/query | Instant query | query, time, timeout, limit | | /api/v1/query_range | Range query | query, start, end, step, timeout, limit | | /api/v1/format_query | Format PromQL | query | | /api/v1/series | Find series by labels | match[], start, end, limit | | /api/v1/labels | List label names | start, end, match[], limit | | /api/v1/label/<name>/values | Label values | start, end, match[], limit | | /api/v1/query_exemplars | Query exemplars | query, start, end |

Metadata & Status Endpoints

| Endpoint | Purpose | |----------|---------| | /api/v1/targets | Target discovery status (state=active\|dropped\|any) | | /api/v1/targets/metadata | Metric metadata from targets | | /api/v1/metadata | All metric metadata | | /api/v1/rules | Alerting/recording rules | | /api/v1/alerts | Active alerts | | /api/v1/alertmanagers | Alertmanager discovery | | /api/v1/status/config | Current config YAML | | /api/v1/status/flags | CLI flags | | /api/v1/status/runtimeinfo | Runtime info | | /api/v1/status/buildinfo | Build info | | /api/v1/status/tsdb | TSDB cardinality stats | | /api/v1/status/walreplay | WAL replay progress |

Admin Endpoints (require --web.enable-admin-api)

| Endpoint | Method | Purpose | |----------|--------|---------| | /api/v1/admin/tsdb/snapshot | POST | Create TSDB snapshot | | /api/v1/admin/tsdb/delete_series | POST | Delete series (match[], start, end) | | /api/v1/admin/tsdb/clean_tombstones | POST | Clean deleted data |

Common PromQL Patterns

# Rate of counter over 5m
rate(http_requests_total[5m])

# Sum by label
sum by (job) (rate(http_requests_total[5m]))

# Percentile from histogram
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

# Filter by label
up{job="prometheus", instance=~".*:9090"}

# Increase over time
increase(http_requests_total[1h])

# Average over time range
avg_over_time(process_cpu_seconds_total[5m])

Result Types

• vector: [{"metric": {...}, "value": [timestamp, "value"]}]
• matrix: [{"metric": {...}, "values": [[ts, "val"], ...]}]
• scalar: [timestamp, "value"]
• string: [timestamp, "string"]

Scripts

Query script: scripts/prom_query.py

# Instant query
python scripts/prom_query.py http://localhost:9090 'up'

# Range query
python scripts/prom_query.py http://localhost:9090 'rate(http_requests_total[5m])' \
  --start '2024-01-01T00:00:00Z' --end '2024-01-01T01:00:00Z' --step '1m'

# Output: table, json, csv
python scripts/prom_query.py http://localhost:9090 'up' --format table

Health check: scripts/prom_health.py

python scripts/prom_health.py http://localhost:9090

Detailed Reference

For complete API documentation: references/api_reference.md

For PromQL functions: references/promql_functions.md

---

Gotchas

• rate() over a counter that resets too often: math is correct but meaningless — use increase() and divide by interval explicitly when counters don't survive scrapes.
• up{} per-target gauge: a flaky target shows up=0 but doesn't trigger alerts unless for is met. Set short for for liveness, long for noise.
• Recording rules evaluate at fixed interval; missed evaluations don't backfill — gaps in the recording series during incidents.
• Federation match[] parameter requires ALL matchers to match — an empty matcher returns no series, which looks like a working query with no data.
• Stale-marker semantics: a series stops being scraped → stale marker after 5 min by default → queries see "no data" not "0". Affects alerts on absent().
• Service Discovery + relabel_config: a bad regex in keep action silently drops all targets — verify with /api/v1/targets after each config change.