lightricks/be-cost-monitoring

Backend Cost Monitoring

When to use

• "Monitor GPU costs"
• "Analyze cost trends (daily/weekly/monthly)"
• "Detect cost anomalies or spikes"
• "Break down API costs by endpoint/model/org"
• "Break down Studio costs by process/workspace"
• "Compare API vs Studio cost distribution"
• "Investigate cost-per-request efficiency drift"
• "Monitor GPU utilization and idle costs"
• "Alert on cost budget breaches"
• "Day-over-day or week-over-week cost comparisons"

Steps

1. Gather Requirements

Ask the user:

• What to monitor: Total costs, cost by product/feature, cost efficiency, utilization, anomalies
• Scope: LTX API, LTX Studio, or both? Specific endpoint/org/process?
• Time window: Daily, weekly, monthly? How far back?
• Analysis type: Trends, comparisons (DoD/WoW), anomaly detection, breakdowns
• Alert threshold (if setting up alerts): Absolute ($X/day) or relative (spike > X% vs baseline)

2. Read Shared Knowledge

Before writing SQL:

• shared/product-context.md — LTX products and business context
• shared/bq-schema.md — GPU cost table schema (lines 418-615)
• shared/metric-standards.md — GPU cost metric patterns (section 13)
• shared/event-registry.yaml — Feature events (if analyzing feature-level costs)
• shared/gpu-cost-query-templates.md — 11 production-ready SQL queries
• shared/gpu-cost-analysis-patterns.md — Analysis workflows and benchmarks

Key learnings:

• Table: ltx-dwh-prod-processed.gpu_costs.gpu_request_attribution_and_cost
• Partitioned by dt (DATE) — always filter for performance
• cost_category: inference (requests), idle, overhead, unused
• Total cost = row_cost + attributed_idle_cost + attributed_overhead_cost (inference rows only)
• For infrastructure cost: SUM(row_cost) across all categories

3. Run Query Templates

If user didn't specify anything: Run all 11 query templates from shared/gpu-cost-query-templates.md to provide comprehensive cost overview.

If user specified specific analysis: Select appropriate template:

| User asks... | Use template | |-------------|-------------| | "What are total costs?" | Daily Total Cost (API + Studio) | | "Break down API costs" | API Cost by Endpoint & Model | | "Break down Studio costs" | Studio Cost by Process | | "Yesterday vs day before" | Day-over-Day Comparison | | "This week vs last week" | Week-over-Week Comparison | | "Detect cost spikes" | Anomaly Detection (Z-Score) | | "GPU utilization breakdown" | Utilization by Cost Category | | "Cost efficiency by model" | Cost per Request by Model | | "Which orgs cost most?" | API Cost by Organization |

See shared/gpu-cost-query-templates.md for all 11 query templates.

4. Execute Query

Run query using:

bq --project_id=ltx-dwh-explore query --use_legacy_sql=false --format=pretty "
<query>
"

Or use BigQuery console with project ltx-dwh-explore.

5. Analyze Results

For cost trends:

• Compare current period vs baseline (7-day avg, prior week, prior month)
• Calculate % change and flag significant shifts (>15-20%)

For anomaly detection:

• Flag days with Z-score > 2 (cost or volume deviates > 2 std devs from rolling avg)
• Investigate root cause: specific endpoint/model/org, error rate spike, billing type change

For breakdowns:

• Identify top cost drivers (endpoint, model, org, process)
• Calculate cost per request to spot efficiency issues
• Check failure costs (wasted spend on errors)

6. Present Findings

Format results with:

• Summary: Key finding (e.g., "GPU costs spiked 45% yesterday")
• Root cause: What drove the change (e.g., "LTX API /v1/text-to-video requests +120%")
• Breakdown: Top contributors by dimension
• Recommendation: Action to take (investigate org X, optimize model Y, alert team)

7. Set Up Alert (if requested)

For ongoing monitoring:

1. Save SQL query
2. Set up in BigQuery scheduled query or Hex Thread
3. Configure notification threshold
4. Route alerts to Slack channel or Linear issue

Schema Reference

For detailed table schema including all dimensions, columns, and cost calculations, see references/schema-reference.md.

Reference Files

| File | Read when | |------|-----------| | references/schema-reference.md | GPU cost table dimensions, columns, and cost calculations | | shared/bq-schema.md | Understanding GPU cost table schema (lines 418-615) | | shared/metric-standards.md | GPU cost metric SQL patterns (section 13) | | shared/gpu-cost-query-templates.md | Selecting query template for analysis (11 production-ready queries) | | shared/gpu-cost-analysis-patterns.md | Interpreting results, workflows, benchmarks, investigation playbooks |

Rules

Query Best Practices

• DO always filter on dt partition column for performance
• DO filter cost_category = 'inference' for request-level analysis
• DO exclude Lightricks team requests with is_lt_team IS FALSE for customer-facing cost analysis
• DO include LT team requests only when analyzing total infrastructure spend or debugging
• DO use ltx-dwh-explore as execution project
• DO calculate cost per request with SAFE_DIVIDE to avoid division by zero
• DO compare against baseline (7-day avg, prior period) for trends
• DO round cost values to 2 decimal places for readability

Cost Calculation

• DO sum all three cost columns (row_cost + attributed_idle + attributed_overhead) for fully loaded cost per request
• DO use SUM(row_cost) across all rows for total infrastructure cost
• DO NOT sum row_cost + attributed_* across all cost_categories (double-counting)
• DO NOT mix inference and non-inference rows in same aggregation without filtering

Analysis

• DO flag anomalies with Z-score > 2 (cost or volume deviation > 2 std devs)
• DO investigate failure costs (wasted spend on errors)
• DO break down by endpoint/model for API, by process for Studio
• DO check cost per request trends to spot efficiency degradation
• DO validate results against total infrastructure spend

Alerts

• DO set thresholds based on historical baseline, not absolute values
• DO alert engineering team for cost spikes > 30% vs baseline
• DO include cost breakdown and root cause in alerts
• DO route API cost alerts to API team, Studio alerts to Studio team