pjt222/detect-anomalies-aiops

Anomalien mit AIOps erkennen

See Extended Examples for complete configuration files and templates.

Anwenden maschinelles Lernen to detect anomalies in operational metrics, correlate alerts, and reduce false positives.

Wann verwenden

• Operations team overwhelmed by alert volume (>100 alerts/day)
• Need to detect complex multi-metric anomalies (not just threshold breaches)
• Seasonal patterns make static thresholds ineffective
• Want to predict issues vor they impact users (proactive detection)
• Need to correlate related alerts to identify root cause
• Monitoring system generates too many false positives
• Want to detect subtle performance degradation trends

Eingaben

• Erforderlich: Time series metrics from monitoring system (CPU, memory, latency, error rate)
• Erforderlich: Historical data (30-90 days minimum)
• Optional: Alarmieren history with labels (true positive / false positive)
• Optional: System topology (service Abhaengigkeiten)
• Optional: Log data for correlation
• Optional: Deployment/change events for context

Vorgehensweise

Schritt 1: Set Up Environment and Laden Data

Installieren Abhaengigkeiten and prepare time series data for analysis.

# Create virtual environment
python -m venv venv
source venv/bin/activate

# Install anomaly detection libraries
pip install prophet scikit-learn pandas numpy
pip install tensorflow keras  # for LSTM models
pip install pyod  # Python Outlier Detection library
pip install statsmodels  # for statistical methods
pip install prometheus-api-client  # if using Prometheus

# Visualization
pip install plotly matplotlib seaborn

Laden and prepare data:

# aiops/data_loader.py
import pandas as pd
import numpy as np
from datetime import datetime, timedelta
from typing import List, Dict
import logging

logging.basicConfig(level=logging.INFO)
# ... (see EXAMPLES.md for complete implementation)

Erwartet: Time series data loaded with regular intervals, missing values handled, features engineered for ML models.

Bei Fehler: If Prometheus connection fails, verify URL and network access, if data gaps exist use forward-fill or interpolation, ensure timestamp column is datetime type, check for memory issues with large date ranges (process in chunks).

Schritt 2: Implementieren Isolation Forest for Multivariate Anomaly Detection

Detect anomalies using unsupervised Isolation Forest algorithm.

# aiops/isolation_forest_detector.py
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler
import pandas as pd
import numpy as np
from typing import Dict, List
import joblib

# ... (see EXAMPLES.md for complete implementation)

Erwartet: Modellieren trained on historical data, anomalies detected with scores, typischerweise 0.5-2% of points flagged as anomalies.

Bei Fehler: If too many anomalies (>5%), reduce contamination parameter or retrain on cleaner baseline period, if too few (<0.1%), increase contamination or check feature scaling, verify features have sufficient variance.

Schritt 3: Implementieren Prophet for Time Series Forecasting and Anomaly Detection

Use Facebook Prophet to model seasonality and detect deviations.

# aiops/prophet_detector.py
from prophet import Prophet
import pandas as pd
import numpy as np
from typing import Dict, Tuple
import logging

logger = logging.getLogger(__name__)
# ... (see EXAMPLES.md for complete implementation)

Erwartet: Prophet models capture daily/weekly seasonality, anomalies detected when actual values fall outside 99% confidence interval, forecasts generated for capacity planning.

Bei Fehler: If Prophet takes too long (>5 min per metric), reduce history to 30 days or disable weekly_seasonality, if too many false positives increase interval_width to 0.995, if missing seasonal patterns add custom seasonalities, ensure timezone consistency in timestamps.

Schritt 4: Correlate Alerts and Identifizieren Root Cause

Group related anomalies and identify potential root causes.

# aiops/alert_correlation.py
import pandas as pd
import numpy as np
from sklearn.cluster import DBSCAN
from typing import List, Dict
from datetime import timedelta
import networkx as nx

# ... (see EXAMPLES.md for complete implementation)

Erwartet: Related anomalies grouped into incidents, root causes identified basierend auf Abhaengigkeit graph, incident summaries generated for investigation.

Bei Fehler: If all anomalies separate incidents, increase time_window_minutes, if root cause detection unclear define metric_relationships explicitly basierend auf architecture, verify timestamp sorting is correct.

Schritt 5: Integrieren with Alerting System

Senden intelligent alerts with context and suppression of noise.

# aiops/intelligent_alerting.py
import requests
import logging
from typing import Dict, List
from datetime import datetime, timedelta
import json

logger = logging.getLogger(__name__)
# ... (see EXAMPLES.md for complete implementation)

Erwartet: High-severity incidents trigger PagerDuty pages, medium-severity go to Slack, low-severity logged only, duplicate alerts suppressed innerhalb 15-minute window.

Bei Fehler: Testen webhook URLs with curl first, verify severity calculation produces reasonable values (0.5-0.9 range), check rate limiting doesn't suppress all alerts, ensure timezone handling is correct for last_alerts tracking.

Schritt 6: Bereitstellen as Continuous Monitoring Service

Einrichten automated pipeline that runs periodically.

# aiops/monitoring_service.py
import schedule
import time
import logging
from datetime import datetime, timedelta
from data_loader import MetricsDataLoader
from isolation_forest_detector import IsolationForestDetector
from prophet_detector import ProphetAnomalyDetector
# ... (see EXAMPLES.md for complete implementation)

Erwartet: Service runs continuously, detects anomalies every 5 minutes, alerts sent for incidents, logs all activity.

Bei Fehler: Verifizieren scheduler process stays alive (use systemd/supervisor for production), check Prometheus connectivity, ensure models are loaded erfolgreich, implement dead man's switch alert if service stops running, monitor memory usage (reload models periodically if memory grows).

Validierung

• [ ] Historical data loaded korrekt with no missing timestamps
• [ ] Isolation Forest detects known anomalies from test set
• [ ] Prophet models capture daily/weekly seasonality in visualizations
• [ ] Alarmieren correlation groups temporally-related anomalies
• [ ] Root cause detection identifies upstream issues korrekt
• [ ] Intelligent alerting suppresses duplicate alerts
• [ ] Severity calculation produces reasonable scores (0.5-0.9)
• [ ] Monitoring service runs continuously ohne crashes for 7+ days
• [ ] False positive rate < 10% (validated gegen labeled data)
• [ ] True positive rate > 80% for critical incidents

Haeufige Stolperfallen

• Training on anomalous data: Sicherstellen baseline period used for training is clean (no incidents); manuell review or use labeled data
• Ignoring seasonality: Static models fail on daily/weekly patterns; use Prophet or add time features
• Too sensitive thresholds: 99% confidence intervals may flag normal peaks; start with 99.5% and tune basierend auf false positives
• Not handling missing data: Gaps in metrics cause model errors; implement robust preprocessing with interpolation
• Alarmieren fatigue from low severity: Filtern alerts unter severity threshold; focus on high-confidence anomalies
• Ignoring system topology: Treating all metrics independently misses cascading failures; define Abhaengigkeit relationships
• Modellieren drift: Models trained on old data become stale; retrain monthly or when system changes
• Resource contention: Running detection on every metric is expensive; prioritize critical services or sample metrics

Verwandte Skills

• monitor-model-drift - Detect when anomaly detection models degrade
• monitor-data-integrity - Data quality checks vor anomaly detection
• setup-prometheus-monitoring - Sammeln operational metrics
• forecast-operational-metrics - Capacity planning with Prophet forecasts