kavi-lin/theme-detector

Theme Detector

Overview

This skill detects and ranks trending market themes by analyzing cross-sector momentum, volume, and breadth signals. It identifies both bullish (upward momentum) and bearish (downward pressure) themes, assesses lifecycle maturity (Emerging/Accelerating/Trending/Mature/Exhausting), and provides a confidence score combining quantitative data with narrative analysis.

3-Dimensional Scoring Model:

1. Theme Heat (0-100): Direction-neutral strength of the theme (momentum, volume, uptrend ratio, breadth)
2. Lifecycle Maturity: Stage classification (Emerging / Accelerating / Trending / Mature / Exhausting) based on duration, extremity clustering, valuation, and ETF proliferation
3. Confidence (Low / Medium / High): Reliability of the detection, combining quantitative breadth with narrative confirmation. Script output is capped at Medium; Claude's WebSearch narrative confirmation step can elevate to High.

Key Features:

• Cross-sector theme detection using FINVIZ industry data
• Direction-aware scoring (bullish and bearish themes)
• Lifecycle maturity assessment to identify crowded vs. emerging trades
• ETF proliferation scoring (more ETFs = more mature/crowded theme)
• Integration with uptrend-dashboard for 3-point evaluation
• Dual-mode operation: FINVIZ Elite (fast) or public scraping (slower, limited)
• WebSearch-based narrative confirmation for top themes

---

When to Use This Skill

Explicit Triggers:

• "What market themes are trending right now?"
• "Which sectors are hot/cold?"
• "Detect current market themes"
• "What are the strongest bullish/bearish narratives?"
• "Is AI/clean energy/defense still a strong theme?"
• "Where is sector rotation heading?"
• "Show me thematic investing opportunities"

Implicit Triggers:

• User wants to understand broad market narrative shifts
• User is looking for thematic ETF or sector allocation ideas
• User asks about crowded trades or late-cycle themes
• User wants to know which themes are emerging vs. exhausted

When NOT to Use:

• Individual stock analysis (use us-stock-analysis instead)
• Specific sector deep-dive with chart reading (use sector-analyst instead)
• Portfolio rebalancing (use portfolio-manager instead)
• Dividend/income investing (use value-dividend-screener instead)

---

Workflow

Step 1: Verify Requirements

Check for required API keys and dependencies:

# Check for FINVIZ Elite API key (optional but recommended)
echo $FINVIZ_API_KEY

# Check for FMP API key (optional, used for valuation metrics)
echo $FMP_API_KEY

Requirements:

• Python 3.7+ with requests, beautifulsoup4, lxml, pandas, numpy, yfinance
• FINVIZ Elite API key (recommended for full industry coverage and speed)
• FMP API key (optional, for P/E ratio valuation data)
• Without FINVIZ Elite, the skill uses public FINVIZ scraping (limited to ~20 stocks per industry, slower rate limits)

Optional dependencies:

• finvizfinance (for FINVIZ Elite mode)
• PyYAML (for --themes-config custom themes)

Installation:

pip install requests beautifulsoup4 lxml pandas numpy yfinance

Step 2: Execute Theme Detection Script

Run the main detection script:

python3 skills/theme-detector/scripts/theme_detector.py \
  --output-dir reports/

Script Options:

# Full run (public FINVIZ mode, no API key required)
python3 skills/theme-detector/scripts/theme_detector.py \
  --output-dir reports/

# With FINVIZ Elite API key
python3 skills/theme-detector/scripts/theme_detector.py \
  --finviz-api-key $FINVIZ_API_KEY \
  --output-dir reports/

# With FMP API key for enhanced stock data
python3 skills/theme-detector/scripts/theme_detector.py \
  --fmp-api-key $FMP_API_KEY \
  --output-dir reports/

# Custom limits
python3 skills/theme-detector/scripts/theme_detector.py \
  --max-themes 5 \
  --max-stocks-per-theme 10 \
  --output-dir reports/

# Explicit FINVIZ mode
python3 skills/theme-detector/scripts/theme_detector.py \
  --finviz-mode public \
  --output-dir reports/

Expected Execution Time:

• FINVIZ Elite mode: ~2-3 minutes (14+ themes)
• Public FINVIZ mode: ~5-8 minutes (rate-limited scraping)

Step 3: Read and Parse Detection Results

The script generates two output files:

• theme_detector_YYYY-MM-DD_HHMMSS.json - Structured data for programmatic use
• theme_detector_YYYY-MM-DD_HHMMSS.md - Human-readable report

Read the JSON output to understand quantitative results:

# Find the latest report
ls -lt reports/theme_detector_*.json | head -1

# Read the JSON output
cat reports/theme_detector_YYYY-MM-DD_HHMMSS.json

Step 4: Narrative Confirmation (optional sidecar)

Step 4 is optional. Quant output from Step 2 already caps confidence at Medium — that is the production default consumed by bridge.py, Dashboard sector card, and investment protocol. The sidecar described here can elevate top themes to High when material narrative support exists.

Codex-compatible path (v3.14.2+) — router-driven script:

python3 skills/theme-detector/scripts/narrative_confirm.py            # latest cache, top-5
python3 skills/theme-detector/scripts/narrative_confirm.py --top-n 8

The script:

• Walks the multi-model fallback chain via scripts/_shared/model_router.run_role("narrative_confirm", ...) — any of claude / gemini / codex can drive it.
• Writes cache/theme_detector_<ts>.narrative.json (sidecar) with confirmations[], narrate_mode = llm | skipped, model_used, and a bumped_count summary.
• On router exhaustion (every model over budget / cooldown / disabled), falls back to narrate_mode = skipped with empty bumps — never raises.

Claude WebSearch path (legacy) — for sessions running under Claude Code with WebSearch / WebFetch available, the analyst may still do manual searches and elevate Confidence in-conversation. The sidecar is preferred because it leaves a deterministic audit trail, but both paths produce the same downstream effect.

Downstream consumption:

• Default: read confidence from theme_detector_<ts>.json (capped at Medium).
• If theme_detector_<ts>.narrative.json exists AND a confirmation row has confidence_bump == "medium->high", bump that theme to High.
• Sidecar absence is the normal case — do NOT error.

Confidence ladder:

• Quantitative High + Narrative medium->high = High confidence
• Quantitative High + Narrative none = Medium (possible momentum divergence)
• Quantitative Low + Narrative medium->high = Medium (narrative may lead price)
• Quantitative Low + Narrative none = Low

Step 5: Analyze Results and Provide Recommendations

Cross-reference detection results with knowledge bases:

Reference Documents to Consult:

1. references/cross_sector_themes.md - Theme definitions and constituent industries
2. references/thematic_etf_catalog.md - ETF exposure options by theme
3. references/theme_detection_methodology.md - Scoring model details
4. references/finviz_industry_codes.md - Industry classification reference

Analysis Framework:

For Hot Bullish Themes (Heat >= 70, Direction = Bullish):

• Identify lifecycle stage (Emerging = opportunity, Mature/Exhausting = caution)
• List top-performing industries within the theme
• Recommend proxy ETFs for exposure
• Flag if ETF proliferation is high (crowded trade warning)

For Hot Bearish Themes (Heat >= 70, Direction = Bearish):

• Identify industries under pressure
• Assess if bearish momentum is accelerating or decelerating
• Recommend hedging strategies or sectors to avoid
• Note potential mean-reversion opportunities if lifecycle is Mature/Exhausting

For Emerging Themes (Heat 40-69, Lifecycle = Emerging):

• These may represent early rotation signals
• Recommend monitoring with watchlist
• Identify catalyst events that could accelerate the theme

For Exhausted Themes (Heat >= 60, Lifecycle = Exhausting):

• Warn about crowded trade risk
• High ETF count confirms excessive retail participation
• Consider contrarian positioning or reducing exposure

Step 6: Generate Final Report

Present the final report to the user using the report template structure:

# Theme Detection Report
**Date:** YYYY-MM-DD
**Mode:** FINVIZ Elite / Public
**Themes Analyzed:** N
**Data Quality:** [note any limitations]

## Theme Dashboard
[Top themes table with Heat, Direction, Lifecycle, Confidence]

## Bullish Themes Detail
[Detailed analysis of bullish themes sorted by Heat]

## Bearish Themes Detail
[Detailed analysis of bearish themes sorted by Heat]

## All Themes Summary
[Complete theme ranking table]

## Industry Rankings
[Top performing and worst performing industries]

## Sector Uptrend Ratios
[Sector-level aggregation if uptrend data available]

## Methodology Notes
[Brief explanation of scoring model]

Save the report to reports/ directory.

---

Resources

Scripts Directory (scripts/)

Main Scripts:

• theme_detector.py - Main orchestrator script

  • Coordinates industry data collection, theme classification, and scoring
  • Generates JSON + Markdown output
  • Usage: python3 theme_detector.py [options]

• theme_classifier.py - Maps industries to cross-sector themes

  • Reads theme definitions from cross_sector_themes.md
  • Calculates theme-level aggregated scores
  • Determines direction (bullish/bearish) from constituent industries
  • Display mapping: "bullish" → "LEAD", "bearish" → "LAG" (see report_generator.py::_direction_label())

• finviz_industry_scanner.py - FINVIZ industry data collection

  • Elite mode: CSV export with full stock data per industry
  • Public mode: Web scraping with rate limiting
  • Extracts: performance, volume, change%, avg volume, market cap

• calculators/lifecycle_calculator.py - Lifecycle maturity assessment

  • Duration scoring, extremity clustering, valuation analysis
  • ETF proliferation scoring from thematic_etf_catalog.md
  • Stage classification: Emerging / Accelerating / Trending / Mature / Exhausting

• report_generator.py - Report output generation

  • Markdown report from template
  • JSON structured output
  • Theme dashboard formatting

References Directory (references/)

Knowledge Bases:

• cross_sector_themes.md - Theme definitions with industries, ETFs, stocks, and matching criteria
• thematic_etf_catalog.md - Comprehensive thematic ETF catalog with counts per theme
• finviz_industry_codes.md - Complete FINVIZ industry-to-filter-code mapping
• theme_detection_methodology.md - Technical documentation of the 3D scoring model

Assets Directory (assets/)

• report_template.md - Markdown template for report generation with placeholder format

---

Important Notes

FINVIZ Mode Differences

| Feature | Elite Mode | Public Mode | |---------|-----------|-------------| | Industry coverage | All ~145 industries | All ~145 industries | | Stocks per industry | Full universe | ~20 stocks (page 1) | | Rate limiting | 0.5s between requests | 2.0s between requests | | Data freshness | Real-time | 15-min delayed | | API key required | Yes ($39.50/mo) | No | | Execution time | ~2-3 minutes | ~5-8 minutes |

Direction Detection Logic

Theme direction is determined by majority vote of constituent industries' relative rank:

1. Industry ranking: All ~145 industries are ranked by multi-timeframe momentum score
2. Rank-based direction: Industries in the top half of the ranked list are classified as "bullish"; bottom half as "bearish"
3. Theme majority vote: _majority_direction() counts bullish vs. bearish industries within each theme; the majority wins

Display mapping: "bullish" → LEAD, "bearish" → LAG (see report_generator.py::_direction_label())

A LEAD theme indicates relative outperformance of its constituent industries. A LAG theme may still have positive absolute returns — it indicates relative underperformance, not a short signal.

Known Limitations

1. Survivorship bias: Only analyzes currently listed stocks and ETFs
2. Lag: FINVIZ data may lag intraday moves by 15 minutes (public mode)
3. Theme boundaries: Some stocks fit multiple themes; classification uses primary industry
4. ETF proliferation: Catalog is static and may not capture very new ETFs
5. Narrative scoring: WebSearch-based and inherently subjective
6. Public mode limitation: ~20 stocks per industry may miss small-cap signals

Disclaimer

This analysis is for educational and informational purposes only.

• Not investment advice
• Past thematic trends do not guarantee future performance
• Theme detection identifies momentum, not fundamental value
• Conduct your own research before making investment decisions

---

Version: 1.0 Last Updated: 2026-02-16 API Requirements: FINVIZ Elite (recommended) or public mode (free); FMP API optional Execution Time: ~2-8 minutes depending on mode Output Formats: JSON + Markdown Themes Covered: 14+ cross-sector themes