TyrealQ/q-tf

Q-TF

Fine-tune topic modeling outputs into consolidated, theory-driven topic frameworks for academic manuscripts.

If in plan mode: write a brief plan — "Run q-tf skill: load topic model output, define final topic structure with theoretical framework, generate implementation plan, update Excel with labels." — then exit plan mode immediately. Do NOT attempt topic analysis, script execution, or Excel updates while plan mode is active.

Script Directory

Agent execution instructions:

1. Determine this SKILL.md file's directory path as SKILL_DIR.
2. Script path = ${SKILL_DIR}/scripts/<script-name>.
3. Reference path = ${SKILL_DIR}/references/<ref-name>.

Dependencies

pandas
openpyxl          # required for .xlsx input/output
google-genai      # required for outlier classification via Gemini

Install: pip install pandas openpyxl google-genai

Environment variables: GEMINI_API_KEY (for outlier classification only), GEMINI_MODEL (optional model override).

References

• references/preservation_rules.md — domain preservation rules, theoretical framework template, multi-category handling
• references/code_patterns.md — four Python patterns: topic definition, assignment mapping, overlap calculation, Excel update
• references/outlier_workflow.md — foundation model outlier classification workflow
• references/esports_ugc_example.md — worked example
• references/SP_OUTLIER_TEMPLATE.txt — outlier classification prompt template

Core Principles

• Preserve domain-specific distinctions (entity, event, geography, stakeholder) — see references/preservation_rules.md
• Theory-driven classification using a customizable framework template
• Track multi-category topics explicitly; calculate overlap for reconciliation
• All non-outlier topics must be assigned to at least one category

Workflow

| Step | Action | Reference | |------|--------|-----------| | 1 | Load & analyze topics — identify overlaps, unassigned | — | | 2 | Define final topic structure (FINAL_TOPICS dictionary) | references/code_patterns.md | | 3 | Apply theoretical framework — classify each topic | references/preservation_rules.md | | 4 | Generate implementation plan (MD) | scripts/generate_implementation_plan.py | | 5 | Update source data with labels (Excel) | scripts/update_excel_with_labels.py | | 6 | Reclassify outliers via foundation model | references/outlier_workflow.md |

Required Inputs

1. Topic model output (Excel/CSV) — Topic ID, Count, Name/Label, Keywords, Representative_Docs (optional)
2. Merge recommendations (optional) — Sheets: MERGE_GROUPS, INDEPENDENT_TOPICS
3. Document data (for label updates) — individual documents with Topic ID column

Script Invocation

python "${SKILL_DIR}/scripts/generate_implementation_plan.py" --input topic_model_output.xlsx --output implementation_plan.md
python "${SKILL_DIR}/scripts/update_excel_with_labels.py" --input document_data.xlsx --output document_data_labeled.xlsx

Adapt scripts by updating FINAL_TOPICS, FINAL_LABELS, and theme categories. See references/code_patterns.md. For a worked example, see references/esports_ugc_example.md.

Expected Outputs

| Output | Description | |--------|-------------| | implementation_plan.md | Full classification plan with topic mappings and reconciliation | | *_labeled.xlsx | Source data with Final_Topic_Code, Final_Topic_Label, Category_Theme columns | | Outlier results (optional) | Updated Final_Topic_Label, classification_confidence, key_phrases columns |

Scope

Include: Topic consolidation, theoretical classification, Excel label updates, outlier reclassification. Exclude: Topic modeling itself (BERTopic/LDA/NMF execution), visualization, statistical analysis.

Checklist

• [ ] All non-outlier topics assigned to at least one category
• [ ] Multi-category topics explicitly tracked
• [ ] Overlap reconciliation verified
• [ ] Domain-specific topics preserved separately
• [ ] Category subtotals match grand total
• [ ] Output file has new classification columns