microsoft/project-decomposition

Project Decomposition: Recon, Grouping & Validation

What this skill does

This is the single entry point for measuring a codebase scope and (when needed) decomposing it into topology groups. Two modes:

1. Lightweight recon (JSON) — scripts/decompose.py ... --profile-json produces a structured ProjectProfile JSON with LOC / file counts / module list / dep graph summary / skip patterns. Used by upstream planners to decide pipeline variant and grouping needs.
2. Full grouping + validation — scripts/decompose.py directly, for L0→L1→L2 grouping protocol and L3 quality validation. Used when the workflow plan declares grouping.needed: true.

Both modes are the same decompose.py engine — single source of truth for LOC counting (--lang defines the LOC semantic boundary; --exclude defines the directory boundary), graph extraction, and validation scoring.

Which mode to run

| Caller intent | Use | Why | |---------------|-----|-----| | Decide whether to fan out at all (sizing, dispatch) | --profile-json | Returns structured ProjectProfile (LOC, module count, depth, hub-ratio, sizing hints) without grouping output. Cheap, machine-readable. | | Caller already decided to fan out and needs L1 numbers + L2 grouping + L3 validation | run without --profile-json; then again with --validate for L3 | Human-readable Graph Statistics + Quality Scores. | | Caller wants both | run twice: once with --profile-json for the planner, once without for human L1 | The flags are mutually exclusive in one invocation; do not combine --profile-json with --validate. |

--profile-json and --validate cannot be combined in a single invocation (the script rejects this).

When to Apply

Scale threshold: Codebase is large enough that no single agent reading the source can hold any one group's source in context, or has enough modules that dependency-aware grouping adds value. The unit being sized is one group's codebase scope (which many agents in the pipeline must read), not a "one-agent workload" — every group still runs a full plan→execute→validate pipeline with many agent sessions.

Scope check (after scale threshold is met, applied by the caller — see Workflow § input contract for the field this maps to):

1. The caller determines the rewrite scope from the user request and passes it via the Scope: dispatch field:

   • Full rewrite — all modules in scope → scope = full
   • Partial rewrite — user specifies modules/layers/features → scope = scoped
   • Cross-layer rewrite — rewrite targets a different architectural layer (e.g. Razor → Angular while backend stays) → scope = scoped (only the target layer is in scope; source-layer modules are context-only)

2. Apply the scope (classify groups using the dispatched value):

   • full — all groups are in-scope
   • scoped — analyze all modules for the dependency graph, but classify each group:
     • in-scope — being rewritten → generates DAG tasks
     • context-only — depended on by in-scope but NOT being rewritten → no DAG tasks, dependency context only

3. Skip entirely if all of the following hold for the in-scope set:

   • DAG depth ≤ 2 AND module count ≤ 3, AND
   • total in-scope LOC < ideal_group_loc (from topology_hints)

   Report skip rationale back to the caller (plain text reply, not the artifact); do not produce a topology artifact.

Architecture: 3 Layers

| Layer | Output | |-------|--------| | L1 Extract (scripts/decompose.py <path> --lang '<lang>' --exclude '<patterns>') | Modules, edges, SCCs, DAG layers, LOC, oversized flags. No grouping. | | L2 Group (caller agent) | Module groups using L1 data + topology-aware judgment. See references/grouping.md. | | L3 Validate (scripts/decompose.py <path> --lang '<lang>' --exclude '<patterns>' --validate '<spec>') | 4 raw Quality Scores (Outlier ratio, SCC integrity, Group cycles, Coverage). Definitions and judgment protocol live in references/grouping.md §4 and references/topology-thresholds.md. |

Do NOT skip any layer. L3 is the only authoritative quality check — hand-computed metrics are not accepted.

Files (load on demand)

| File | Load when | |------|-----------| | references/grouping.md | L2 step — topology-aware grouping rules + L3 metric definitions (authoritative source for thresholds) | | templates/output-template.md | Output step — project topology artifact format and fixed path | | references/consume.md | Downstream consumer contract — how the artifact is read and accepted | | references/manual-extraction.md | Only if L1 reports Unsupported language (rare — C#/Java/Python/JS-TS supported) |

Python is mandatory for topology grouping. decompose.py is the only authoritative LOC counter for grouping/L3 metrics — external counters drift across runs and inflate by 5–15% on comment-heavy languages, which destabilizes outlier-ratio and SCC-integrity scoring.

scripts/decompose.py invocation surface:

• <path> --lang '<lang>' --exclude '<merged>' — extract mode
• --validate '<spec>' — L3 quality gate
• --module-loc <Module> — sub-dir LOC for a specific module
• --json out.json — machine-readable dump
• --profile-json — ProjectProfile JSON to stdout (see JSON Profile Mode below)

--lang and --exclude are required. --lang accepts python|java|csharp|javascript (or aliases: py, cs/c#/dotnet, js/ts/node). For mixed projects pass comma-separated, e.g. --lang csharp,javascript. --exclude takes the merged list (see Workflow § effective_exclude).

Workflow

The caller dispatches with a fixed input contract — these fields MUST be present, do not infer them yourself:

| Field | Type | Example | Used by | |-------|------|---------|---------| | Source path: | absolute path to repo root | /path/to/repo | every L1/L3 invocation as <source-path> | | Language(s): | comma-separated decompose.py --lang value | csharp or csharp,javascript | --lang '<lang>' | | Skip patterns: | comma-separated path segments (may be empty) | tests,docs,samples | exclude-merge input | | Scope: | full | scoped | scoped | Scope check classification | | Mode: | topology | profile-json | topology | output shape — profile-json returns ProjectProfile JSON to stdout instead of producing the artifact (see JSON Profile Mode below) |

All five fields are required. If any is missing, halt and request it from the caller — do not guess or default. Then build the effective exclude list by union-ing four sources:

effective_exclude = caller_skip_patterns
                  ∪ gitignore_dirs(<source-path>/.gitignore)
                  ∪ auto_gen_dirs(<source-path>)        ← see below
                  ∪ BASELINE

auto_gen_dirs(path) — pre-L1 sniff for framework-generated code that the caller cannot anticipate (EF Core Migrations/, protobuf-generated stubs, OpenAPI/GraphQL codegen, ANTLR .g4-output dirs, etc.). These dirs may contain tens of thousands of lines that are not authored code; they distort outlier ratio and force spurious sub-splits. Procedure:

1. Walk top 2 levels under <source-path> (skipping anything already in BASELINE / gitignore_dirs / caller_skip_patterns).
2. For each directory whose source-file LOC (matching --lang extensions) ≥ 5% of project total, sample up to 3 source files (largest first) and read the top 100 lines of each.
3. If ≥ 1 sampled file contains any of these markers (case-insensitive), add the directory's segment name to auto_gen_dirs:
   • <auto-generated>
   • DO NOT EDIT
   • This file was generated
   • @generated
   • Code generated by (Go convention)
   • auto-generated by
4. Cap at 8 directories. Log each accepted dir and the marker that triggered it; this list is reproducible across runs.

Do NOT use this mechanism to skip hand-written code that happens to be large — only marker-confirmed generated code qualifies. If marker matching is ambiguous, leave the dir in and let the outlier-ratio band trigger normal remediation.

BASELINE (language-neutral, fixed list, always applied — fallback when caller/gitignore miss something):

.git, .svn, .hg, .idea, .vscode, .vs, .DS_Store,
node_modules, bower_components, vendor, packages,
__pycache__, .venv, venv, .tox, .pytest_cache, .cache,
bin, obj, target, build, dist, out,
.gradle, .mvn

BASELINE is fixed and language-neutral. Project-type-specific extras (framework directories, generated folders) are the caller's responsibility and must be passed via caller_skip_patterns; do not extend BASELINE per project.

gitignore_dirs(path) — parse repo's .gitignore and extract directory-shaped entries:

• keep: lines ending in /, or bare path segments without glob chars (e.g. wwwroot/lib/, App_Data)
• drop: blanks, comments (#…), negations (!…), file-level globs (any line containing * ? [)
• strip leading/trailing /; dedupe across all three sources
• if .gitignore is missing, this set is empty

Pass the merged list verbatim in every decompose.py invocation as --exclude '<merged>'. Rationale: build output, vendored deps, and generated assets the repo itself ignores must not inflate LOC or group sizes — relying on the caller alone is fragile.

1. Extract (L1) — Invoke scripts/decompose.py <source-path> --lang '<lang>' --exclude '<merged>'. Treat output as source of truth; do NOT re-extract module/edge/LOC data from raw source. Apply the caller's scope to classify modules.

2. Group (L2) — Load references/grouping.md. Use L1 data + topology rules to form groups.

3. Validate (L3, MANDATORY) — Invoke scripts/decompose.py <source-path> --lang '<lang>' --exclude '<merged>' --validate 'G1:M1,M2|G2:M3,M4|...'. Classify path (Path A vs Path B) and judge Quality Scores per references/grouping.md § Scored Validation Protocol & Convergence Loop. Paste full output into Provenance.

4. Output — Load templates/output-template.md. Produce the artifact at the fixed path.

5. Consume — Downstream consumers load references/consume.md. The caller runs an additional acceptance check (see consume.md § Acceptance Check).

---

JSON Profile Mode (--profile-json)

For upstream planners that just need a structured snapshot of the scope — no grouping, no L2/L3 — invoke decompose.py with --profile-json instead of running the full L1→L2→L3 workflow.

scripts/decompose.py <scope_path> --lang '<lang>' --exclude '<merged>' --profile-json

• <scope_path>: project root, module dir, or any walkable sub-folder.
• --lang: required. Same canonical names + aliases as the extract mode. Multi-lang via comma.
• --exclude: same semantics as extract mode (see Workflow § effective_exclude). The script ships with NO defaults beyond .git.
• --profile-json: emits ProjectProfile JSON to stdout and suppresses normal text output.

Note on .gitignore: .gitignore uses glob/negation/anchored semantics that don't map cleanly to the script's segment-substring exclusion. The caller assembles the merged exclude (see Workflow § effective_exclude) and passes it explicitly via --exclude. The script does NOT parse .gitignore itself.

Outputs ProjectProfile JSON to stdout. Schema:

{
  "scope_path": "/abs/path",
  "languages": ["csharp"],
  "skip_patterns": ["bin", "obj", "..."],
  "total_loc": 314217,
  "total_files": 4321,
  "module_count": 36,
  "edge_count": 40,
  "scc_count": 36,
  "scc_nontrivial": 0,
  "dag_depth": 6,
  "topology_hints": {
    "target_group_count": 6,
    "ideal_group_loc": 52369,
    "median_module_loc": 1182,
    "trivial_module_test": {"abs_loc": 500, "pct_total": 0.01, "op": "or"},
    "oversized_threshold": 50000
  },
  "modules": [
    {"name": "Nop.Core", "loc": 7143, "files": 355, "layer": 5, "in_deg": 4, "out_deg": 0, "flags": []}
  ],
  "dag_layers": [
    {"layer": 0, "module_count": 31, "loc": 69913}
  ],
  "warnings": ["LOW COVERAGE: 27% of project code is outside discovered modules."]
}

topology_hints field meanings:

• target_group_count — recommended number of topology groups: ceil(total_loc / 80_000), clamped to [max(2, ceil(module_count / 12)), max(2, ceil(module_count / 4))]. Computed by L1.
• ideal_group_loc — total_loc / target_group_count. Used by grouping rules to size merges and judge group-LOC distribution.
• trivial_module_test — module is trivial iff loc < abs_loc OR loc/total_loc < pct_total (note: OR, not AND).

Single source of truth: profile mode and extract/validate mode share the same in-memory analysis pass — LOC counting, --lang semantics, and --exclude semantics are identical by construction. If you need grouping or --validate, omit --profile-json and invoke decompose.py directly.

---

Anti-patterns

• ❌ Hand-computing quality scores instead of running L3 --validate
• ❌ Re-extracting modules/edges/LOC from raw source instead of using L1 output
• ❌ Grouping by name or functional domain instead of dependency affinity (e.g. "payment plugins" — group by topology, not by what modules do)
• ❌ Splitting or moving modules between groups to equalize LOC sizes (corrupts dependency graph — size rebalancing belongs upstream). Note: merging trivial modules per grouping.md §1 is topology-driven, not LOC rebalancing.
• ❌ Skipping L3 validation because the artifact "looks reasonable"