oaustegard/generating-lattice

Generating Lattice

Generate and maintain lat.md/ knowledge graphs — structured, cross-referenced markdown anchored to source code symbols. The output is validated by the lat CLI, which catches documentation drift whenever code or docs change.

The core mechanism is bidirectional linking:

• Top-down (docs → code): [[src/auth.js#getValidToken]] wiki links in lat.md/ files point to specific symbols in source. When a symbol is renamed or deleted, lat check catches the broken link.
• Bottom-up (code → docs): // @lat: [[auth#Token Refresh]] comments in source point back to the concept they implement. When a section is renamed or removed, lat check catches the dangling reference.

Without both directions, the lattice is a static essay that drifts silently. With both, lat check enforces consistency.

Dependency: Requires mapping-codebases skill. The structural maps (_MAP.md files) are the input that makes LLM-assisted authoring token-efficient — read maps to understand API surfaces, then selectively read source files only where design rationale lives.

Installation

npm install -g lat.md
lat --version  # verify (requires Node.js 22+)

The lat CLI provides lat check (validation), lat search (semantic search), lat section (browsing), lat refs (reference lookup), and lat mcp (agent integration).

Generation Pipeline

Phase 1: Structural Scan

Ensure _MAP.md files exist. Generate them if missing:

# Install mapping-codebases dependencies
uv venv /home/claude/.venv
uv pip install tree-sitter-language-pack --python /home/claude/.venv/bin/python

# Generate maps
/home/claude/.venv/bin/python /mnt/skills/user/mapping-codebases/scripts/codemap.py /path/to/repo \
  --skip tests,.github,node_modules,vendor

Read the root _MAP.md first for high-level orientation, then drill into subdirectory maps. The maps are the symbol inventory — they tell you what exists and where, so you can write source code links without reading every file.

Phase 2: Selective Source Reading

Maps reveal the API surface (exports, signatures, line numbers) without reading full files. Read source selectively to understand design rationale:

Read fully: Files with complex algorithms, business logic, configuration constants, data schemas. The maps identify these by export density and file size.

Read partially: Header comments and constant blocks in large files.

Skip: Files where the map signature tells the full story (thin wrappers, simple CRUD, format utilities).

Phase 3: Generate Anchored Sections

Create lat.md/ directory and generate markdown files. Every section must be anchored to source code symbols — this is what makes lat check catch drift.

Section Structure Rules

• Every section MUST have a leading paragraph ≤250 characters (excluding [[wiki link]] content). This is the section's identity in search results.
• lat check validates this rule.

# Good Section

Brief overview anchored to [[src/auth.js#refreshToken]] and [[src/api.js#fetchWithRetry]].

More detail can follow in subsequent paragraphs.

Source Code Links (the top-down anchors)

Use [[src/path/file.ext#symbolName]] wiki links to anchor documentation to specific symbols. Supported extensions: .ts, .tsx, .js, .jsx, .py, .rs, .go, .c, .h.

## Token Refresh

OAuth token lifecycle managed by [[src/auth.js#getValidToken]] with automatic
refresh via [[src/auth.js#refreshToken]]. The refresh window
([[src/auth.js#REFRESH_THRESHOLD_MS]]) triggers proactive renewal before expiry.

Symbol granularity matters. Link to the specific function, class, constant, or method — not just the file. [[src/auth.js#refreshToken]] is useful; [[src/auth.js]] tells you nothing that the filename didn't already say.

For class methods: [[src/server.ts#App#listen]]. For Rust impl methods: [[src/lib.rs#Greeter#greet]]. For Python: [[lib/utils.py#parse_args]].

Section-to-Section Links

Use [[file#Section#Subsection]] or short form [[file#Section]] when the file stem is unique:

The sync pipeline ([[data#Sync Pipeline]]) uses the same token refresh
mechanism described in [[auth#Token Refresh]].

Conceptual Grouping

Organize by domain concept, not file structure. A data.md file might reference db.js, sync.js, and demo.js because they're all part of the data layer. The _MAP.md files already provide file-level structure; lat.md/ adds the semantic layer explaining WHY things connect.

Index File

lat.md/lat.md must contain a bullet list of all files with one-sentence descriptions using wiki links. lat check --index validates completeness:

- [[architecture]] — System design, OAuth flow, routing, signals
- [[data]] — IndexedDB schema, sync pipeline, demo mode
- [[awards]] — Award computation engine, data quality rules

Subdirectories need their own index: lat.md/api/api.md.

Phase 4: Add Back-Links in Source Code

Add // @lat: or # @lat: comments in source code pointing back to lat.md/ sections. This is not optional — back-links are what make lat check detect when source code changes break documentation.

Run the back-link helper to identify where annotations are needed:

python3 SKILL_DIR/scripts/suggest_backlinks.py /path/to/repo

This parses all [[src/...]] wiki links from lat.md/ files, looks up referenced symbols in _MAP.md files for O(1) line number resolution, and suggests @lat: comment placements. Requires _MAP.md files from Phase 1.

Comment syntax by language:

• JS/TS/Rust/Go/C: // @lat: [[section#Subsection]]
• Python: # @lat: [[section#Subsection]]

Place one @lat: comment per section reference, at the relevant code location — not at the top of the file:

// @lat: [[auth#Token Refresh]]
async function refreshToken(token) { ... }

To auto-apply suggestions (review the output first):

python3 SKILL_DIR/scripts/suggest_backlinks.py /path/to/repo --apply

Phase 4b: Annotate Maps with Lattice Cross-References

After backlinks exist in source, annotate _MAP.md files so someone browsing a code map can find the lat.md section that explains WHY a module is designed the way it is:

python3 SKILL_DIR/scripts/annotate_maps.py /path/to/repo

This scans @lat: comments in source and adds > Documented in: lines to _MAP.md file headers. Idempotent — safe to re-run after regenerating maps.

Example output in _MAP.md:

### auth.js
> Documented in: [[auth#Token Refresh]], [[auth#Server]]
> Imports: `express`
- **refreshToken** (f) `(token)` :15

require-code-mention for Critical Sections

For sections where bidirectional traceability is essential (test specs, invariants, key business rules), add frontmatter:

---
lat:
  require-code-mention: true
---

This makes lat check fail if ANY leaf section in the file lacks a corresponding @lat: comment in the codebase. Use for test spec files where every test must trace to its specification.

Phase 5: Validate

cd /path/to/repo
lat check

This runs ALL checks:

• md — every [[wiki link]] in lat.md/ resolves to a real section or source symbol
• code-refs — every @lat: comment in source points to a real section; and every leaf section in require-code-mention files is referenced by code
• sections — every section has a leading paragraph ≤250 chars
• index — every directory in lat.md/ has a complete index file

All four must pass. Fix errors iteratively until lat check is clean.

Phase 6: Agent Integration

Set up files that make coding agents maintain the lattice automatically.

Cache exclusion — create lat.md/.gitignore:

.cache/

Agent instructions — append to CLAUDE.md (or AGENTS.md) using markers:

%% lat:begin %%
# Before starting work

- Run `lat search` to find sections relevant to your task.
- Run `lat expand` on user prompts to expand any `[[refs]]`.

# Post-task checklist (REQUIRED — do not skip)

After EVERY task, before responding to the user:

- [ ] Update `lat.md/` if you changed functionality, architecture, or behavior
- [ ] Run `lat check` — all wiki links and code refs must pass
%% lat:end %%

For full agent integration (hooks that auto-inject search results and block completion when lat check fails), run lat init interactively. This sets up agent hooks in .claude/settings.json, .cursor/hooks.json, etc. The hooks require absolute paths to the lat binary, which is machine-local.

Drift Prevention: How It Works

The lattice stays in sync because changes to either side break lat check:

| What changed | What breaks | How lat check catches it | |---|---|---| | Symbol renamed in source | [[src/file#oldName]] wiki links in lat.md/ | lat check --md reports broken source ref | | Symbol deleted | Same as above | Same — dead source link | | Section renamed in lat.md/ | @lat: [[old#Section]] comments in source | lat check --code-refs reports dangling ref | | Section deleted | Same as above | Same — orphaned code ref | | New code added without docs | No back-link for new functions | Manual review (or require-code-mention enforcement) | | New docs without code anchors | Sections with no [[src/...]] links | Manual review — but this is the failure to prevent |

The critical gap: new code without docs and new docs without anchors are NOT caught automatically unless require-code-mention is set. The agent integration hooks (Phase 6) address this by reminding agents to update lat.md/ after every task and running lat check before completion.

Quality Criteria

A good generated lattice:

• Passes lat check on all four checks — md, code-refs, sections, index
• Has dense source code links — most sections reference specific symbols via [[src/...#symbol]], not just file-level links
• Has back-links — source code has @lat: comments pointing to the sections that describe them
• Captures WHY, not just WHAT — design rationale, invariants, constraints
• Groups by concept — not a 1:1 mirror of the file tree
• Concise leading paragraphs — ≤250 chars, the section's identity in search

Anti-patterns (what our v0.1 got wrong):

• Sections with only section-to-section links ([[clusters#Visual Pipeline]]) and no source anchors — lat check cannot catch code changes
• Architectural essays that describe the codebase without linking to it — a static document, not a knowledge graph
• Treating back-links as optional — without @lat: comments, the bottom-up view doesn't exist and half the drift detection is missing

Token Budget

The maps-first approach significantly reduces LLM token cost:

• Mapping-codebases (Phase 1): zero LLM tokens — pure AST extraction
• Selective reading (Phase 2): ~30-50% of source bytes vs reading everything
• Generation (Phase 3): the actual LLM work — proportional to conceptual complexity, not codebase size
• Back-links (Phase 4): zero LLM tokens — the suggest_backlinks.py script is deterministic
• Validation (Phase 5): zero LLM tokens — deterministic lat check