paolomoz/site-auditor

Site Auditor

Quick Reference

| Category | Trigger | Complexity | Source | |----------|---------|------------|--------| | audit | "auditing a website", "finding content gaps", "site quality audit", "content inventory analysis" | High | 8 projects |

Perform a comprehensive site audit by fetching and cross-referencing three independent data sources — the query index JSON, the XML sitemap, and live navigation links. The audit identifies 11 categories of content gaps, metadata issues, and quality problems, producing a structured analysis report that downstream skills (pagespeed-audit, report-hub-generator) can consume.

When to Use

• User wants to understand the overall health of a website's content
• User needs to find pages that exist in one system but not another (index vs. sitemap mismatches)
• User is looking for stale content that has not been updated in over 12 months
• User needs a content inventory with categorization by path prefix
• User wants to detect missing metadata (descriptions, OG images) before a launch or redesign
• User is migrating or restructuring a site and needs a baseline audit
• A downstream skill (pagespeed-audit, report-hub-generator) requests a site audit as input

Instructions

Step 1: Collect Data from Three Sources

Every audit begins by fetching three independent data sources from the target origin. All three are required for a complete audit; if one fails, proceed with the remaining sources and note the gap in the summary.

Source 1: Query Index JSON

Fetch {origin}/query-index.json. This is the primary content database, typically generated by a CMS or static site generator.

Expected response shape:

{
  "total": 847,
  "offset": 0,
  "limit": 512,
  "data": [
    {
      "path": "/blog/2024/site-performance-tips",
      "title": "10 Tips for Better Site Performance",
      "description": "Learn how to optimize your website for Core Web Vitals and user experience.",
      "image": "/media_1a2b3c.png",
      "lastModified": "1709251200"
    }
  ]
}

Key fields: path (required), title, description, image, lastModified (Unix timestamp as string).

Handle pagination: if total > offset + limit, fetch subsequent pages with ?offset={next} until all entries are collected. Concatenate all data[] arrays into a single list.

Source 2: Sitemap XML

Fetch {origin}/sitemap.xml. Parse all <loc> tags to extract URLs.

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://example.com/blog/2024/site-performance-tips</loc></url>
  <url><loc>https://example.com/about</loc></url>
</urlset>

Handle sitemap indexes: if the response contains <sitemapindex>, recursively fetch each <sitemap><loc> entry and merge all URLs.

Convert absolute URLs to relative paths by stripping the origin. Apply path normalization (see Step 2).

Source 3: Navigation Links

Fetch the homepage ({origin}/) and parse all <a href> attributes from navigation elements. Target these selectors in priority order:

1. nav a[href] — semantic navigation elements
2. header a[href] — header links (often includes main nav)
3. footer a[href] — footer links (often includes sitemap-style navigation)
4. .nav a[href], .navigation a[href] — class-based navigation

Filter to internal links only (same origin or relative paths). Remove duplicates and apply path normalization.

---

Step 2: Normalize All Paths

Before any comparison, normalize every path from every source using these rules:

1. Remove trailing slashes: /about/ becomes /about
2. Remove hash fragments: /page#section becomes /page
3. Remove query parameters: /page?ref=nav becomes /page
4. Lowercase the path: /About-Us becomes /about-us
5. Decode URL-encoded characters: /%2Fblog becomes /blog
6. Ensure leading slash: about becomes /about

function normalizePath(rawPath) {
  try {
    let path = rawPath.startsWith('http')
      ? new URL(rawPath).pathname
      : rawPath

    path = decodeURIComponent(path)
      .toLowerCase()
      .split('#')[0]
      .split('?')[0]
      .replace(/\/+$/, '')

    if (!path.startsWith('/')) path = '/' + path
    return path || '/'
  } catch {
    return null  // Discard unparseable paths
  }
}

Discard any path that normalizes to null. Log discarded paths for debugging but do not include them in gap analysis.

---

Step 3: Categorize Paths by Prefix

Group all collected paths by their top-level prefix to enable section-level analysis. Extract the first two path segments as the category key:

/developer/guides/getting-started → /developer/guides
/blog/2024/performance-tips       → /blog/2024
/products/analytics               → /products
/about                            → /about

If a path has only one segment, use that segment as the category. Track counts per category per source (index, sitemap, nav) to spot sections that are underrepresented in any source.

---

Step 4: Run Gap Analysis Across All 11 Categories

Compare the three normalized path sets against each other and against metadata quality criteria. For each gap found, record the path, the gap category, and any relevant metadata.

Category 1: In Index but Not Sitemap

Pages present in query-index.json but missing from sitemap.xml. These pages exist in the CMS but are invisible to search engines.

Severity: High — direct SEO impact.

Category 2: In Sitemap but Not Index

URLs in sitemap.xml with no corresponding entry in query-index.json. These may be orphaned pages, redirects, or pages excluded from the content index.

Severity: Medium — may indicate stale sitemap entries or content pipeline issues.

Category 3: Stale Content

Pages in the index where lastModified is older than 12 months from the current date. Convert the Unix timestamp string to a date and compare:

function isStale(lastModifiedStr) {
  const lastModified = new Date(parseInt(lastModifiedStr) * 1000)
  const twelveMonthsAgo = new Date()
  twelveMonthsAgo.setMonth(twelveMonthsAgo.getMonth() - 12)
  return lastModified < twelveMonthsAgo
}

Severity: Low to Medium — depends on content type (blog posts age naturally, product pages should stay current).

Category 4: Missing Descriptions

Index entries where description is empty, null, undefined, or shorter than 20 characters (too short to be meaningful).

Severity: Medium — affects search snippets and social sharing.

Category 5: Missing OG Images

Index entries where image is empty, null, or points to a default/placeholder image. Detect placeholders by checking for common patterns: placeholder, default, no-image, or images smaller than 200x200 if dimensions are available.

Severity: Medium — affects social media card rendering and visual search results.

Category 6: Duplicate Titles

Two or more index entries sharing the exact same title after trimming whitespace. Group duplicates together and flag each group.

Severity: Medium — confuses search engines and users navigating search results.

Category 7: Broken Navigation Links

Paths found in navigation (Source 3) that do not appear in either the index or the sitemap. These are links visible to users that may lead to 404 pages.

Severity: High — directly impacts user experience.

Category 8: Old Branding References

Index entries where title or description contains outdated brand terms. Build a configurable list of old brand terms to scan for (e.g., old company names, retired product names, deprecated terminology).

Common patterns to check:

• Previous company names after a rebrand
• Retired product line names
• Old taglines or slogans
• Deprecated feature names

Severity: Medium — brand consistency issue.

Category 9: Pages in Navigation but Not Index

Subset of Category 7 — links in the nav that appear in the sitemap but NOT in the index. These pages are navigable and crawlable but missing from the content database.

Severity: Medium — content pipeline gap.

Category 10: Deprecated Pages

Index entries whose path contains signals of deprecation: /deprecated/, /archive/, /legacy/, /old/, /v1/, /v2/ (when v3+ exists). Also flag pages with titles containing "deprecated", "archived", "legacy", or "end of life".

Severity: Low — informational, but should be reviewed for removal from active navigation.

Category 11: Lab/Experimental Pages

Index entries whose path contains /lab/, /labs/, /experimental/, /beta/, /preview/, /sandbox/. These may be internal-only pages accidentally exposed in the index or sitemap.

Severity: Low to Medium — potential unintended public exposure.

---

Step 5: Perform Freshness Analysis

Beyond the binary stale/fresh check in Category 3, build a freshness distribution:

{
  "freshnessDistribution": {
    "last30days": 42,
    "last90days": 128,
    "last6months": 234,
    "last12months": 387,
    "older": 460
  },
  "oldestPage": { "path": "/about", "lastModified": "2019-03-15" },
  "newestPage": { "path": "/blog/2024/latest", "lastModified": "2024-12-01" },
  "averageAgeDays": 342
}

This helps the user understand the overall content velocity and identify sections that are systematically neglected.

---

Step 6: Generate the Output

Write all output to data/audit/ relative to the project root.

File 1: data/audit/query-index.json

The raw query index data as fetched (all pages concatenated if paginated). This serves as a cache for downstream skills.

File 2: data/audit/analysis.json

The structured analysis report:

{
  "meta": {
    "origin": "https://example.com",
    "auditDate": "2024-12-15T10:30:00Z",
    "sources": {
      "queryIndex": { "status": "ok", "count": 847 },
      "sitemap": { "status": "ok", "count": 912 },
      "navigation": { "status": "ok", "count": 64 }
    }
  },
  "summary": {
    "totalUniquePages": 934,
    "gapCount": 127,
    "criticalGaps": 23,
    "categories": {
      "/blog": { "index": 340, "sitemap": 355, "nav": 5 },
      "/developer": { "index": 210, "sitemap": 198, "nav": 18 }
    }
  },
  "gaps": {
    "inIndexNotSitemap": ["/blog/draft-post", "/internal/test-page"],
    "inSitemapNotIndex": ["/old-landing-page", "/event/2022-conference"],
    "brokenNavLinks": ["/products/retired-product"]
  },
  "metadata": {
    "missingDescriptions": ["/about", "/contact"],
    "missingOgImages": ["/blog/2023/quick-update"],
    "duplicateTitles": [
      { "title": "Home", "paths": ["/", "/home", "/index"] }
    ]
  },
  "quality": {
    "stalePages": [{ "path": "/blog/2022/old-post", "lastModified": "2022-01-15" }],
    "oldBranding": [{ "path": "/about", "match": "OldCo Inc." }],
    "deprecatedPages": ["/legacy/v1-api"],
    "labPages": ["/labs/experimental-feature"]
  },
  "freshness": {
    "distribution": { "last30days": 42, "last90days": 128 },
    "averageAgeDays": 342
  },
  "allEntries": [
    {
      "path": "/blog/2024/site-performance-tips",
      "title": "10 Tips for Better Site Performance",
      "inIndex": true,
      "inSitemap": true,
      "inNav": false,
      "gaps": [],
      "category": "/blog/2024"
    }
  ]
}

The allEntries array is a denormalized view of every unique path across all three sources, with boolean flags for each source and a list of applicable gap categories. This is the primary input for report-hub-generator.

---

Step 7: RUM Integration (Optional)

If Real User Monitoring data is available (see pagespeed-audit skill for RUM bundle format), cross-reference audit findings with traffic data:

• Stale pages WITH traffic: High priority to refresh — users are actively visiting outdated content
• Missing OG images WITH social referral traffic: Critical — broken social cards are actively hurting click-through rates
• Pages in index but NOT sitemap WITH organic search traffic: Surprising — these pages are getting traffic despite being invisible to crawlers (possibly through internal links)
• Broken nav links WITH clicks: Critical — users are actively encountering dead links

Add a rumEnriched boolean to the meta object and include traffic-weighted severity scores when RUM data is available.

---

Step 8: Present Results to the User

Summarize findings in a clear, prioritized format:

1. Critical issues (broken nav links, high-traffic stale pages): Fix immediately
2. SEO issues (index/sitemap mismatches, missing descriptions): Fix within a sprint
3. Quality issues (old branding, duplicate titles): Fix during content review cycles
4. Informational (deprecated pages, lab pages): Review and decide

Always tell the user the exact counts and provide the top 5 examples for each category. Point them to data/audit/analysis.json for the complete dataset.

---

Troubleshooting

| Problem | Cause | Fix | |---------|-------|-----| | Query index returns 404 | Site does not use a query index | Skip this source; audit with sitemap + nav only. Note reduced coverage in summary. | | Sitemap returns a sitemap index | Nested sitemaps | Recursively fetch all child sitemaps and merge URLs | | Navigation returns thousands of links | Mega-menu or footer with every page linked | Filter to unique paths and apply a max of 500 nav links | | lastModified is "0" or missing | CMS does not track modification dates | Exclude from freshness analysis; flag as "unknown freshness" | | All titles are the same | Site uses a default title template | Flag as duplicate titles but note the pattern (likely a CMS configuration issue, not a content issue) | | Audit takes too long | Large site with 10K+ pages | Process in batches of 500; write intermediate results to disk |

Cross-References

• pagespeed-audit: Provides RUM data for traffic-weighted severity scoring and Core Web Vitals per page
• accessibility-auditor: Complements content audit with accessibility compliance checks
• report-hub-generator: Consumes analysis.json to produce formatted audit reports (PDF, HTML, dashboard)
• brand-extractor: Provides old branding terms list for Category 8 detection