pjt222/test-cli-application

Test a CLI Application

Write integration tests for a Node.js CLI using the built-in node:test module with execSync.

When to Use

• Adding tests to an existing CLI application
• Testing a newly created command
• Verifying adapter/plugin behavior across target frameworks
• Setting up CI that validates CLI correctness
• Catching regressions after refactoring CLI internals

Inputs

• Required: Path to the CLI entry point (e.g., cli/index.js)
• Required: Commands to test
• Optional: Framework adapters to test (dry-run mode)
• Optional: Cleanup requirements (files/symlinks created by tests)

Procedure

Step 1: Set Up Test Infrastructure

import { describe, it, before, after } from 'node:test';
import assert from 'node:assert/strict';
import { execSync } from 'child_process';
import { existsSync, rmSync } from 'fs';
import { resolve } from 'path';

const CLI = 'node cli/index.js';
const ROOT = process.cwd();

function run(args) {
  return execSync(`${CLI} ${args}`, {
    cwd: ROOT,
    encoding: 'utf8',
    timeout: 10000,
  });
}

Key design decisions:

• node:test is built-in — no test runner dependency needed
• execSync runs the CLI as a subprocess — tests the actual binary, not internal functions
• 10-second timeout prevents hanging on interactive prompts
• encoding: 'utf8' gives string output for regex matching
• All paths relative to ROOT for reproducibility

Expected: A test file that imports from node:test and has a working run() helper.

On failure: If node:test is not available, your Node.js version is below 18. Upgrade or use a polyfill.

Step 2: Write Smoke Tests

Smoke tests verify the CLI starts, parses arguments, and produces expected output shapes:

describe('meta', () => {
  it('shows version', () => {
    const out = run('--version');
    assert.match(out, /\d+\.\d+\.\d+/);
  });

  it('shows help with all commands', () => {
    const out = run('--help');
    assert.match(out, /install/);
    assert.match(out, /list/);
    assert.match(out, /detect/);
  });
});

describe('registry', () => {
  it('list shows expected counts', () => {
    const out = run('list --domains');
    assert.match(out, /\d+ domains/);
  });

  it('search finds known items', () => {
    const out = run('search "docker"');
    assert.match(out, /result\(s\) for "docker"/);
  });

  it('search returns 0 for nonsense', () => {
    const out = run('search "xyzzy-nonexistent"');
    assert.match(out, /0 result/);
  });
});

Smoke test patterns:

• --version and --help always work
• Registry loading validates data integrity
• Search with known and unknown terms

Expected: Smoke tests confirm the CLI is functional and data is loaded.

On failure: If registry counts change frequently, use \d+ instead of hardcoded numbers.

Step 3: Write Lifecycle Tests

Lifecycle tests verify create → verify → delete sequences with cleanup:

describe('install', () => {
  const testPath = resolve(ROOT, '.agents/skills/commit-changes');

  after(() => {
    // Always clean up, even if tests fail
    try { rmSync(testPath); } catch {}
    try { rmSync(resolve(ROOT, '.agents/skills'), { recursive: true }); } catch {}
    try { rmSync(resolve(ROOT, '.agents'), { recursive: true }); } catch {}
  });

  it('dry-run does not create files', () => {
    const out = run('install commit-changes --dry-run');
    assert.match(out, /DRY RUN/);
    assert.ok(!existsSync(testPath));
  });

  it('installs creates the target', () => {
    run('install commit-changes');
    assert.ok(existsSync(testPath));
  });

  it('skips already installed', () => {
    const out = run('install commit-changes');
    assert.match(out, /skipped/);
  });

  it('uninstall removes the target', () => {
    run('uninstall commit-changes');
    assert.ok(!existsSync(testPath));
  });
});

Cleanup rules:

• Use after() hooks, not afterEach() — lifecycle tests build on each other
• Wrap cleanup in try/catch — cleanup must not fail the test suite
• Clean from leaf to root (file → parent dir → grandparent dir)
• If the test modifies shared state (symlinks, config files), restore it

Expected: Tests run in sequence within the describe block, cleanup runs even on failure.

On failure: If tests run in parallel (non-default in node:test), force sequential with { concurrency: 1 }.

Step 4: Write Dry-Run Tests for Each Adapter

Test each adapter's target path without making changes:

describe('adapter: cursor (dry-run)', () => {
  it('targets .cursor/skills/ path', () => {
    const out = run('install commit-changes --framework cursor --dry-run');
    assert.match(out, /\.cursor\/skills/i);
  });
});

describe('adapter: copilot (dry-run)', () => {
  it('targets .github/ path', () => {
    const out = run('install commit-changes --framework copilot --dry-run');
    assert.match(out, /\.github/i);
  });
});

This pattern scales to any number of adapters. Each test:

• Uses --framework to bypass auto-detection
• Uses --dry-run so no files are created
• Asserts the target path appears in output

Expected: One describe block per adapter, each with at least a path assertion.

On failure: If the adapter doesn't exist in the project, the test will fail with "Unknown framework." This is correct — adapter tests should only exist for implemented adapters.

Step 5: Write Error Case Tests

describe('errors', () => {
  it('rejects unknown items', () => {
    assert.throws(
      () => run('install nonexistent-skill-xyz'),
      /No matching items|Unknown/,
    );
  });

  it('rejects unknown framework', () => {
    assert.throws(
      () => run('install commit-changes --framework nonexistent'),
      /Unknown framework/,
    );
  });

  it('handles missing state gracefully', () => {
    assert.throws(
      () => run('scatter nonexistent-team'),
      /not burning|Unknown/,
    );
  });
});

Error testing patterns:

• assert.throws catches non-zero exit codes from execSync
• Regex match on the error message (captured from stderr)
• Test both "item not found" and "invalid option" errors
• Verify error messages suggest corrective actions

Expected: All error paths produce non-zero exit codes and helpful messages.

On failure: execSync throws on non-zero exit. The error's stderr or stdout contains the message. Check error.stdout if assert.throws regex doesn't match.

Step 6: Write JSON Output Tests

describe('json output', () => {
  it('campfire --json outputs valid JSON', () => {
    const out = run('campfire --json');
    const data = JSON.parse(out);
    assert.ok(typeof data.totalTeams === 'number');
    assert.ok(Array.isArray(data.fires));
  });

  it('gather --dry-run --json outputs structured data', () => {
    const out = run('gather tending --dry-run --json');
    // JSON may follow a DRY RUN header — extract from first '{'
    const jsonStart = out.indexOf('{');
    assert.ok(jsonStart >= 0, 'Should contain JSON');
    const data = JSON.parse(out.slice(jsonStart));
    assert.equal(data.team, 'tending');
  });
});

JSON testing gotchas:

• Some commands prefix JSON with human-readable text (e.g., DRY RUN header)
• Extract JSON by finding the first { character
• Validate structure (key presence, types), not exact values
• Values like counts may change as content is added

Expected: JSON output is parseable and contains expected keys.

On failure: If JSON.parse fails, the command may be mixing human text with JSON. Either fix the command to output pure JSON in --json mode, or extract the JSON substring.

Step 7: Handle Cleanup and State Restoration

describe('stateful commands', () => {
  const stateDir = resolve(ROOT, '.agent-almanac');

  after(() => {
    // Remove state file created by tests
    try { rmSync(stateDir, { recursive: true }); } catch {}
  });

  // Tests that create/modify state...
});

// Restore symlinks that destructive tests may remove
describe('destructive tests', () => {
  after(() => {
    // Restore symlinks that scatter/uninstall removed
    const skills = ['heal', 'meditate', 'remote-viewing'];
    for (const skill of skills) {
      const link = resolve(ROOT, `.claude/skills/${skill}`);
      if (!existsSync(link)) {
        try {
          execSync(`ln -s ../../skills/${skill} ${link}`, { cwd: ROOT });
        } catch {}
      }
    }
  });
});

State restoration rules:

• State files (.agent-almanac/state.json) must be cleaned after tests
• Symlinks removed by scatter/uninstall must be restored
• Manifest files (agent-almanac.yml) created by init must be removed
• Order: after() hooks run in reverse declaration order — declare restore hooks last

Expected: The test suite leaves the project in the same state it found it.

On failure: If CI reports leftover files after test runs, add the cleanup to after(). Use git status after test runs to detect leaked state.

Validation

• [ ] Test file runs with node --test cli/test/cli.test.js
• [ ] All tests pass (0 failures)
• [ ] Smoke tests cover --version, --help, and registry loading
• [ ] Lifecycle tests verify create → verify → delete with cleanup
• [ ] At least one adapter dry-run test exists per implemented adapter
• [ ] Error cases test non-zero exit codes with message matching
• [ ] JSON output tests parse actual output (not mocked)
• [ ] After hooks restore all state modified by tests

Common Pitfalls

• Hardcoded counts that break: Registry totals change as content is added. Use \d+ regex or read the count dynamically instead of asserting 329 skills.
• Tests that depend on execution order: node:test runs suites in declaration order by default, but tests within a suite may not. Use lifecycle suites (create → verify → delete) within a single describe to guarantee order.
• Missing cleanup on test failure: If a test fails mid-lifecycle, after() still runs. But if you throw in before(), subsequent tests and after() may not run. Keep before() minimal.
• Interactive prompts hanging tests: Commands with confirmation prompts will hang execSync. Either pipe echo y | or ensure --yes is always passed in tests.
• Testing with real installs in CI: Tests that create files in .claude/skills/ or .agents/skills/ modify the working tree. CI may fail on "dirty working directory" checks. Always clean up.

Related Skills

• scaffold-cli-command — build the commands that these tests verify
• build-cli-plugin — build the adapters tested in Step 4
• design-cli-output — output patterns that tests assert against