mattwynne/exploratory-testing

Exploratory Testing

Overview

Exploratory testing discovers what automated tests miss. The agent acts as a curious, methodical user - running yx commands in a sandbox, observing output, trying edge cases, and logging findings. Each session is guided by a charter that focuses exploration on a specific area with specific heuristics.

When to Use

• After implementing a new feature or fixing a bug
• Before a release, to shake out edge cases
• When a command's behaviour feels under-tested
• When you want to stress-test error handling or output formatting
• Periodically, to discover regressions or UX papercuts

Don't use when:

• You need deterministic, repeatable test coverage (write Cucumber scenarios)
• The area has no working implementation yet

Phase 1: Charter

Agree on a Target

The user provides (or the agent suggests) a target area to explore. Good targets are specific commands, workflows, or quality attributes:

• "the yx add command"
• "yak hierarchy and the --under flag"
• "output formatting across --format options"
• "error messages when things go wrong"
• "a full workflow: add, organise, work, complete, prune"

Select Heuristics

Pick 2-4 heuristics from the menu below that suit the target:

| Heuristic | CLI Application | |-----------|----------------| | CRUD | Add, list, show, update, remove yaks through full lifecycle | | Zero, One, Many | Empty state, single yak, many yaks, deep nesting | | Boundary Values | Long names, special chars, spaces, empty strings, unicode | | Never and Always | Invariants (done yaks always show done, removed yaks never listed) | | Follow the Data | Add -> list -> modify -> list -> verify consistency | | Some, None, All | Filters with matching/non-matching/all items | | Starve | Missing .yaks directory, non-existent directories, no permissions | | Interrupt | Broken pipes (yx ls \| head -1), partial stdin, Ctrl-C | | Configuration Tour | --format options, --only filters, env vars | | Claims Tour | Does --help text match actual behaviour? | | Sequence Variation | Unusual command orders (done before add, prune with no done) | | User Tour | Common real-world workflows end-to-end |

Write the Charter

Format the charter using Elisabeth Hendrickson's template:

Explore [target area] With [selected heuristics] To Discover [risks or information we seek]

Example:

Explore the yx add command With Boundary Values, Zero/One/Many, Claims Tour To Discover how it handles edge-case input and whether --help accurately describes its behaviour

Approval Gate

Present the charter to the user. Do not proceed until the user confirms the charter. The user may adjust the target, heuristics, or risk focus.

Phase 2: Exploration

Set Up the Sandbox

Create an isolated environment so exploration never touches the project's real .yaks/ data.

Follow the yx-sandbox skill to set up and use a temp directory. Use the sandbox prefix on every yx command during exploration.

Seed the Environment

Before exploring, create enough data to work with:

# Using the literal sandbox path from mktemp output:
cd /tmp/tmp.xYz123AbC
YX_SKIP_GIT_CHECKS=1 yx add make the tea
YX_SKIP_GIT_CHECKS=1 yx add buy biscuits
YX_SKIP_GIT_CHECKS=1 yx add wash the cups --under make the tea
YX_SKIP_GIT_CHECKS=1 yx state wash the cups wip
YX_SKIP_GIT_CHECKS=1 yx done buy biscuits

Adapt seeding to the charter - if exploring hierarchy, create deeper nesting; if exploring empty state, skip seeding entirely at first.

Explore Systematically

Work through each chartered heuristic. For each one:

1. State what you're testing (the heuristic and specific probe)
2. Run the command with the env var prefix
3. Record the result in your session log

Keep a running session log in this format:

### [Heuristic Name]

**Probe:** [what you're trying]
**Command:** `cd <sandbox-path> && YX_SKIP_GIT_CHECKS=1 yx ...`
**Expected:** [what you thought would happen]
**Actual:** [what actually happened]
**Verdict:** OK | BUG | UX-ISSUE | INCONSISTENCY | UNEXPECTED | QUESTION
**Notes:** [any additional observations]

Exploration Guidelines

• Stay within the charter scope. If you discover something interesting outside scope, note it for a future session.
• Vary your inputs. Don't just test happy paths.
• Pay attention to output formatting. Alignment, colours, whitespace, and truncation are all worth probing.
• Check exit codes. echo $? after commands that should fail - do they return non-zero?
• Try composition. Pipe output to other commands, use --format plain for scripting.
• Time-box yourself. 15-30 minutes of exploration per charter is usually enough.

Phase 3: Report

Present Findings

After exploration, present a structured report to the user.

Summary

One-line summary: how many probes, how many findings, overall impression.

Findings by Category

Group findings into these categories (skip empty ones):

| Category | Description | |----------|-------------| | Bugs | Incorrect behaviour, crashes, wrong exit codes | | UX Issues | Confusing output, unclear errors, surprising defaults | | Inconsistencies | Behaviour differs between similar commands | | Missing Error Handling | No error where one is expected, unhelpful messages | | Unexpected Behaviour | Works but not how a user would expect | | Worked Well | Things that behaved exactly right, good UX moments |

For each finding, include:

• The command that triggered it
• What happened vs what was expected
• Severity estimate (low / medium / high)

Suggested Follow-Ups

Concrete next steps:

• Yaks to create for bugs or improvements
• Cucumber scenarios to write for discovered edge cases
• Documentation to update if --help is misleading
• Areas worth a dedicated ET session

Clean Up

Remove the sandbox as described in the yx-sandbox skill.

Quick Reference

| Phase | What Happens | Gate | |-------|-------------|------| | Charter | Agree target, select heuristics, write charter | User approves charter | | Exploration | Sandbox setup, systematic probing, session log | None - explore autonomously | | Report | Categorised findings, follow-ups, cleanup | User reviews findings |

Common Mistakes

| Mistake | Fix | |---------|-----| | Exploring without a charter | Always agree on target and heuristics first | | Only testing happy paths | Heuristics exist to push beyond the obvious | | Logging only failures | Record successes too - they confirm expected behaviour | | Exploring everything at once | Pick 2-4 heuristics per session, stay focused |

| Skipping exit code checks | echo $? after commands that should fail | | Not seeding enough data | Adapt seed data to what the charter needs |

Sources

• Elisabeth Hendrickson: Explore It! (Pragmatic Bookshelf)
• James Bach: Session-Based Test Management
• Michael Bolton: "Testing vs. Checking"