microsoft/run-e2e-tests

Run E2E (end-to-end) tests to verify complete user workflows work correctly.

When to Use This Skill

• Before submitting a PR with significant changes
• After modifying environment discovery, creation, or selection logic
• Before a release to validate full workflows
• When user reports a workflow is broken

Note: Run smoke tests first. If smoke tests fail, E2E tests will also fail.

Quick Reference

| Action | Command | | ----------------- | -------------------------------------------------------------- | | Run all E2E tests | npm run compile && npm run compile-tests && npm run e2e-test | | Run specific test | npm run e2e-test -- --grep "discovers" | | Debug in VS Code | Debug panel → "E2E Tests" → F5 |

How E2E Tests Work

Unlike unit tests (mocked) and smoke tests (quick checks), E2E tests:

1. Launch a real VS Code instance with the extension
2. Exercise complete user workflows via the real API
3. Verify end-to-end behavior (discovery → selection → execution)

They take longer (1-3 minutes) but catch integration issues.

Workflow

Step 1: Compile and Run

npm run compile && npm run compile-tests && npm run e2e-test

Step 2: Interpret Results

Pass:

  E2E: Environment Discovery
    ✓ Can trigger environment refresh
    ✓ Discovers at least one environment
    ✓ Environments have required properties
    ✓ Can get global environments

  4 passing (45s)

Fail: Check error message and see Debugging section.

Debugging Failures

| Error | Cause | Fix | | ---------------------------- | ---------------------- | ------------------------------------------- | | No environments discovered | Python not installed | Install Python, verify it's on PATH | | Extension not found | Build failed | Run npm run compile | | API not available | Activation error | Debug with F5, check Debug Console | | Timeout exceeded | Slow operation or hang | Increase timeout or check for blocking code |

For detailed debugging: Debug panel → "E2E Tests" → F5

Prerequisites

E2E tests have system requirements:

• Python installed - At least one Python interpreter must be discoverable
• Extension builds - Run npm run compile before tests
• CI needs webpack build - Run npm run compile (webpack) before tests, not just npm run compile-tests (tsc)

Adding New E2E Tests

Create files in src/test/e2e/ with pattern *.e2e.test.ts:

import * as assert from 'assert';
import * as vscode from 'vscode';
import { waitForCondition } from '../testUtils';
import { ENVS_EXTENSION_ID } from '../constants';

suite('E2E: [Workflow Name]', function () {
    this.timeout(120_000); // 2 minutes

    let api: ExtensionApi;

    suiteSetup(async function () {
        const extension = vscode.extensions.getExtension(ENVS_EXTENSION_ID);
        assert.ok(extension, 'Extension not found');
        if (!extension.isActive) await extension.activate();
        api = extension.exports;
    });

    test('[Test description]', async function () {
        // Use real API (flat structure, not nested!)
        // api.getEnvironments(), not api.environments.getEnvironments()
        await waitForCondition(
            async () => (await api.getEnvironments('all')).length > 0,
            60_000,
            'No environments found',
        );
    });
});

Test Files

| File | Purpose | | ----------------------------------------------- | ------------------------------------ | | src/test/e2e/environmentDiscovery.e2e.test.ts | Discovery workflow tests | | src/test/e2e/index.ts | Test runner entry point | | src/test/testUtils.ts | Utilities (waitForCondition, etc.) |

Notes

• E2E tests are slower than smoke tests (expect 1-3 minutes)
• They may create/modify files - cleanup happens in suiteTeardown
• First run downloads VS Code (~100MB, cached in .vscode-test/)
• For more details on E2E tests and how they compare to other test types, refer to the project's testing documentation.