neuron-core/neuron-test-engineer

Neuron AI Test Engineer

This skill helps you write comprehensive tests for Neuron AI applications using the built-in testing utilities in NeuronAI\Testing.

Testing Philosophy

Neuron AI provides fake implementations that:

• Never make real API calls - All AI provider calls are mocked
• Record all interactions - Inspect what was sent and when
• Provide fluent assertions - PHPUnit-style assertions for verification

Core Testing Utilities

1. FakeAIProvider

The primary tool for testing agents without real AI API calls.

use NeuronAI\Testing\FakeAIProvider;
use NeuronAI\Chat\Messages\AssistantMessage;
use NeuronAI\Chat\Messages\UserMessage;

// Create with predetermined responses
$provider = new FakeAIProvider(
    new AssistantMessage('Hello! How can I help you?'),
    new AssistantMessage('The answer is 42.')
);

// Or use static constructor
$provider = FakeAIProvider::make(
    new AssistantMessage('Response 1'),
    new AssistantMessage('Response 2')
);

Key Features:

• Responses are returned sequentially from queue
• Supports chat(), stream(), and structured() methods
• Records all requests for assertion

2. FakeVectorStore

For testing RAG systems without real vector databases.

use NeuronAI\Testing\FakeVectorStore;
use NeuronAI\RAG\Document;

// Pre-populate with search results
$vectorStore = new FakeVectorStore([
    new Document('France is a country in Europe. Its capital is Paris.'),
    new Document('Germany is a country in Europe. Its capital is Berlin.'),
]);

// Or create empty and set results later
$vectorStore = FakeVectorStore::make();
$vectorStore->setSearchResults([
    new Document('Relevant document content')
]);

3. FakeEmbeddingsProvider

For testing embeddings without real API calls.

use NeuronAI\Testing\FakeEmbeddingsProvider;

// Create with default dimensions (8)
$embeddings = new FakeEmbeddingsProvider();

// Or specify dimensions
$embeddings = new FakeEmbeddingsProvider(dimensions: 1536);

// Use static constructor
$embeddings = FakeEmbeddingsProvider::make();

4. FakeMcpTransport

For testing MCP (Model Context Protocol) integrations without a real MCP server.

use NeuronAI\Testing\FakeMcpTransport;

// Queue predetermined responses
$transport = new FakeMcpTransport(
    ['result' => ['tools' => [['name' => 'search', 'description' => 'Search the web']]]],
    ['result' => ['content' => [['type' => 'text', 'text' => 'Search results...']]]],
);

// Or add responses later
$transport->addResponses(['result' => ['content' => 'More data']]);

Key Features:

• Responses returned sequentially from queue via receive()
• Records all sent/received data for assertion
• Fluent MCP-specific assertions (assertInitialized, assertToolCalled, etc.)

5. FakeMiddleware

For testing workflow middleware behavior.

use NeuronAI\Testing\FakeMiddleware;

$middleware = FakeMiddleware::make();

// Configure custom handlers
$middleware->setBeforeHandler(function ($node, $event, $state): void {
    $state->set('injected_data', 'value');
});

// Configure exceptions for testing error handling
$middleware->setThrowOnBefore(new \Exception('Test exception'));

Test Patterns by Component

Testing Agent Chat

use PHPUnit\Framework\TestCase;
use NeuronAI\Agent\Agent;
use NeuronAI\Chat\Messages\AssistantMessage;
use NeuronAI\Chat\Messages\UserMessage;
use NeuronAI\Testing\FakeAIProvider;

class MyAgentTest extends TestCase
{
    public function test_agent_returns_expected_response(): void
    {
        $provider = new FakeAIProvider(
            new AssistantMessage('Expected response')
        );

        $agent = Agent::make();
        $agent->setAiProvider($provider);

        $message = $agent->chat(new UserMessage('Hello'))->getMessage();

        $this->assertSame('Expected response', $message->getContent());
        $provider->assertCallCount(1);
    }

    public function test_agent_uses_system_prompt(): void
    {
        $provider = new FakeAIProvider(new AssistantMessage('OK'));

        $agent = Agent::make();
        $agent->setAiProvider($provider);
        $agent->setInstructions('Always respond in French.');

        $agent->chat(new UserMessage('Hello'))->getMessage();

        $provider->assertSystemPrompt('Always respond in French.');
    }
}

Testing Agent with Tools

use NeuronAI\Tools\Tool;
use NeuronAI\Tools\ToolProperty;
use NeuronAI\Tools\PropertyType;
use NeuronAI\Chat\Messages\ToolCallMessage;

public function test_agent_executes_tool_and_returns_result(): void
{
    $searchTool = Tool::make('search', 'Search the web')
        ->addProperty(new ToolProperty('query', PropertyType::STRING, 'Search query', true))
        ->setCallable(fn (string $query): string => "Results for: {$query}");

    // First response: model calls the tool
    // Second response: model uses tool result to answer
    $provider = new FakeAIProvider(
        new ToolCallMessage(null, [
            (clone $searchTool)->setCallId('call_1')->setInputs(['query' => 'PHP frameworks']),
        ]),
        new AssistantMessage('Based on my search, here are the top PHP frameworks...')
    );

    $agent = Agent::make();
    $agent->setAiProvider($provider);
    $agent->addTool($searchTool);

    $message = $agent->chat(new UserMessage('What are the best PHP frameworks?'))->getMessage();

    $this->assertSame('Based on my search, here are the top PHP frameworks...', $message->getContent());
    $provider->assertCallCount(2); // Tool call + final response
    $provider->assertToolsConfigured(['search']);
}

Testing Streaming

use NeuronAI\Chat\Messages\Stream\Chunks\TextChunk;

public function test_agent_streams_response(): void
{
    $provider = new FakeAIProvider(new AssistantMessage('Hello world'));
    $provider->setStreamChunkSize(5); // Control chunk size for predictable tests

    $agent = Agent::make();
    $agent->setAiProvider($provider);

    $handler = $agent->stream(new UserMessage('Hi'));

    $chunks = [];
    foreach ($handler->events() as $event) {
        if ($event instanceof TextChunk) {
            $chunks[] = $event->content;
        }
    }

    $this->assertSame(['Hello', ' worl', 'd'], $chunks);

    $state = $handler->run();
    $this->assertSame('Hello world', $state->getMessage()->getContent());
}

Testing Structured Output

use NeuronAI\Chat\Messages\AssistantMessage;

public function test_agent_extracts_structured_data(): void
{
    $provider = new FakeAIProvider(
        new AssistantMessage('{"name": "Alice", "age": 30}')
    );

    $agent = Agent::make();
    $agent->setAiProvider($provider);

    class Person
    {
        #[SchemaProperty(description: 'The person name', required: true)]
        public string $name;

        #[SchemaProperty(description: 'The person age')]
        public int $age;
    }

    $person = $agent->structured(
        new UserMessage('My name is Alice and I am 30 years old'),
        Person::class
    );

    $this->assertInstanceOf(Person::class, $person);
    $this->assertSame('Alice', $person->name);
    $this->assertSame(30, $person->age);

    $provider->assertMethodCallCount('structured', 1);
}

Testing RAG Systems

use NeuronAI\RAG\RAG;
use NeuronAI\RAG\Document;
use NeuronAI\Testing\FakeAIProvider;
use NeuronAI\Testing\FakeEmbeddingsProvider;
use NeuronAI\Testing\FakeVectorStore;

class MyRAGTest extends TestCase
{
    public function test_rag_retrieves_and_answers(): void
    {
        $provider = new FakeAIProvider(
            new AssistantMessage('Paris is the capital of France.')
        );

        $vectorStore = new FakeVectorStore([
            new Document('France is a country in Europe. Its capital is Paris.'),
        ]);

        $rag = RAG::make();
        $rag->setAiProvider($provider);
        $rag->setEmbeddingsProvider(new FakeEmbeddingsProvider());
        $rag->setVectorStore($vectorStore);

        $message = $rag->chat(new UserMessage('What is the capital of France?'))->getMessage();

        $this->assertSame('Paris is the capital of France.', $message->getContent());
        $provider->assertCallCount(1);
        $vectorStore->assertSearchCount(1);
    }

    public function test_rag_adds_documents(): void
    {
        $embeddings = new FakeEmbeddingsProvider();
        $vectorStore = new FakeVectorStore();

        $rag = RAG::make();
        $rag->setAiProvider(new FakeAIProvider());
        $rag->setEmbeddingsProvider($embeddings);
        $rag->setVectorStore($vectorStore);

        $rag->addDocuments([
            new Document('First document'),
            new Document('Second document'),
        ]);

        $embeddings->assertCallCount(2);
        $vectorStore->assertDocumentCount(2);
        $vectorStore->assertHasDocumentWithContent('First document');
        $vectorStore->assertHasDocumentWithContent('Second document');
    }
}

Testing Workflows

use NeuronAI\Workflow\Workflow;
use NeuronAI\Workflow\WorkflowState;
use NeuronAI\Workflow\Events\StartEvent;
use NeuronAI\Workflow\Events\StopEvent;
use NeuronAI\Workflow\Node;

class MyWorkflowTest extends TestCase
{
    public function test_workflow_executes_nodes_in_sequence(): void
    {
        $workflow = Workflow::make()
            ->addNodes([
                new FirstNode(),
                new SecondNode(),
                new ThirdNode(),
            ]);

        $finalState = $workflow->init()->run();

        $this->assertTrue($finalState->get('first_executed'));
        $this->assertTrue($finalState->get('second_executed'));
        $this->assertTrue($finalState->get('third_executed'));
    }

    public function test_workflow_with_initial_state(): void
    {
        $workflow = Workflow::make(
            state: new WorkflowState(['input' => 'test_value'])
        )->addNodes([
            new ProcessNode(),
        ]);

        $finalState = $workflow->init()->run();

        $this->assertEquals('test_value', $finalState->get('original_input'));
    }
}

Testing Middleware

use NeuronAI\Testing\FakeMiddleware;

class MyMiddlewareTest extends TestCase
{
    public function test_middleware_runs_on_all_nodes(): void
    {
        $middleware = FakeMiddleware::make();

        Workflow::make()
            ->addGlobalMiddleware($middleware)
            ->addNodes([new NodeOne(), new NodeTwo(), new NodeThree()])
            ->init()
            ->run();

        // 3 nodes = 3 before + 3 after calls
        $middleware->assertBeforeCalledTimes(3);
        $middleware->assertAfterCalledTimes(3);
        $middleware->assertCallCount(6);
    }

    public function test_middleware_only_runs_for_specific_node(): void
    {
        $middleware = FakeMiddleware::make();

        Workflow::make()
            ->addMiddleware(NodeTwo::class, $middleware)
            ->addNodes([new NodeOne(), new NodeTwo(), new NodeThree()])
            ->init()
            ->run();

        $middleware->assertBeforeCalledTimes(1);
        $middleware->assertBeforeCalledForNode(NodeTwo::class);
    }

    public function test_middleware_can_modify_state(): void
    {
        $middleware = FakeMiddleware::make()
            ->setBeforeHandler(function ($node, $event, $state): void {
                $state->set('injected_by_middleware', true);
            });

        $finalState = Workflow::make()
            ->addMiddleware(NodeOne::class, $middleware)
            ->addNodes([new NodeOne(), new NodeTwo()])
            ->init()
            ->run();

        $this->assertTrue($finalState->get('injected_by_middleware'));
    }
}

Testing Workflow Interruption

use NeuronAI\Workflow\Interrupt\WorkflowInterrupt;
use NeuronAI\Workflow\Persistence\InMemoryPersistence;

class MyInterruptTest extends TestCase
{
    public function test_workflow_interrupts_and_resumes(): void
    {
        $workflow = Workflow::make(
            persistence: new InMemoryPersistence(),
            resumeToken: 'test-workflow'
        )->addNodes([
            new NodeOne(),
            new InterruptableNode(),
            new NodeThree(),
        ]);

        // First run should interrupt
        $interrupt = null;
        try {
            $workflow->init()->run();
            $this->fail('Expected WorkflowInterrupt exception');
        } catch (WorkflowInterrupt $e) {
            $interrupt = $e;
        }

        $this->assertNotNull($interrupt);
        $this->assertEquals('human input needed', $interrupt->getRequest()->getMessage());

        // Resume with human feedback
        $finalState = $workflow->init($interrupt->getRequest())->run();

        $this->assertTrue($finalState->get('interruptable_node_executed'));
    }
}

Testing MCP Integrations

Use FakeMcpTransport to test code that interacts with MCP servers without running a real server.

use NeuronAI\Testing\FakeMcpTransport;

class McpIntegrationTest extends TestCase
{
    public function test_mcp_initialization_handshake(): void
    {
        $transport = new FakeMcpTransport(
            ['result' => ['capabilities' => [], 'serverInfo' => ['name' => 'test-server']]],
            ['result' => []],
        );

        $transport->connect();

        // Simulate initialization handshake
        $transport->send(['method' => 'initialize', 'params' => ['capabilities' => []]]);
        $transport->receive(); // consume capabilities response

        $transport->send(['method' => 'notifications/initialized']);
        $transport->receive(); // consume ack

        $transport->assertInitialized();
        $transport->assertConnected();
    }

    public function test_mcp_tool_call(): void
    {
        $transport = new FakeMcpTransport(
            ['result' => ['tools' => [['name' => 'search', 'description' => 'Search']]]],
            ['result' => ['content' => [['type' => 'text', 'text' => 'Found 3 results']]]],
        );

        $transport->connect();

        $transport->send(['method' => 'tools/list', 'params' => []]);
        $transport->receive();

        $transport->send(['method' => 'tools/call', 'params' => ['name' => 'search', 'arguments' => ['query' => 'test']]]);
        $transport->receive();

        $transport->assertToolsListCalled();
        $transport->assertToolCalled('search');
        $transport->assertSendCount(2);
        $transport->assertReceiveCount(2);
    }
}

Assertion Reference

FakeAIProvider Assertions

// Verify number of calls
$provider->assertCallCount(3);

// Verify specific method was called
$provider->assertMethodCallCount('chat', 2);
$provider->assertMethodCallCount('stream', 1);
$provider->assertMethodCallCount('structured', 1);

// Verify system prompt
$provider->assertSystemPrompt('You are a helpful assistant.');

// Verify tools were configured
$provider->assertToolsConfigured(['search', 'calculator']);

// Verify no calls were made
$provider->assertNothingSent();

// Custom assertion with callback
$provider->assertSent(function (RequestRecord $record): bool {
    return $record->method === 'chat'
        && str_contains($record->messages[0]->getContent(), 'keyword');
});

FakeVectorStore Assertions

// Verify search count
$vectorStore->assertSearchCount(2);

// Verify document count
$vectorStore->assertDocumentCount(5);

// Verify specific document exists
$vectorStore->assertHasDocumentWithContent('Expected content');

// Verify store is empty
$vectorStore->assertNothingStored();

FakeEmbeddingsProvider Assertions

// Verify embedding call count
$embeddings->assertCallCount(3);

// Verify specific text was embedded
$embeddings->assertEmbeddedText('Expected text to embed');

// Verify no embeddings were made
$embeddings->assertNothingEmbedded();

FakeMiddleware Assertions

// Verify before() was called
$middleware->assertBeforeCalled();
$middleware->assertBeforeNotCalled();
$middleware->assertBeforeCalledTimes(3);
$middleware->assertBeforeCalledForNode(NodeOne::class);

// Verify after() was called
$middleware->assertAfterCalled();
$middleware->assertAfterNotCalled();
$middleware->assertAfterCalledTimes(3);
$middleware->assertAfterCalledForNode(NodeOne::class);

// Verify total call count
$middleware->assertCallCount(6);
$middleware->assertNotCalled();

FakeMcpTransport Assertions

// Verify connection state
$transport->assertConnected();
$transport->assertDisconnected();

// Verify send/receive counts
$transport->assertSendCount(3);
$transport->assertReceiveCount(3);
$transport->assertNothingSent();
$transport->assertNothingReceived();

// Verify specific MCP methods
$transport->assertMethodSent('initialize', 1);
$transport->assertMethodReceived('initialize', 1);

// Convenience assertions for common MCP patterns
$transport->assertInitialized();          // initialize + notifications/initialized
$transport->assertToolsListCalled(1);     // tools/list sent N times
$transport->assertToolCalled('search', 2); // tools/call with specific tool name

// Custom assertion with callback
$transport->assertSent(function (array $data): bool {
    return ($data['method'] ?? null) === 'tools/call'
        && ($data['params']['name'] ?? null) === 'search';
});

Testing Multiple Turns

public function test_conversation_remembers_context(): void
{
    $provider = new FakeAIProvider(
        new AssistantMessage('Hi! I can help with that.'),
        new AssistantMessage('The capital of France is Paris.'),
    );

    $agent = Agent::make();
    $agent->setAiProvider($provider);

    $first = $agent->chat(new UserMessage('Hello'))->getMessage();
    $second = $agent->chat(new UserMessage('What is the capital of France?'))->getMessage();

    $this->assertSame('Hi! I can help with that.', $first->getContent());
    $this->assertSame('The capital of France is Paris.', $second->getContent());
    $provider->assertCallCount(2);
}

Inspecting Recorded Calls

RequestRecord Properties

foreach ($provider->getRecorded() as $record) {
    $record->method;          // 'chat', 'stream', or 'structured'
    $record->messages;        // Message[] passed to provider
    $record->systemPrompt;    // ?string system prompt
    $record->tools;           // ToolInterface[] configured tools
    $record->structuredClass; // ?string output class (structured only)
    $record->structuredSchema;// array schema (structured only)
}

MiddlewareRecord Properties

foreach ($middleware->getRecorded() as $record) {
    $record->method;  // 'before' or 'after'
    $record->node;    // NodeInterface being executed
    $record->event;   // Event passed/returned
    $record->state;   // WorkflowState at call time
}

Running Tests

# Run all tests
composer test

# Run specific test file
vendor/bin/phpunit tests/Agent/AgentTest.php

# Run specific test method
vendor/bin/phpunit --filter test_chat_with_tools

# Run with verbose output
vendor/bin/phpunit --colors=always -v

Best Practices

1. Use descriptive test names - Test names should describe the behavior being verified
2. One assertion per concept - Group related assertions but keep tests focused
3. Test edge cases - Empty results, errors, null values
4. Test streaming consumption - Always consume generators in tests
5. Verify call counts - Ensure the expected number of API calls are made
6. Use custom assertions - assertSent() with callbacks for complex verification
7. Test middleware order - Verify execution order when order matters
8. Test state changes - Verify workflow state after execution

Common Pitfalls

Generator Not Consumed

// WRONG: Generator not consumed, code never runs
$generator = $provider->stream(new UserMessage('Hi'));
// Body of generator hasn't executed yet!

// CORRECT: Consume the generator
$generator = $provider->stream(new UserMessage('Hi'));
foreach ($generator as $chunk) {
    // Process chunks
}
$finalMessage = $generator->getReturn();

Empty Response Queue

// WRONG: No responses queued
$provider = new FakeAIProvider();
$provider->chat(new UserMessage('Hi')); // Throws ProviderException!

// CORRECT: Queue responses before calling
$provider = new FakeAIProvider(new AssistantMessage('Response'));
$provider->chat(new UserMessage('Hi'));

Hidden Tools Not Sent to Provider

// Hidden tools are executable but not sent to AI
$hiddenTool = Tool::make('secret', 'Secret tool')
    ->setCallable(fn ($input) => "Result: {$input}")
    ->visible(false);

// This will NOT include 'secret' in tools configured
$provider->assertToolsConfigured(['search']); // Only visible tools