cnoe-io/streaming-testing

Streaming Testing

Compare A2A streaming behaviour across supervisor versions (e.g. 0.3.0 vs 0.2.41). Captures SSE events, analyzes metadata correctness, and generates side-by-side markdown reports.

Prerequisites

• Docker Compose dev stack running (docker-compose.dev.yaml)
• 0.3.0 supervisor on port 8000 (caipe-supervisor)
• 0.2.41 supervisor on port 8041 (caipe-supervisor-041)
• Python 3.10+ (scripts use stdlib only — no pip install needed)

Quick Start

# One command — captures from both supervisors, analyzes, compares
./skills/streaming-testing/run_comparison.sh "what can you do?"

# Output lands in /tmp/streaming-comparison-<timestamp>.md

Individual Scripts

All scripts live in scripts/ and use Python stdlib only.

| Script | Purpose | Usage | |--------|---------|-------| | trace_a2a_streaming.py | SSE timeline with summary stats | python3 scripts/trace_a2a_streaming.py 8000 "query" | | capture_a2a_events.py | Save raw A2A events to JSON | python3 scripts/capture_a2a_events.py http://localhost:8000/ "query" /tmp/out.json | | analyze_a2a_metadata.py | Validate metadata flags in a capture | python3 scripts/analyze_a2a_metadata.py /tmp/out.json | | compare_a2a_events.py | Side-by-side comparison of two captures | python3 scripts/compare_a2a_events.py /tmp/a.json /tmp/b.json | | analyze_accumulation_flow.py | Content accumulation + duplicate detection | python3 scripts/analyze_accumulation_flow.py "query" --port 8000 | | validate_artifacts.py | Parse/validate artifact reports | python3 scripts/validate_artifacts.py --query "query" --port 8000 |

Workflow

Step 1: Capture events from both supervisors

python3 scripts/capture_a2a_events.py http://localhost:8000/ "what can you do?" /tmp/cap-030.json
python3 scripts/capture_a2a_events.py http://localhost:8041/ "what can you do?" /tmp/cap-041.json

Step 2: Analyze metadata on each capture

python3 scripts/analyze_a2a_metadata.py /tmp/cap-030.json
python3 scripts/analyze_a2a_metadata.py /tmp/cap-041.json

This validates:

• is_final_answer is present on streaming_result artifacts
• No [FINAL ANSWER] marker leaked into text
• No internal metadata leaked (is_task_complete=, Returning structured response)
• is_narration and is_final_answer are mutually exclusive
• final_result or partial_result artifact exists
• Streaming content is non-empty

Step 3: Compare side-by-side

python3 scripts/compare_a2a_events.py /tmp/cap-030.json /tmp/cap-041.json --output /tmp/comparison.md

Generates a markdown report with:

• Summary table (total time, time to first content, artifact counts, char counts)
• Artifact name breakdown
• Safety checks (marker leaks, metadata leaks)
• Per-artifact detail table with metadata flags

Step 4 (optional): Deep-dive traces

For a live timeline view of each SSE event as it arrives:

python3 scripts/trace_a2a_streaming.py 8000 "what can you do?"
python3 scripts/trace_a2a_streaming.py 8041 "what can you do?"

Interpreting Results

Key Metrics

| Metric | What it means | Good value | |--------|--------------|------------| | Time to first content | How long until the user sees text | < 5s | | Total time | End-to-end response time | < 20s (simple), < 120s (RAG) | | has is_final_answer | Supervisor correctly tags the final answer | True | | Narration events | Narration chunks for typing-status display | 0 for simple queries, > 0 for RAG | | Marker leaks | [FINAL ANSWER] leaked to client | Must be 0 | | Metadata leaks | Internal state leaked to client | Must be 0 |

What to look for

• 0.3.0 has a "dead gap" before content: The generate_structured_response node builds JSON preamble before the content key. Time to first content will be higher than 0.2.41.
• 0.2.41 may leak narration text: Pre-marker buffer flush sends LLM thinking text (e.g. "Excellent! I now have comprehensive information...") to the client.
• Narration events should NOT appear in 0.2.41: The is_narration flag was added in 0.3.0.
• Both versions should have 0 marker leaks: The Slack bot and agent_executor both strip markers.

Troubleshooting

Supervisor not reachable

# Check containers
docker ps | grep supervisor

# Restart
docker restart caipe-supervisor
docker restart caipe-supervisor-041

Script hangs

The scripts have a default 120s (trace) or 300s (capture) timeout. For RAG queries, increase the timeout:

# capture_a2a_events.py reads timeout from the URL (not a CLI arg).
# For trace, it's hardcoded at 120s in http.client timeout.
# If queries consistently time out, check supervisor logs:
docker logs caipe-supervisor 2>&1 | tail -20

Empty captures

If capture_a2a_events.py produces 0 artifacts:

1. Verify the supervisor is healthy: curl http://localhost:8000/.well-known/agent.json
2. Check if the supervisor uses SSE or JSON-RPC (the script handles both)
3. Look at supervisor logs for errors: docker logs caipe-supervisor 2>&1 | tail -50

Examples

# Simple query comparison
./skills/streaming-testing/run_comparison.sh "what can you do?"

# RAG query comparison (takes longer)
./skills/streaming-testing/run_comparison.sh "what is caipe?"

# Just analyze a single capture
python3 scripts/analyze_a2a_metadata.py /tmp/cap-030.json --verbose

# Custom labels in comparison report
python3 scripts/compare_a2a_events.py /tmp/cap-030.json /tmp/cap-041.json \
    --label-a "0.3.0 (post-fix)" --label-b "0.2.41 (golden)"