aiskillstore/backend-hang-debug

Backend Hang Debug

Purpose

• Detect and resolve event-loop hangs where the FastAPI app stops responding (e.g., curl http://localhost:8000/ times out) due to synchronous executor shutdown in the SSE news stream.
• Provide a repeatable triage flow using py-spy to capture live stacks and pinpoint blocking code.

Scope

• Backend: backend/app/api/routes/stream.py (news stream), backend/app/services/rss_ingestion.py (RSS workers), startup processes.
• Tooling: py-spy for live stack dumps; curl with timeouts for smoke tests.

Quick Triage

1. Reproduce hang: curl -m 5 http://localhost:8000/ and curl -m 5 http://localhost:8000/health; note timeouts.
2. Process check: ss -tlnp | grep 8000 to confirm listener; ls /proc/$(pgrep -f "uvicorn app.main")/fd | wc -l to rule out FD leak.
3. Stack capture (inside backend venv): uv pip install py-spy then sudo /home/bender/classwork/Thesis/backend/.venv/bin/py-spy dump --pid $(pgrep -f "uvicorn app.main") (and worker pid if multiprocess). Look for ThreadPoolExecutor.shutdown in api/routes/stream.py frames.

Fix Pattern (non-blocking executor)

• Replace synchronous context manager with ThreadPoolExecutor(...): inside event_generator with a long-lived executor plus explicit non-blocking shutdown:
  • Create executor outside the context manager.
  • On client disconnect, cancel pending futures instead of awaiting shutdown.
  • In finally, call executor.shutdown(wait=False, cancel_futures=True).
• Rationale: context manager calls shutdown(wait=True), blocking the event loop if RSS worker threads hang on network I/O.

Implementation Steps

1. Update stream executor usage in backend/app/api/routes/stream.py:
   • Instantiate executor = concurrent.futures.ThreadPoolExecutor(max_workers=5).
   • Dispatch work via loop.run_in_executor(executor, _process_source_with_debug, ...).
   • On disconnect, cancel() pending futures.
   • In finally, executor.shutdown(wait=False, cancel_futures=True).
2. Keep RSS executor as-is (rss_ingestion.py) since it runs in background threads, but ensure request timeouts remain reasonable (currently 60s per RSS requests.get).
3. Retest:
   • Restart uvicorn; curl -m 5 http://localhost:8000/health should respond.
   • Start a stream request and abort the client; server must stay responsive.
   • Re-run py-spy dump to verify no ThreadPoolExecutor.shutdown(wait=True) frames in main thread.

Verification Checklist

• [ ] curl -m 5 http://localhost:8000/ returns a response (no hang).
• [ ] curl -m 5 http://localhost:8000/health succeeds.
• [ ] Aborting /news/stream does not freeze subsequent requests.
• [ ] py-spy dump shows event loop not blocked on ThreadPoolExecutor.shutdown.
• [ ] Frontend no longer stalls waiting on root/health while backend is busy with streams.

Notes & Future Hardening

• Consider adding request timeout middleware to fail fast on slow handlers.
• Add per-source network timeouts and shorter retries for RSS feeds to reduce long-lived threads.
• If multi-worker uvicorn is used, run py-spy on each worker pid when diagnosing hangs.