aiskillstore/storage-debug-instrumentation

Storage Debug Instrumentation

Purpose

Enable rapid diagnosis of storage state, synchronization health, and backend performance bottlenecks by exposing:

• Raw article inspection from both PostgreSQL and ChromaDB
• Storage drift detection (missing/dangling entries)
• Detailed startup timeline breakdown (DB init, cache preload, vector store, RSS refresh)
• One-page debug dashboard consolidating all diagnostics

Scope

• Backend: app/services/startup_metrics.py, app/main.py, app/vector_store.py, app/database.py, app/api/routes/debug.py
• Frontend: frontend/lib/api.ts, frontend/app/debug/page.tsx
• No schema changes; purely additive instrumentation and debug routes

Workflow

1. Create startup metrics service

File: backend/app/services/startup_metrics.py

• Implement thread-safe StartupMetrics class to record phase timings
• Expose record_event(name, started_at, detail, metadata) for phase capture
• Support add_note(key, value) for arbitrary annotations
• Export singleton startup_metrics for app-wide use

2. Instrument vector store initialization

File: backend/app/vector_store.py

• Import startup_metrics
• In VectorStore.__init__(), wrap initialization with time.time() timer
• Record event with metadata: host, port, collection, documents
• Catch connection errors and annotate them

3. Instrument FastAPI startup sequence

File: backend/app/main.py

• Call startup_metrics.mark_app_started() at beginning of on_startup()
• Wrap each phase (DB init, schedulers, cache preload, RSS refresh, migration) with record_event()
• Include metadata: cache_size, article_count, oldest_article_hours
• Call startup_metrics.mark_app_completed() at end
• Add app version notes via add_note()

4. Add database pagination helpers

File: backend/app/database.py

• Implement fetch_articles_page() to support:
  • Limit/offset pagination
  • Optional source filter
  • Missing-embeddings-only flag
  • Published date range filters
  • Sort direction (asc/desc)
  • Return oldest/newest timestamp bounds
• Implement fetch_article_chroma_mappings() to return all article→chroma ID mappings for drift analysis

5. Add vector store pagination helpers

File: backend/app/vector_store.py

• Implement list_articles(limit, offset) to return paginated Chroma documents with metadata and previews
• Implement list_all_ids() to return all stored Chroma IDs for drift detection (used by /debug/storage/drift)

6. Expose debug API endpoints

File: backend/app/api/routes/debug.py

• Add GET /debug/startup → returns startup metrics timeline (events + notes)
• Add GET /debug/chromadb/articles → returns paginated raw Chroma entries with limit/offset
• Add GET /debug/database/articles → returns paginated Postgres rows with filters (source, embeddings, date range, sort)
• Add GET /debug/storage/drift → compares Chroma IDs vs Postgres mappings, returns missing/dangling counts + samples

7. Add frontend API bindings

File: frontend/lib/api.ts

• Export types: StartupEventMetric, StartupMetricsResponse, ChromaDebugResponse, DatabaseDebugResponse, StorageDriftReport
• Export fetchers: fetchStartupMetrics(), fetchChromaDebugArticles(), fetchDatabaseDebugArticles(), fetchStorageDrift()
• Ensure snake_case→camelCase mapping for response fields

8. Build debug dashboard page

File: frontend/app/debug/page.tsx

• Create /debug route with multi-tab inspection UI
• Render startup timeline: phase name, duration, metadata badges (cache size, vectors, migrated records)
• Display Chroma browser: paginated table with ID, title, source, preview
• Display Postgres browser: paginated table with filters (source, date range, missing-embeddings-only flag)
• Display drift report: sample tables for missing-in-chroma and dangling-in-chroma entries
• Include summary cards for quick metrics (boot time, total articles, vector count, drift count)

Implementation checklist

• [ ] Create backend/app/services/startup_metrics.py
• [ ] Instrument backend/app/vector_store.py::VectorStore.__init__()
• [ ] Instrument backend/app/main.py::on_startup() (all phases)
• [ ] Add fetch_articles_page() and fetch_article_chroma_mappings() to backend/app/database.py
• [ ] Add list_articles() and list_all_ids() to backend/app/vector_store.py
• [ ] Add /debug/startup, /debug/chromadb/articles, /debug/database/articles, /debug/storage/drift to backend/app/api/routes/debug.py
• [ ] Add types and fetchers to frontend/lib/api.ts
• [ ] Create frontend/app/debug/page.tsx with dashboard layout
• [ ] Run uvx ruff check backend → all checks pass
• [ ] Test endpoints in curl or Postman to verify response structure

Verification checklist

• [ ] GET http://localhost:8000/debug/startup returns valid timeline with events and notes
• [ ] GET http://localhost:8000/debug/chromadb/articles?limit=50&offset=0 returns paginated Chroma docs
• [ ] GET http://localhost:8000/debug/database/articles?source=bbc&missing_embeddings_only=false filters correctly
• [ ] GET http://localhost:8000/debug/storage/drift compares counts and returns drift samples
• [ ] http://localhost:3000/debug loads without errors and displays all four sections
• [ ] Refresh button triggers all four API calls in parallel
• [ ] Pagination controls update limit/offset correctly
• [ ] Database filters (source, date range) update and refresh data
• [ ] Startup timeline shows non-zero phase durations if backend just started

Future enhancements

• Streaming startup metrics via SSE (live tail during boot)
• Export startup report as JSON/CSV for performance tracking over time
• Automated drift alerts (post to Slack/email if dangling > threshold)
• Performance graphs (startup time trends, article throughput)
• Sync-on-demand action (button to force vector store refresh for missing articles)