dpla/dpla-ingest-debug
docs/ai-context/archive/cursor-skills/dpla-ingest-debug/SKILL.md
Debug and fix DPLA hub ingestion failures (harvest/mapping/enrichment/jsonl/s3-sync/anomaly). Use when user asks why a hub failed, to debug an ingest failure, check an escalation report, or retry a failed hub/stage.
$ install --global
skillsauthnpx skillsauth add dpla/ingestion3 dpla-ingest-debugInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: May 26, 2026, 3:36 AM117.0s1 file scanned
SKILL.md
- name:
- dpla-ingest-debug
- description:
- Debug and fix DPLA hub ingestion failures (harvest/mapping/enrichment/jsonl/s3-sync/anomaly). Use when user asks why a hub failed, to debug an ingest failure, check an escalation report, or retry a failed hub/stage.
DPLA Ingest Debugging
Goal
Quickly identify what stage failed, find the relevant logs/escalation report, apply a targeted fix, and re-run only the necessary steps.
Environment
For any commands that depend on project env (especially the orchestrator), run source .env first so JAVA_HOME, DPLA_DATA, I3_CONF, and SLACK_WEBHOOK are available.
Gather Context (fast path)
- If this was an orchestrator run, start with the escalation report (if present):
ls -lt data/escalations/ | head
# then open the relevant failures-<run_id>.md
- Check current per-hub status (orchestrator writes these continuously):
./scripts/status/ingest-status.sh --watch
# or for one hub:
./scripts/status/ingest-status.sh <hub>
- Find recent logs:
ls -lt logs/ | head
Classify the Failure Stage
Look for one of: harvest, mapping, enrichment, jsonl, sync, anomaly.
If you have only a hub name and “it failed”, the quickest approach is:
- check
logs/status/<hub>.status(JSON), and - check the newest matching log in
logs/.
Re-run the Minimal Steps
All commands below assume you’re running from repo root.
Harvest failed:
./scripts/harvest.sh <hub>
Mapping/enrichment/jsonl failed (re-run remap):
./scripts/remap.sh <hub>
S3 sync failed (or you want to re-sync after a successful run):
./scripts/s3-sync.sh <hub>
Orchestrator retry (failed hubs from last run):
./venv/bin/python -m scheduler.orchestrator.main --retry-failed
Orchestrator retry (one hub):
./venv/bin/python -m scheduler.orchestrator.main --hub=<hub>
Common Issues / Fix Heuristics
- Timeout / unreachable feed: retry harvest; if repeated, confirm the endpoint in i3.conf and capture the exact error line for escalation.
- Missing input (mapping complains about harvest input): harvest didn’t produce output → re-run harvest, then remap.
- sbt contention / orphan processes: prefer using the fat JAR path via scripts (they rebuild as needed). If you must intervene, inspect before killing:
pgrep -fl 'sbt|java.*ingestion3'- then
kill <pid>(avoid broadpkillpatterns unless you’re sure).
- Smithsonian / NARA special workflows: do not improvise; follow the dedicated runbooks (Smithsonian preprocessing, NARA delta workflow).
Verify Output
After re-running, verify _SUCCESS markers and counts:
# Examples
ls "$DPLA_DATA/<hub>/harvest"/*/_SUCCESS
ls "$DPLA_DATA/<hub>/mapping"/*/_SUCCESS
ls "$DPLA_DATA/<hub>/jsonl"/*/_SUCCESS
cat "$DPLA_DATA/<hub>/mapping"/*/_SUMMARY | head
Related Skills
dpla/dpla-hub-info
data-ai
VerifiedTrustedCommunity
Show key i3.conf config for a hub (provider, harvest.type, harvest.endpoint, schedule, email, setlist). Use when user asks for hub config, harvest type/endpoint, who gets emails, schedule months, or OAI setlist details.
35SKILL.mdUpdated Apr 16, 2026
dpla/dpla-community-webs-ingest
development
VerifiedTrustedCommunity
Run Community Webs ingest. Use when the user says harvest community-webs, run community-webs ingest, export community webs, or process community webs DB.
35SKILL.mdUpdated Apr 15, 2026
dpla/dpla-verify-and-notify
testing
VerifiedTrustedCommunity
Verify ingest outcomes and send failure or status notifications to Slack or [email protected]. Use when the user asks to verify the ingest, check if it succeeded, notify about a failure, or post to tech-alerts.
33SKILL.mdUpdated May 26, 2026
dpla/dpla-staged-report
business
VerifiedTrustedCommunity
Report which hubs have new JSONL staged in S3 for a given month, and optionally post the report to Slack. Use when user asks what hubs are staged/ready for indexing, /ingest staged, or what changed this month in S3.
33SKILL.mdUpdated May 26, 2026