dpla/dpla-oai-harvest-watch
docs/ai-context/archive/cursor-skills/dpla-oai-harvest-watch/SKILL.md
Watch an OAI harvest log and report set-by-set progress + ETA (for hubs using harvest.setlist). Use when user asks to watch OAI harvest progress, track sets, estimate completion, or monitor a long OAI harvest.
$ install --global
skillsauthnpx skillsauth add dpla/ingestion3 dpla-oai-harvest-watchInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: May 26, 2026, 3:37 AM131.9s1 file scanned
SKILL.md
- name:
- dpla-oai-harvest-watch
- description:
- Watch an OAI harvest log and report set-by-set progress + ETA (for hubs using harvest.setlist). Use when user asks to watch OAI harvest progress, track sets, estimate completion, or monitor a long OAI harvest.
dpla-oai-harvest-watch
Uses scripts/status/watch-oai-harvest.py to parse ListRecords&set=... transitions in a harvest log file.
Workflow
- Run the harvest with output captured to a log file.
- Run the watcher against that log.
Commands
# Export vars from .env so child processes (harvest + watcher) see them
set -a
source .env
set +a
HUB=<hub>
LOG="logs/harvest-${HUB}-$(date +%Y%m%d_%H%M%S).log"
CONF="${I3_CONF:-$HOME/dpla/code/ingestion3-conf/i3.conf}"
# Start harvest with log capture
./scripts/harvest.sh "$HUB" 2>&1 | tee "$LOG"
# In another terminal: watch set progress (total auto-parsed from i3.conf when possible)
./venv/bin/python scripts/status/watch-oai-harvest.py --log="$LOG" --conf="$CONF" --hub="$HUB"
# If the hub does not have harvest.setlist in i3.conf, you can provide a manual total:
# ./venv/bin/python scripts/status/watch-oai-harvest.py --log="$LOG" --total=<n>
Notes
- This works best for OAI hubs that paginate by
set=(i.e., aharvest.setlistin i3.conf). - If the harvest output does not contain
ListRecords&set=..., use orchestrator status instead:./scripts/status/ingest-status.sh --watch.
Related Skills
dpla/dpla-hub-info
data-ai
VerifiedTrustedCommunity
Show key i3.conf config for a hub (provider, harvest.type, harvest.endpoint, schedule, email, setlist). Use when user asks for hub config, harvest type/endpoint, who gets emails, schedule months, or OAI setlist details.
35SKILL.mdUpdated Apr 16, 2026
dpla/dpla-community-webs-ingest
development
VerifiedTrustedCommunity
Run Community Webs ingest. Use when the user says harvest community-webs, run community-webs ingest, export community webs, or process community webs DB.
35SKILL.mdUpdated Apr 15, 2026
dpla/dpla-verify-and-notify
testing
VerifiedTrustedCommunity
Verify ingest outcomes and send failure or status notifications to Slack or [email protected]. Use when the user asks to verify the ingest, check if it succeeded, notify about a failure, or post to tech-alerts.
33SKILL.mdUpdated May 26, 2026
dpla/dpla-staged-report
business
VerifiedTrustedCommunity
Report which hubs have new JSONL staged in S3 for a given month, and optionally post the report to Slack. Use when user asks what hubs are staged/ready for indexing, /ingest staged, or what changed this month in S3.
33SKILL.mdUpdated May 26, 2026