dpla/dpla-community-webs-ingest
docs/ai-context/archive/cursor-skills/dpla-community-webs-ingest/SKILL.md
Run Community Webs ingest from SQLite DB. Use when the user says harvest community-webs, run community-webs ingest, export community webs, or process community webs DB.
$ install --global
skillsauthnpx skillsauth add dpla/ingestion3 dpla-community-webs-ingestInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: May 23, 2026, 2:47 AM36.9s1 file scanned
SKILL.md
- name:
- dpla-community-webs-ingest
- description:
- Run Community Webs ingest from SQLite DB. Use when the user says harvest community-webs, run community-webs ingest, export community webs, or process community webs DB.
DPLA Community Webs Ingest
Purpose
Run the Community Webs ingest workflow: export SQLite DB → JSONL → ZIP → harvest → (optional) full pipeline. Internet Archive sends a SQLite database; the scripts handle the intermediate export step automatically.
When to Use
- "Harvest community-webs"
- "Run community-webs ingest"
- "Export community webs DB"
- "Process community webs"
- "Ingest community-webs from database"
Environment: Source .env before running so JAVA_HOME, DPLA_DATA, I3_CONF are set. Scripts that source common.sh load $I3_HOME/.env when present. Full checklist: AGENTS.md § Environment and build.
Workflow
- Place the DB in
$DPLA_DATA/community-webs/originalRecords/(or provide path via--db=). - Run export + harvest (or full pipeline):
- Export + harvest only:
./scripts/harvest/community-webs-ingest.sh - Full pipeline (harvest + map + enrich + jsonl):
./scripts/harvest/community-webs-ingest.sh --full - Update i3.conf with output path:
./scripts/harvest/community-webs-ingest.sh --full --update-conf
- Export + harvest only:
- Verify outputs:
_SUCCESSin harvest/mapping/enrichment/jsonl dirs.
Scripts
| Script | Purpose |
|--------|---------|
| community-webs-export.sh | Export DB → JSONL → ZIP; validates schema before zipping |
| community-webs-ingest.sh | Orchestrates export + harvest (+ optional full pipeline) |
Options
community-webs-export.sh:
--db=PATH— Explicit DB path (default: auto-detect latest*.dbin originalRecords)--update-conf— Update i3.confcommunity-webs.harvest.endpoint--skip-validate— Skip JSONL schema validation (not recommended)
community-webs-ingest.sh:
--db=PATH— Pass to export script--skip-export— Use existing ZIP (endpoint must already point to it)--full— Run harvest + mapping + enrichment + jsonl--update-conf— Update i3.conf with export output directory
Output
- Export:
$DPLA_DATA/community-webs/originalRecords/<YYYYMMDD>/community-webs-<timestamp>.zip - Harvest:
$DPLA_DATA/community-webs/harvest/ - Full pipeline: mapping, enrichment, jsonl under
$DPLA_DATA/community-webs/
Validation
The export script runs community-webs-validate-jsonl.py on the JSONL before zipping. It checks:
- Each record has required field
id - Each line is valid JSON
- Records with
"status":"deleted"are skipped (harvester filters these)
Tests: ./venv/bin/python -m pytest scripts/tests/test_community_webs_validation.py -v
Critical Rules
- DB location: Place
*.dbin$DPLA_DATA/community-webs/originalRecords/or use--db=/path/to/file.db - Prerequisites:
sqlite3,jq(brew install / apt install) - Output path: i3.conf
community-webs.harvest.endpointmust point to directory containing the ZIP
Key References
| Resource | Path | |----------|------| | Script reference | scripts/SCRIPTS.md | | Ingest docs | README_INGESTS.md | | Agent guide | AGENTS.md |
Related Skills
dpla/dpla-hub-info
data-ai
VerifiedTrustedCommunity
Show key i3.conf config for a hub (provider, harvest.type, harvest.endpoint, schedule, email, setlist). Use when user asks for hub config, harvest type/endpoint, who gets emails, schedule months, or OAI setlist details.
35SKILL.mdUpdated Apr 16, 2026
dpla/dpla-community-webs-ingest
development
VerifiedTrustedCommunity
Run Community Webs ingest. Use when the user says harvest community-webs, run community-webs ingest, export community webs, or process community webs DB.
35SKILL.mdUpdated Apr 15, 2026
dpla/dpla-verify-and-notify
testing
VerifiedTrustedCommunity
Verify ingest outcomes and send failure or status notifications to Slack or [email protected]. Use when the user asks to verify the ingest, check if it succeeded, notify about a failure, or post to tech-alerts.
33SKILL.mdUpdated May 26, 2026
dpla/dpla-staged-report
business
VerifiedTrustedCommunity
Report which hubs have new JSONL staged in S3 for a given month, and optionally post the report to Slack. Use when user asks what hubs are staged/ready for indexing, /ingest staged, or what changed this month in S3.
33SKILL.mdUpdated May 26, 2026