monkilabs/data-engineering
src/orchestrator/skills/data-engineering/SKILL.md
Transforms, validates, loads data in ETL pipelines. Use when building scrapers, validating NDJSON feeds, or importing data into CMS/DB targets.
$ install --global
skillsauthnpx skillsauth add monkilabs/opencastle data-engineeringInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: May 19, 2026, 7:26 AM133.7s2 files scanned
SKILL.md
- name:
- data-engineering
- description:
- Transforms, validates, loads data in ETL pipelines. Use when building scrapers, validating NDJSON feeds, or importing data into CMS/DB targets.
Data Engineering
Generic pipeline patterns. For project-specific sources, full schema references see REFERENCE.md.
Scraper Architecture
Launch a headless browser cluster (Puppeteer Cluster / Playwright) with retryLimit: 3, retryDelay: 5000, timeout: 30000, args: ['--no-sandbox', '--disable-setuid-sandbox'].
NDJSON Output
One record per line. Schema:
| Field | Type | Notes |
|-------|------|-------|
| name | Required | Preserve original encoding |
| lat/lng | Required | GPS coordinates |
| address | Required | Full text |
| source | Required | e.g. google-maps |
| sourceId | Required | Source-unique ID |
| category | Required | Domain category |
| rating, reviewCount, phone, website, openingHours, photos, priceLevel | Optional | — |
Recommended Workflow (numbered, with validation)
- Scrape: run scraper in
--dry-runto collect sample (50–200 records).- Checkpoint: sample contains expected fields, geo data.
- Recovery: fix extractor selectors; re-run sample.
- Validate NDJSON: run line-by-line JSON parse + schema validator (see
validate-ndjson.jsexample).- Checkpoint: 0 parse errors, required fields present.
- Recovery: run
ndjson-filterto isolate failing records; inspect source HTML.
- Dry-run import: import into staging with
createOrReplacedisabled; check counts, duplicates.- Checkpoint: counts match expectation ±5%; no duplicates inserted.
- Recovery: revert staging; adjust dedupe key.
- Backup: snapshot current target (DB export); store with timestamp.
- Import: run import with idempotent keys; monitor logs; on failure revert to backup.
Quick executable pipeline (copy & adapt)
node ./scripts/scrape-to-ndjson.js --out=data.ndjson --pages=100
node ./scripts/validate-ndjson.js data.ndjson
node ./scripts/dry-import.js data.ndjson --target=staging
node ./scripts/import.js data.ndjson --target=production
Inline: minimal NDJSON validator
const fs = require('fs'), rl = require('readline'), { z } = require('zod');
const schema = z.object({ name: z.string(), source: z.string(), sourceId: z.string() });
const iface = rl.createInterface({ input: fs.createReadStream(process.argv[2]) });
let line = 0, errors = 0;
for await (const l of iface) { line++; try { schema.parse(JSON.parse(l)); } catch(e) { console.error(`Line ${line}:`, e.message); errors++; } }
if (errors) { console.error(`${errors} errors`); process.exit(2); }
console.log('OK');
Full scraper, extended validator: see REFERENCE.md.
Related Skills
monkilabs/validation-gates
development
VerifiedTrustedCommunity
Defines 10 sequential validation gates: secret scanning, lint/test/build checks, blast radius analysis, dependency auditing, browser testing, cache management, regression checks, smoke tests. Use when running pre-deploy validation or CI checks, CI/CD pipelines, deployment pipeline validation, pre-merge checks, continuous integration, or pull request validation.
44SKILL.mdUpdated Apr 21, 2026
monkilabs/testing-workflow
development
VerifiedTrustedCommunity
Generates test plans, writes unit/integration/E2E test files, identifies coverage gaps, flags common testing anti-patterns. Use when writing tests, creating test suites, planning test strategies, mocking dependencies, measuring code coverage, or test planning.
44SKILL.mdUpdated Apr 21, 2026
monkilabs/team-lead-reference
development
VerifiedTrustedCommunity
Provides model routing rules, validates delegation prerequisites, supplies cost tracking templates, defines dead-letter queue formats for Team Lead orchestration. Load when assigning tasks to agents, choosing model tiers, starting delegation session, running multi-agent workflow, delegating work, choosing which model to use, or assigning tasks.
44SKILL.mdUpdated Apr 21, 2026
monkilabs/session-checkpoints
testing
VerifiedTrustedCommunity
Saves, restores session state including task progress, file changes, delegation history. Use when saving progress, resuming interrupted work, picking up where you left off, or checkpointing current work.
44SKILL.mdUpdated Apr 21, 2026