etylsarin/data-engineering
src/orchestrator/skills/data-engineering/SKILL.md
Transforms, validates, and loads data in ETL pipelines. Use when building scrapers, validating NDJSON feeds, or importing data into CMS/DB targets.
$ install --global
skillsauthnpx skillsauth add etylsarin/opencastle data-engineeringInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: May 17, 2026, 3:08 AM324.4s2 files scanned
SKILL.md
- name:
- data-engineering
- description:
- Transforms, validates, and loads data in ETL pipelines. Use when building scrapers, validating NDJSON feeds, or importing data into CMS/DB targets.
Data Engineering
Generic pipeline patterns. For project-specific sources and full schema references see REFERENCE.md.
Scraper Architecture
Launch a headless browser cluster (Puppeteer Cluster / Playwright) with retryLimit: 3, retryDelay: 5000, timeout: 30000, args: ['--no-sandbox', '--disable-setuid-sandbox'].
NDJSON Output
One record per line. Schema:
| Field | Type | Notes |
|-------|------|-------|
| name | Required | Preserve original encoding |
| lat/lng | Required | GPS coordinates |
| address | Required | Full text |
| source | Required | e.g. google-maps |
| sourceId | Required | Source-unique ID |
| category | Required | Domain category |
| rating, reviewCount, phone, website, openingHours, photos, priceLevel | Optional | — |
Recommended Workflow (numbered, with validation)
- Scrape: run scraper in
--dry-runto collect a sample (50–200 records).- Checkpoint: sample contains expected fields and geo data.
- Recovery: fix extractor selectors, re-run sample.
- Validate NDJSON: run line-by-line JSON parse + schema validator (see
validate-ndjson.jsexample).- Checkpoint: 0 parse errors, required fields present.
- Recovery: run
ndjson-filterto isolate failing records and inspect source HTML.
- Dry-run import: import into staging with
createOrReplacedisabled; check counts and duplicates.- Checkpoint: counts match expectation ±5% and no duplicates inserted.
- Recovery: revert staging and adjust dedupe key.
- Backup: snapshot current target (DB export) and store with timestamp.
- Import: run import with idempotent keys and monitor logs; on failure revert to backup.
Quick executable pipeline (copy & adapt)
node ./scripts/scrape-to-ndjson.js --out=data.ndjson --pages=100
node ./scripts/validate-ndjson.js data.ndjson
node ./scripts/dry-import.js data.ndjson --target=staging
node ./scripts/import.js data.ndjson --target=production
Inline: minimal NDJSON validator
const fs = require('fs'), rl = require('readline'), { z } = require('zod');
const schema = z.object({ name: z.string(), source: z.string(), sourceId: z.string() });
const iface = rl.createInterface({ input: fs.createReadStream(process.argv[2]) });
let line = 0, errors = 0;
for await (const l of iface) { line++; try { schema.parse(JSON.parse(l)); } catch(e) { console.error(`Line ${line}:`, e.message); errors++; } }
if (errors) { console.error(`${errors} errors`); process.exit(2); }
console.log('OK');
Full scraper and extended validator: see REFERENCE.md.
Related Skills
etylsarin/validation-gates
development
VerifiedTrustedCommunity
Defines 10 sequential validation gates: secret scanning, lint/test/build checks, blast radius analysis, dependency auditing, browser testing, cache management, regression checks, and smoke tests. Use when running pre-deploy validation or CI checks, CI/CD pipelines, deployment pipeline validation, pre-merge checks, continuous integration, or pull request validation.
44SKILL.mdUpdated May 17, 2026
etylsarin/testing-workflow
development
VerifiedTrustedCommunity
Generates test plans, writes unit/integration/E2E test files, identifies coverage gaps, and flags common testing anti-patterns. Use when writing tests, creating test suites, planning test strategies, mocking dependencies, measuring code coverage, or test planning.
44SKILL.mdUpdated May 17, 2026
etylsarin/team-lead-reference
development
VerifiedTrustedCommunity
Provides model routing rules, validates delegation prerequisites, supplies cost tracking templates, and defines dead-letter queue formats for Team Lead orchestration. Load when assigning tasks to agents, choosing model tiers, starting a delegation session, running a multi-agent workflow, delegating work, choosing which model to use, or assigning tasks.
44SKILL.mdUpdated May 17, 2026
etylsarin/session-checkpoints
testing
VerifiedTrustedCommunity
Saves and restores session state including task progress, file changes, and delegation history. Use when saving progress, resuming interrupted work, picking up where you left off, or checkpointing current work.
44SKILL.mdUpdated May 17, 2026