activer007/brightdata
skills_categorized/arts-crafts/brightdata/SKILL.md
Progressive four-tier URL content scraping with automatic fallback strategy. USE WHEN user says "scrape this URL", "fetch this page", "get content from", "can't access this site", "use Bright Data", "pull content from URL", or needs to retrieve web content that may have bot detection or access restrictions.
$ install --global
skillsauthnpx skillsauth add activer007/ordinary-claude-skills brightdataInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: Mar 29, 2026, 10:09 PM37.3s2 files scanned
SKILL.md
- name:
- brightdata
- description:
- |
Workflow Routing (SYSTEM PROMPT)
CRITICAL: This workflow implements progressive escalation for URL content retrieval.
When user requests scraping/fetching URL content: Examples: "scrape this URL", "fetch this page", "get content from [URL]", "pull content from this site", "retrieve [URL]", "can't access this site", "this site is blocking me", "use Bright Data to fetch" → READ: ${PAI_DIR}/skills/brightdata/workflows/four-tier-scrape.md → EXECUTE: Four-tier progressive scraping workflow (WebFetch → Curl → Browser Automation → Bright Data MCP)
When to Activate This Skill
Direct Scraping Requests (Categories 1-4)
- "scrape this URL", "scrape [URL]", "scrape this page"
- "fetch this URL", "fetch [URL]", "fetch this page", "fetch content from"
- "pull content from [URL]", "pull this page", "pull from this site"
- "get content from [URL]", "retrieve [URL]", "retrieve this page"
- "do scraping on [URL]", "run scraper on [URL]"
- "basic scrape", "quick scrape", "simple fetch"
- "comprehensive scrape", "deep scrape", "full content extraction"
Access & Bot Detection Issues (Categories 5-7)
- "can't access this site", "site is blocking me", "getting blocked"
- "bot detection", "CAPTCHA", "access denied", "403 error"
- "need to bypass bot detection", "get around blocking"
- "this URL won't load", "can't fetch this page"
- "use Bright Data", "use the scraper", "use advanced scraping"
Result-Oriented Requests (Category 8)
- "get me the content from [URL]"
- "extract text from [URL]"
- "download this page content"
- "convert [URL] to markdown"
- "need the HTML from this site"
Use Case Indicators
- User needs web content for research or analysis
- Standard methods (WebFetch) are failing
- Site has bot detection or rate limiting
- Need reliable content extraction
- Converting web pages to structured format (markdown)
Core Capabilities
Progressive Escalation Strategy:
- Tier 1: WebFetch - Fast, simple, built-in Claude Code tool
- Tier 2: Customized Curl - Chrome-like browser headers to bypass basic bot detection
- Tier 3: Browser Automation - Full browser automation using Playwright for JavaScript-heavy sites
- Tier 4: Bright Data MCP - Professional scraping service that handles CAPTCHA and advanced bot detection
Key Features:
- Automatic fallback between tiers
- Preserves content in markdown format
- Handles bot detection and CAPTCHA
- Works with any URL
- Efficient resource usage (only escalates when needed)
Workflow Overview
four-tier-scrape.md - Complete URL content scraping with four-tier fallback strategy
- When to use: Any URL content retrieval request
- Process: Start with WebFetch → If fails, use curl with Chrome headers → If fails, use Browser Automation → If fails, use Bright Data MCP
- Output: URL content in markdown format
Extended Context
Integration Points:
- WebFetch Tool - Built-in Claude Code tool for basic URL fetching
- Bash Tool - For executing curl commands with custom headers
- Browser Automation - Playwright-based browser automation for JavaScript rendering
- Bright Data MCP -
mcp__Brightdata__scrape_as_markdownfor advanced scraping
When Each Tier Is Used:
- Tier 1 (WebFetch): Simple sites, public content, no bot detection
- Tier 2 (Curl): Sites with basic user-agent checking, simple bot detection
- Tier 3 (Browser Automation): Sites requiring JavaScript execution, dynamic content loading
- Tier 4 (Bright Data): Sites with CAPTCHA, advanced bot detection, residential proxy requirements
Configuration: No configuration required - all tools are available by default in Claude Code
Examples
Example 1: Simple Public Website
User: "Scrape https://example.com"
Skill Response:
- Routes to three-tier-scrape.md
- Attempts Tier 1 (WebFetch)
- Success → Returns content in markdown
- Total time: <5 seconds
Example 2: Site with JavaScript Requirements
User: "Can't access this site https://dynamic-site.com"
Skill Response:
- Routes to four-tier-scrape.md
- Attempts Tier 1 (WebFetch) → Fails (blocked)
- Attempts Tier 2 (Curl with Chrome headers) → Fails (JavaScript required)
- Attempts Tier 3 (Browser Automation) → Success
- Returns content in markdown
- Total time: ~15-20 seconds
Example 3: Site with Advanced Bot Detection
User: "Scrape https://protected-site.com"
Skill Response:
- Routes to four-tier-scrape.md
- Attempts Tier 1 (WebFetch) → Fails (blocked)
- Attempts Tier 2 (Curl) → Fails (advanced detection)
- Attempts Tier 3 (Browser Automation) → Fails (CAPTCHA)
- Attempts Tier 4 (Bright Data MCP) → Success
- Returns content in markdown
- Total time: ~30-40 seconds
Example 4: Explicit Bright Data Request
User: "Use Bright Data to fetch https://difficult-site.com"
Skill Response:
- Routes to four-tier-scrape.md
- User explicitly requested Bright Data
- Goes directly to Tier 4 (Bright Data MCP) → Success
- Returns content in markdown
- Total time: ~5-10 seconds
Related Documentation:
${PAI_DIR}/skills/CORE/SKILL-STRUCTURE-AND-ROUTING.md- Canonical structure guide${PAI_DIR}/skills/CORE/CONSTITUTION.md- Overall Kai philosophy
Last Updated: 2025-11-23
Related Skills
activer007/ts-agent-sdk
tools
VerifiedTrustedCommunity
Generate typed TypeScript SDKs for AI agents to interact with MCP servers. Converts verbose JSON-RPC curl commands to clean function calls (docs.createDocument() vs curl). Auto-detects MCP tools from server modules, generates TypeScript types and client methods, creates runnable example scripts. Use when: building MCP-enabled applications, need typed programmatic access to MCP tools, want Claude Code to manage apps via scripts, eliminating manual JSON-RPC curl commands, validating MCP inputs/outputs, or creating reusable agent automation.
10SKILL.mdUpdated Apr 16, 2026
activer007/task-generator
testing
VerifiedTrustedCommunity
Generate structured task lists from specs or requirements. IMPORTANT: After completing ANY spec via ExitSpecMode, ALWAYS ask the user: "Would you like me to generate a task list for this spec?" Use when user confirms or explicitly requests task generation from a plan/spec/PRD.
10SKILL.mdUpdated Apr 16, 2026
activer007/story-explanation
tools
VerifiedTrustedCommunity
Create compelling story-format summaries using UltraThink to find the best narrative framing. Support multiple formats - 3-part narrative, n-length with inline links, abridged 5-line, or comprehensive via Foundry MCP. USE WHEN user says 'create story explanation', 'narrative summary', 'explain as a story', or wants content in Daniel's conversational first-person voice.
10SKILL.mdUpdated Apr 16, 2026
activer007/Siberian Shamanic Navigation
testing
VerifiedTrustedCommunity
Navigate through the original three-world shamanic technology. Deploy when soul retrieval, power animal guidance, or journey between realms emerges. Deeply respectful of Tungus, Buryat, Yakut, Evenki traditions. Use for consciousness navigation, NOT cultural appropriation.
10SKILL.mdUpdated Apr 16, 2026