aeondave/hakrawler

Hakrawler

Fast Go web crawler — discover URLs, JS files, forms, and endpoints.

Quick Start

go install github.com/hakluke/hakrawler@latest

# Crawl a domain
echo https://target.com | hakrawler

# Depth 3, include subdomains
echo https://target.com | hakrawler -d 3 -subs

# Output as JSON
echo https://target.com | hakrawler -json

# Pipe multiple domains
cat domains.txt | hakrawler -d 2

Core Flags

| Flag | Purpose | |------|---------| | -d N | Depth (default: 1) | | -subs | Include subdomains | | -u | Unique URLs only | | -insecure | Skip TLS verification | | -t N | Threads | | -timeout N | Timeout per request (s) | | -H "K:V" | Custom header | | -json | JSON output | | -scope REGEX | Limit to URL pattern | | -plain | Print plain text (no color) | | -proxy <url> | HTTP/SOCKS5 proxy | | -cookie <str> | Cookie string | | -dr | Disable following redirects | | -w N | Wait N ms between requests |

Common Workflows

Build URL inventory for fuzzing:

echo https://target.com | hakrawler -d 3 -u | tee urls.txt
# Feed to ffuf
ffuf -w urls.txt:URL -u URL -mc 200

Discover JS files:

echo https://target.com | hakrawler -d 2 | grep "\.js$"

Combine with httpx for live check:

cat domains.txt | hakrawler | httpx -silent -mc 200

Scope-limited crawl (stay in scope):

echo https://target.com | hakrawler -d 3 -scope ".*\.target\.com.*" -u

Extract API endpoints:

echo https://target.com | hakrawler -d 3 -u | \
  grep -E "(/api/|/v[0-9]+/|\.json|\.xml)" | sort -u

Form action discovery:

echo https://target.com | hakrawler -json | \
  jq -r 'select(.type=="form") | .source'

Multi-target via live hosts:

subfinder -d target.com -silent | \
  httpx -silent | \
  hakrawler -d 2 -u | sort -u > all_urls.txt

Resources

| File | When to load | |------|--------------| | references/crawl-tips.md | Scope filtering, JS analysis, pipeline patterns, JS secret extraction |