aeondave/spiderfoot

SpiderFoot

Automated OSINT platform — 200+ modules, correlates all data types around a target.

Quick Start

git clone https://github.com/smicallef/spiderfoot
cd spiderfoot
pip install -r requirements.txt

# CLI scan on domain
python3 sf.py -s target.com -t INTERNET_NAME -o json -q

# Web UI (interactive)
python3 sf.py -l 127.0.0.1:5001
# → http://127.0.0.1:5001

Target Types (-t)

| Type | Description | Example | |------|-------------|---------| | INTERNET_NAME | Hostname / domain | target.com | | IP_ADDRESS | IPv4 address | 1.2.3.4 | | EMAILADDR | Email address | [email protected] | | USERNAME | Social username | johndoe | | PHONE_NUMBER | Phone number | +14151234567 | | HUMAN_NAME | Person's name | "John Doe" | | NETBLOCK_OWNER | ASN or CIDR | AS12345 | | BGP_AS_OWNER | ASN number | 12345 | | URL_FORM | URL to scan | https://target.com/login |

CLI Usage

# Domain scan — all passive modules
python3 sf.py -s target.com -t INTERNET_NAME -o json -q > results.json

# Email scan
python3 sf.py -s [email protected] -t EMAILADDR -o json -q

# Specific modules only
python3 sf.py -s target.com -t INTERNET_NAME -m sfp_dnsresolve,sfp_sublist3r,sfp_crtsh -o json -q

# List all modules
python3 sf.py -M

# List modules by type
python3 sf.py -M | grep -i "passive"

Key CLI Flags

| Flag | Purpose | |------|---------| | -s <target> | Target value | | -t <type> | Target type | | -m <modules> | Specific modules (comma-separated) | | -o json/csv/tab | Output format | | -q | Quiet (no progress) | | -l <host:port> | Start web UI | | -M | List all modules | | --timeout <sec> | Global timeout |

Module Categories

# List all categories
python3 sf.py -M | awk '{print $1}' | sort -u

# Useful module groups
sfp_dnsresolve      # DNS resolution
sfp_crtsh           # Certificate transparency
sfp_sublist3r       # Subdomain enumeration
sfp_shodan          # Shodan integration (API key)
sfp_virustotal      # VirusTotal (API key)
sfp_hunter          # Hunter.io emails (API key)
sfp_emailharvest    # Email harvesting
sfp_haveibeenpwned  # Breach check (API key)
sfp_linkedin        # LinkedIn profiles
sfp_twitter         # Twitter profiles
sfp_github          # GitHub user/org data
sfp_pastebin        # Pastebin leaks
sfp_darkweb         # Dark web mentions (Tor)
sfp_threatintel     # Threat intelligence feeds
sfp_whois           # WHOIS data

Web UI Workflow

python3 sf.py -l 127.0.0.1:5001

1. Browse to http://127.0.0.1:5001
2. New Scan → enter target + type
3. Select scan profile: Passive, Investigate, Footprint, All
4. Run → watch live graph of discovered entities
5. Browse results by data type: emails, IPs, subdomains, leaked credentials, social profiles
6. Export: JSON/CSV for downstream use

Scan Profiles

| Profile | Modules | Use | |---------|---------|-----| | Passive | ~50 | No direct target contact | | Investigate | ~100 | Mixed passive + active | | Footprint | ~150 | Full external footprint | | All | 200+ | Everything (loud) |

API Key Configuration

# spiderfoot.db stores settings after first run
# Or configure via web UI: Settings → API Keys

High-value API keys:

Hunter.io       → email harvest
Shodan          → port/service data
VirusTotal      → malware/domain intel
HaveIBeenPwned  → breach lookup
SecurityTrails  → DNS history
IntelX          → dark web / leaks

Parse JSON Output

import json

with open("results.json") as f:
    results = json.load(f)

# Group by data type
from collections import defaultdict
by_type = defaultdict(list)
for item in results:
    by_type[item["type"]].append(item["data"])

# Print emails found
for email in set(by_type.get("EMAILADDR", [])):
    print(email)

# Print subdomains
for sub in set(by_type.get("INTERNET_NAME", [])):
    print(sub)

Resources

| File | When to load | |------|--------------| | references/modules.md | Full module list with descriptions, API requirements, and recommended scan configurations |