0xDarkMatter/net-ops

Network Operations

Diagnose network problems on Windows, macOS, or Linux with a layered ladder that isolates faults to the smallest possible scope, then pattern-match against OS-specific culprits. Designed for the common case: someone reports "internet broken" on a box you can shell into (locally or via SSH).

The Universal Insight

Bypass-tool succeeds while OS-resolver fails is a smoking gun on every platform. It means DNS infrastructure is healthy but the operating system's name-resolution path is hooked or misconfigured. The bypass tool differs per OS but the discriminator is identical:

| OS | Bypass tool | OS resolver tool | If bypass works but resolver fails | |---|---|---|---| | Windows | nslookup | Resolve-DnsName, browsers | NRPT, WFP, HOSTS, LSP, local 127.0.0.1:53 proxy | | macOS | dig @1.1.1.1 | dscacheutil -q host, browsers | /etc/resolver/*, scutil DNS, profiles, mDNSResponder, kext | | Linux | dig @1.1.1.1 | getent hosts, resolvectl query | systemd-resolved, /etc/resolv.conf, NetworkManager, dnsmasq, NSS |

The bypass tool implements its own resolver and talks straight to UDP/53. The OS resolver tool goes through the full system name-service path including all hooks. Comparing the two narrows the suspect list dramatically.

The Diagnostic Ladder

Walk down the layers in order. Do not skip rungs. Each rung has a binary outcome that eliminates everything above it. Per-OS tools are in references/diagnostic-ladder.md; the structure is universal.

1. Link layer        — interface up, valid IP, gateway present
2. IP reachability   — ping public IPs over ICMP
3. Socket reach.     — TCP/443 + UDP/53 to known destinations (raw socket DNS)
4. DNS infrastructure — bypass tool: nslookup / dig @<server>
5. OS resolver path  — the hook layer (most interesting on modern systems)
6. Application       — real HTTP request to a real hostname

The most common mistake: jumping to rung 6 ("HTTPS doesn't work, must be a cert / proxy") when rung 5 is the actual problem (an orphan VPN DNS rule on Windows, a stale /etc/resolver/ file on macOS, a misconfigured systemd-resolved on Linux). Discipline prevents this.

Workflow

1. Identify the target OS

If local: uname -s (Unix) or check shell environment. If remote over SSH, the bootstrap script auto-detects:

scripts/ssh-bootstrap.sh <user>@<host>

2. Run the OS-appropriate probe

| OS | Script | |---|---| | Windows | scripts/windows/probe.ps1 (via -EncodedCommand over SSH) | | macOS | scripts/macos/probe.sh | | Linux | scripts/linux/probe.sh |

Each prints structured [PASS]/[FAIL] per rung. Scan for the first FAIL — that's where to drill in.

3. Drill into the failing layer

The interesting failures are almost always rung 5. Per-OS deep-dive scripts:

| OS | Script | What it does | |---|---|---| | Windows | scripts/windows/nrpt-audit.ps1 | Dump NRPT rules with attribution + registry forensics | | macOS | scripts/macos/dns-audit.sh | Dump scutil --dns, /etc/resolver/*, mDNSResponder state, profiles | | Linux | scripts/linux/dns-audit.sh | Dump systemd-resolved status, resolv.conf chain, NM config, NSS order |

4. Apply the minimum reversible fix

Repair scripts default to dry-run and protect known-good config (Tailscale MagicDNS, MDM-managed entries). Apply only when the dry-run output matches expectation.

| OS | Repair script | |---|---| | Windows | scripts/windows/nrpt-clean.ps1 (removes orphan NRPT catch-alls, protects Tailscale) | | macOS | scripts/macos/resolver-clean.sh (removes orphan /etc/resolver/* from disconnected VPNs) | | Linux | scripts/linux/resolved-reset.sh (resets systemd-resolved per-link config) |

Quick Reference: Smoking Guns

| Platform | Symptom | Most likely cause | Quick test | |---|---|---|---| | Windows | nslookup works, browsers fail | Orphan NRPT catch-all (VPN residue) | Get-DnsClientNrptRule \| Where Namespace -eq '.' | | Windows | Public DoH resolver IPs blocked on 443, other 443 works | AV "Encrypted DNS Detection" | Get-CimInstance -Ns root/SecurityCenter2 -Class AntiVirusProduct | | macOS | dig works, browsers fail | Stale /etc/resolver/* from disconnected VPN | ls /etc/resolver/ && scutil --dns \| head -40 | | macOS | All DNS fails post-VPN install | Configuration profile with DNS override | profiles list -type configuration | | Linux | dig works, getent hosts fails | systemd-resolved misconfigured | resolvectl status | | Linux | DNS works on some apps, not others | NSS order in /etc/nsswitch.conf excludes resolve | grep ^hosts /etc/nsswitch.conf | | All | DNS suddenly broken after sleep/wake | VPN client failed disconnect cleanup | OS-specific (see above) |

SSH Transport Patterns

Windows targets

PowerShell-over-SSH has notorious escaping issues. Always pass scripts via -EncodedCommand with UTF-16LE base64:

B64=$(printf '%s' "$PS_SCRIPT" | iconv -t UTF-16LE | base64)
ssh <target> "powershell -NoProfile -EncodedCommand $B64"

Unix targets (macOS, Linux)

Heredoc works cleanly; no special encoding needed:

ssh <target> 'bash -s' < scripts/linux/probe.sh
# or, with arguments:
ssh <target> "bash -s -- arg1 arg2" < scripts/linux/probe.sh

For consistency, scripts/ssh-bootstrap.sh handles both transports based on detected OS.

Pattern Recognition

After a few sessions, certain symptom triplets become instantly diagnosable. See references/case-studies.md for worked examples. Hall-of-fame entries:

Windows: nslookup works, Resolve-DnsName times out identically across all servers, Invoke-WebRequest says "remote name could not be resolved" → orphan NRPT catch-all from a disconnected VPN. Common gateway IP patterns are listed in references/common-culprits.md.

macOS: dig <host> works, browsers say "cannot find server," scutil --dns shows extra "resolver #N" entries pointing at private-range gateways with domain : listed → leftover /etc/resolver/<domain> files from a disconnected VPN.

Linux: dig @<public-resolver> <host> works, getent hosts <host> fails → /etc/nsswitch.conf may have an NSS chain that skips resolve, OR /etc/resolv.conf is no longer symlinked to the systemd-resolved stub.

Safety Notes

• Read before write. Always dump current state before modifying a resolver config. The forensics may be load-bearing for explaining what happened.
• Don't disable security tools without consent. AV / firewall hooks are intrusive but legitimate. Pause is preferred over uninstall.
• Tailscale's name-resolution config looks like junk but is essential. Always filter on protected nameserver patterns (100.100.100.100 on all OSes) before bulk-deleting.
• Resolver config persists across reboots. Removing a rule is forever (until the VPN re-creates it). Confirm the source/comment before deletion.
• macOS profile DNS overrides may be MDM-managed. Removing them may violate enterprise policy and may be re-applied automatically. Coordinate with IT.

References

• references/diagnostic-ladder.md — full ladder methodology with per-OS commands per rung
• references/common-culprits.md — detection + fix catalog for Windows / macOS / Linux
• references/case-studies.md — worked examples and template for adding new ones

Scripts

• scripts/ssh-bootstrap.sh — establish SSH session, auto-detect target OS, emit usable invocation
• scripts/windows/probe.ps1 — full layered diagnostic for Windows
• scripts/windows/nrpt-audit.ps1 — NRPT forensics with attribution
• scripts/windows/nrpt-clean.ps1 — safe NRPT cleanup (protects Tailscale)
• scripts/macos/probe.sh — full layered diagnostic for macOS
• scripts/macos/dns-audit.sh — scutil + /etc/resolver + profile + mDNSResponder dump
• scripts/macos/resolver-clean.sh — remove orphan /etc/resolver/* files
• scripts/linux/probe.sh — full layered diagnostic for Linux
• scripts/linux/dns-audit.sh — systemd-resolved + NM + NSS + resolv.conf dump
• scripts/linux/resolved-reset.sh — reset systemd-resolved per-link state