asets-gobizit/hermes-health

Hermes Health Check Skill

When to Use

• User says "hermes is stuck", "fix hermes", "check hermes", "hermes health", "hermes not working"
• User reports Telegram bot not responding
• User reports slow responses from Hermes
• After any Hermes config change

How to Invoke

python "C:\Users\dansk\.claude\skills\hermes-health\hermes_health.py"
python "C:\Users\dansk\.claude\skills\hermes-health\hermes_health.py" --fix
python "C:\Users\dansk\.claude\skills\hermes-health\hermes_health.py" --fix --restart-gateway

CLI Flags

| Flag | Description | |------|-------------| | --fix | Auto-fix issues found (restart Ollama, unload models, fix config) | | --restart-gateway | Also restart Hermes gateway after fixes | | --verbose | Show detailed diagnostic output |

Expected Routing Logic (FINAL - April 3, 2026)

• Messages up to 1000 characters -> qwen/qwen3.6-plus:free on OpenRouter (FREE - $0/token)
• Messages over 1000 characters -> Claude Sonnet 4.6 on OpenRouter (paid)

Why Qwen 3.6 Plus over local Ollama

• Qwen 3.6 Plus is ~100B+ params vs 7B local -- far smarter and more accurate
• Truly free ($0/token confirmed via OpenRouter API)
• Responds in 1-3 seconds vs 5-15 seconds for local CPU inference
• 1,000,000 token context window
• Handles tool-use protocol properly (local 7-8B models cannot)
• Only downside: requires internet (not an issue for Danny)

Config Location

• Hermes config: /home/dansk/.hermes/config.yaml (inside WSL)
• Hermes secrets: /home/dansk/.hermes/.env (inside WSL)
• Smart routing section in config.yaml:

smart_model_routing:
  enabled: true
  max_simple_chars: 1000
  max_simple_words: 180
  cheap_model:
    provider: openrouter
    model: qwen/qwen3.6-plus:free

Troubleshooting History (April 3, 2026)

What broke and why

Danny tried adding qwen2.5:32b (19GB) to local Ollama for higher quality. This caused:

1. Ollama froze -- 32B model too large for CPU-only inference
2. While debugging, Hermes auto-updated from v0.5.x to v0.6.0 (57 commits)
3. The update changed provider resolution -- provider: openai no longer routes to localhost
4. Switching to provider: custom reached localhost but Hermes v0.6.0 sends full tool definitions with every message
5. llama3.1:8b cannot handle tool-use protocol -- hangs forever on tool schemas

What was tried (in order)

| Attempt | Result | |---------|--------| | provider: openai + localhost | v0.6.0 resolves to real OpenAI API, not localhost | | provider: ollama + localhost | "Unknown provider 'ollama'" | | provider: custom + localhost + llama3.1:8b | Reaches Ollama but 8B model hangs on tool definitions | | provider: openai (original working setup) | Auth flow changed in v0.6.0, no longer works | | provider: custom + localhost + qwen2.5:7b | WORKED -- but slower and weaker than cloud | | Free OpenRouter model (qwen3.6-plus:free) | FINAL DECISION -- $0/token, fast, smart |

Key lessons learned

1. Never update Hermes during a fix -- hermes update changed provider resolution mid-debug
2. llama3.1:8b cannot handle Hermes tool-use -- Hermes v0.6.0 sends 20+ tool schemas with every message. Use qwen2.5:7b which supports tool-use
3. provider: custom is the only working option for local Ollama in v0.6.0
4. Large models (32B+) will freeze on Danny's hardware (no GPU) -- stick to 7-8B for local
5. Unload large models from Ollama if loaded: curl localhost:11434/api/generate -d '{"model":"qwen2.5:32b","keep_alive":0}'

Code fix applied

File: /home/dansk/.hermes/hermes-agent/agent/smart_model_routing.py Added direct api_key reading from cheap_model config (Hermes only checked api_key_env):

    if not explicit_api_key:
        direct_key = str(route.get("api_key") or "").strip()
        if direct_key:
            explicit_api_key = direct_key

NOTE: This fix may be overwritten by hermes update. Re-apply if routing breaks after update.

Self-Patched Source Files (discovered April 3, 2026)

What happens

Hermes can modify its own source code (smart_model_routing.py, gateway/run.py) during conversations when debugging itself. These patches may have errors and break the response pipeline, causing Handler returned empty/None response in the gateway logs.

How to detect

# Check for modified source files (run in WSL)
cd /home/dansk/.hermes/hermes-agent && git status

If agent/smart_model_routing.py or gateway/run.py show as modified, Hermes has patched itself.

How to fix

# 1. Restore broken files to clean upstream state
cd /home/dansk/.hermes/hermes-agent
git restore agent/smart_model_routing.py gateway/run.py

# 2. Pull latest upstream (if behind)
git pull

# 3. Restart gateway
hermes gateway stop && sleep 2 && hermes gateway start

Key symptoms

• Telegram messages go in but no response comes back
• Gateway logs show: WARNING gateway.platforms.base: [Telegram] Handler returned empty/None response
• Gateway log shows 🔧 patch entries with [error] — Hermes tried to patch itself and failed
• Gateway restarts/stops repeatedly in short succession

Prevention

• After running hermes update, check git status in the source directory
• If Hermes starts self-modifying code during a conversation, restart the gateway to stop it

Corrupted Thinking Block Auto-Recovery (patched April 4, 2026)

What happens

When a session accumulates many messages, a "thinking" block can get corrupted (invalid signature). The API rejects every retry with HTTP 400: Invalid signature in thinking block. The session becomes permanently stuck.

Symptoms

• Telegram bot stops responding
• Gateway log shows: API call failed after 3 retries. HTTP 400: Provider returned error
• Error details contain: Invalid signature in thinking block
• Retries don't help because the same corrupted history is sent each time

Patch applied

File: /home/dansk/.hermes/hermes-agent/run_agent.py Patch script: C:\Users\dansk\.paperclip\hermes-thinking-patch.py

The patch detects "invalid signature + thinking" errors and automatically clears the session history, keeping only the last user message. This recovers the session instead of aborting.

Re-apply after hermes update

python "C:\Users\dansk\.paperclip\hermes-thinking-patch.py"
wsl -d Ubuntu -- bash -c "source ~/.local/bin/env 2>/dev/null; hermes gateway stop; sleep 2; hermes gateway start"

NOTE: This patch WILL be overwritten by hermes update. Always re-apply after updating.

Auto-Start (verified April 3, 2026)

Everything starts automatically on PC reboot:

• Ollama -- Ollama.lnk in Windows Startup folder
• Hermes -- Hermes Agent.lnk in Startup folder + systemd linger enabled
• WSL -- starts when Hermes shortcut runs No manual steps needed after reboot.

Clean Restart Script (added April 4, 2026)

Location: /home/dansk/.hermes/hermes-restart.sh

Usage

# From Claude Code (Windows):
wsl -e bash -c "source ~/.local/bin/env; bash /home/dansk/.hermes/hermes-restart.sh"

# With update:
wsl -e bash -c "source ~/.local/bin/env; bash /home/dansk/.hermes/hermes-restart.sh --update"

# Full kill (force-kill all hermes processes first):
wsl -e bash -c "source ~/.local/bin/env; bash /home/dansk/.hermes/hermes-restart.sh --full"

What it does (in order)

1. Sends hermes gateway stop (graceful)
2. Finds and kills ALL hermes python processes (SIGTERM, then SIGKILL if needed)
3. Resets gateway_state.json and processes.json to clean state
4. (Optional) Runs hermes update if --update flag passed
5. Starts gateway with nohup in background
6. Waits 8 seconds, verifies process is alive and Telegram is connecting
7. Reports status

When to use

• Gateway stuck / not responding
• After config changes
• After hermes update
• Zombie processes blocking restart
• User says "restart hermes", "hermes is stuck", "fix hermes"

Restart Cheat Sheet

| What | When needed | How | |------|-------------|-----| | Full clean restart | Gateway stuck, zombie processes | Claude runs: wsl -e bash -c "source ~/.local/bin/env; bash /home/dansk/.hermes/hermes-restart.sh" | | Hermes gateway only | Config changed, quick restart | Claude runs: wsl.exe bash -c "hermes gateway stop; sleep 2; hermes gateway start" | | Hermes CLI | Only if CLI open in terminal | User types /quit then hermes | | Ollama | Model stuck, not responding | Claude runs: powershell Stop-Process -Name ollama -Force (auto-restarts) | | WSL | WSL itself frozen (rare) | User opens PowerShell as admin: wsl --shutdown then wsl |

Key rule: You almost NEVER need to restart WSL. 99% of the time it's just the restart script or Ollama.