Elmanda1/Troubleshoot DataPipeline OS

Troubleshooting Guide

Diagnostic Steps

When an error occurs, follow this triage order:

1. Check the terminal log — Flask server prints all engine logs with timestamps
2. Check the UI terminal — Center panel shows color-coded logs (SYS/OK/WARN/ERR)
3. Check debug screenshots — config/login_*.png for Twitter login issues
4. Check output folder — data_output/ for partial data that was written

Known Issues and How to Handle Them

Twitter Login Failures

Symptom: Engine falls back to error state with login failure message

Root Cause: Twitter frequently changes HTML selectors on the login page.

Fix:

1. Check screenshot at config/login_debug.png or config/login_debug_pwd.png
2. Open engines/twitter_engine.py
3. Find USERNAME_SELECTORS (around line 314) and PASSWORD_SELECTORS (around line 320)
4. Open Twitter login page manually in browser → Inspect Element → find the current selector
5. Update the selector list
6. Delete old session: via UI "Clear Session" button or delete config/chromium_profile/ and config/twitter_session.json

Twitter White/Blank Login Page

Symptom: Playwright opens Twitter login but page is blank white/grey (no form rendered)

Root Cause: Twitter login is a React SPA. The page DOM loads but JavaScript hasn't mounted the React app yet.

Fix (already applied in current engine):

1. Engine uses wait_until="networkidle" instead of "domcontentloaded"
2. Stealth JS injection overrides navigator.webdriver to prevent detection
3. Auto-detects blank page via document.body.scrollHeight < 200 and reloads
4. If still blank after reload, check:
   • Internet connection speed (slow connections need longer wait)
   • Chromium profile may be corrupted → Clear session from UI
   • Twitter may be blocking the IP → Try with VPN/different IP
5. Manual fix: delete config/chromium_profile/ and config/twitter_session.json, then retry

Twitter Rate Limiting

Symptom: Log shows rate limit warning, engine stops early

Fix: This is by design. Engine auto-skips to next keyword. Wait 15 minutes before rerunning. For large date ranges, reduce keywords or narrow the range.

Port 5000/8080 Conflict

Symptom: OSError: [Errno 48] Address already in use

Fix:

1. Check what's using the port:

   netstat -ano | findstr :8080   # Windows
   lsof -i :8080                  # macOS/Linux
   

2. Kill the process, or change port in app.py (last line):

   app.run(debug=True, port=8081, use_reloader=False)
   

Reddit API Rejection

Symptom: 403 Forbidden from Reddit API

Fix: Reddit requires accounts with some activity history. New accounts are rejected. Wait 5-7 days with normal activity, then try again.

Google Trends Rate Limit

Symptom: TooManyRequestsError from pytrends

Fix: pytrends is limited to ~10 requests per minute by Google. Reduce number of keywords or increase delay between requests in trends_engine.py.

ModuleNotFoundError

Symptom: ModuleNotFoundError: No module named 'xxx'

Fix:

# Ensure venv is active
venv\Scripts\activate          # Windows
source venv/bin/activate       # macOS/Linux

# Reinstall
pip install -r requirements.txt --timeout 120

Empty Output / No Data

Symptom: Pipeline completes but CSV is empty or very small

Possible Causes:

1. Keywords too specific — Try broader terms
2. Date range too narrow — Expand the range
3. API credentials missing — Check .env, engines will use simulation mode
4. All feeds down — News engine depends on RSS feed availability

Diagnostic:

# Check if engines ran in simulation mode
# Look for "_sim" in the source column of output CSV

Playwright / Chromium Issues

Symptom: playwright._impl._errors.Error: Executable doesn't exist

Fix:

pip install playwright
playwright install chromium

If behind a proxy:

set HTTPS_PROXY=http://proxy:port      # Windows
export HTTPS_PROXY=http://proxy:port   # macOS/Linux
playwright install chromium

Engine-Specific Debugging

Check Engine Status via API

curl http://localhost:8080/api/state | python -m json.tool

Look at engines object — each engine shows status, rows, ram_mb.

Force Simulation Mode

To test without any API keys, ensure all API-related variables in .env are empty. All engines will automatically fall back to simulation.

Check Twitter Session

curl http://localhost:8080/api/twitter/info

Returns: session_exists, profile_exists, username, has_password.