Elmanda1/Run Pipeline

How to Run the Pipeline

Quick Start (Windows)

# Option 1: Double-click run.bat in the project root

# Option 2: Manual
cd datapipeline_os
venv\Scripts\activate
python app.py
# Open http://localhost:8080

Quick Start (macOS/Linux)

# Option 1: ./run.sh

# Option 2: Manual
cd datapipeline_os
source venv/bin/activate
python app.py
# Open http://localhost:8080

First-Time Setup

1. Create virtual environment:

   python -m venv venv
   

2. Activate venv:

   venv\Scripts\activate          # Windows
   source venv/bin/activate       # macOS/Linux
   

3. Install dependencies:

   pip install -r requirements.txt --timeout 120
   

4. Configure environment:

   copy .env.example .env         # Windows
   cp .env.example .env           # macOS/Linux
   

   Then edit .env with your API credentials.

5. For Twitter live scraping (optional):

   pip install playwright
   playwright install chromium
   

Running a Pipeline via UI

1. Open http://localhost:8080 in browser
2. Left Panel — Fill in:
   • Project Name (e.g., Analisis_Tren_Korea)
   • Keywords (comma-separated)
   • Date Range (start → end)
   • Select platforms (checkboxes)
   • Output format (CSV/JSON/SQLite)
   • Chunk size (default: 500)
3. Click "Run Pipeline"
4. Center Panel — Monitor progress in real-time
5. Right Panel — Download files when complete

Running via API (Programmatic)

curl -X POST http://localhost:8080/api/run \
  -H "Content-Type: application/json" \
  -d '{
    "project_name": "Analisis_Tren_Korea",
    "keywords": ["Drama Korea", "Kpop"],
    "date_range": {"start": "2020-01-01", "end": "2024-12-31"},
    "platforms": ["google_trends", "twitter"],
    "language": "id",
    "chunk_size": 500,
    "output": {"format": "csv", "path": "./data_output/"}
  }'

Output

• Files are saved in data_output/YYYYMMDD-HHMMSS/
• Per-source files: <project>_<source>.csv
• Combined file: <project>_combined.csv
• All files follow the unified 5-column schema

Stopping a Run

• Click "Stop" in the UI, or:

  curl -X POST http://localhost:8080/api/stop
  

• Partial data collected so far will still be exported

Common Issues

| Problem | Fix | |---|---| | Port 8080 in use | Change port in app.py last line | | Venv not activated | Run venv\Scripts\activate first | | Module not found | Run pip install -r requirements.txt | | Playwright missing | Run pip install playwright && playwright install chromium |