hsliuustc0106/skills/vllm-omni-nightly-local

vLLM-Omni Nightly Local (cluster run & log sync)

Overview

1. Login - SSH, squeue / srun --overlap, docker exec without -it (see sections below).
2. Run cases — In the same container shell as the script: source /rebase/.venv/bin/activate (see references/nightly-local-environment.md), then set Hugging Face / vLLM env (see below and that reference), optionally nvidia-smi → CUDA_VISIBLE_DEVICES, then bash tools/nightly/run_nightly_jobs.sh with LOG_DIR="$REPO_ROOT/logs/nightly_jobs"。Long runs: run inside tmux / screen on the cluster, or start the script with nohup and redirect stdout/stderr to a log file under $REPO_ROOT/logs/, so an SSH disconnect does not stop the workload (details: references/nightly-local-environment.md — Long runs / SSH disconnect).
3. Sync logs — On your laptop, (1) if $REPO_ROOT/logs/nightly_perf_manual.xlsx exists from the last run, copy it to nightly_perf_manual.prev.xlsx (baseline for report ↑/↓). (2) Remove the local $REPO_ROOT/logs/nightly_jobs tree (rm -rf) so the new pull does not mix old job folders with the latest run. (3) Copy nightly_jobs and nightly_perf_manual.xlsx into $REPO_ROOT/logs/: references/nightly-local-log-fetch.md.

Analyze and write the HTML test report in vllm-omni-test-report (report kind nightly): export REPO_ROOT=/path/to/local/vllm-omni then python scripts/nightly_local_log_report.py --html-report ... (defaults to $REPO_ROOT/logs/nightly_jobs; pass --log-dir only if you used a different tree).

Required user inputs

| Input | Meaning | |-------|---------| | SSH connection name | Host alias or user@host. | | Slurm username | For squeue -u .... | | Docker container name | For docker exec. | | Empty GPU count | Optional but recommended before run_nightly_jobs.sh: how many free GPUs to use (X). The agent runs nvidia-smi, picks X indices with no (or minimal) load, and export CUDA_VISIBLE_DEVICES=… in the same shell / docker exec bash -lc as the script. See §2 and the section CUDA_VISIBLE_DEVICES — empty GPUs in references/nightly-local-environment.md. |

Optional: REPO_ROOT inside the container.

---

1. Login environment

1.1 SSH

ssh -v "<SSH_CONNECTION_NAME>"

Load module load slurm (or site equivalent) before srun if needed.

1.2 Find JOBID

SLURM_USER="<username>"
squeue -u "$SLURM_USER" -t RUNNING -h -o "%i"

Confirm JOBID when multiple rows exist.

1.3 Run in container (no TTY)

JOBID="<chosen_jobid>"
srun --jobid="$JOBID" --overlap docker exec "<CONTAINER_NAME>" bash -lc '<commands>'

Nightly one-liner:

srun --jobid="$JOBID" --overlap docker exec "<CONTAINER_NAME>" bash -lc 'source /rebase/.venv/bin/activate && export REPO_ROOT=/path/to/vllm-omni && cd "$REPO_ROOT" && bash tools/nightly/run_nightly_jobs.sh'

1.4 New allocation if no JOBID

srun -p q-fq9hpsac -w hk01dgx006 --gres=gpu:0 --mem-per-cpu=8G --pty  --job-name=ci_local_test

Then docker exec "<CONTAINER_NAME>" bash -lc '<commands>'.

1.5 Optional: docker exec -it for debugging only

1.6 Agent: BatchMode SSH

ssh -o BatchMode=yes -o ConnectTimeout=30 "<SSH_CONNECTION_NAME>" \
  "bash -lc 'type module >/dev/null 2>&1 && module load slurm 2>/dev/null; squeue -u \"<SLURM_USER>\" -t RUNNING -h -o \"%i\"'"

ssh -o BatchMode=yes -o ConnectTimeout=120 "<SSH_CONNECTION_NAME>" \
  "bash -lc 'type module >/dev/null 2>&1 && module load slurm 2>/dev/null; srun --jobid=\"<JOBID>\" --overlap docker exec \"<CONTAINER_NAME>\" bash -lc \"<INNER_CMD>\"'"

Details: references/nightly-local-environment.md.

---

2. Run test cases

Before bash tools/nightly/run_nightly_jobs.sh (inside the same docker exec … bash -lc '…' or interactive shell on the node):

0. Python venv (required inside the container) — run first in that inner shell:

   source /rebase/.venv/bin/activate
   

   Details: references/nightly-local-environment.md (Python venv inside the container).

1. Model / HF / vLLM environment (required unless the user gives a different site policy) — same shell as the script:

   export HF_HOME="/home/models/"
   unset HF_HUB_CACHE
   unset TRANSFORMERS_CACHE
   export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
   

   Details: references/nightly-local-environment.md (Hugging Face cache and vLLM).

2. Ask the user for X = number of empty / free GPUs to use (or use a value they already gave in Required user inputs).

3. Run nvidia-smi, select X GPU indices that are idle (typically 0 MiB used and 0% util — site-specific thresholds in the reference), then set:

   export CUDA_VISIBLE_DEVICES='<comma-separated GPU indices>'
   

4. In that same environment, cd to $REPO_ROOT and run bash tools/nightly/run_nightly_jobs.sh (or your local test entrypoint).

Copy-paste patterns and fallback when fewer than X GPUs are strictly empty: references/nightly-local-environment.md (CUDA_VISIBLE_DEVICES — empty GPUs).

Example inner command (replace placeholders; X and device list come from nvidia-smi selection):

srun --jobid="$JOBID" --overlap docker exec "$CONTAINER_NAME" bash -lc '
  source /rebase/.venv/bin/activate
  export REPO_ROOT=/path/to/vllm-omni
  export HF_HOME="/home/models/"
  unset HF_HUB_CACHE
  unset TRANSFORMERS_CACHE
  export VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
  export CUDA_VISIBLE_DEVICES="0,1"
  cd "$REPO_ROOT" && bash tools/nightly/run_nightly_jobs.sh
'

(CUDA_VISIBLE_DEVICES shown as "0,1" only for illustration — derive from nvidia-smi, do not hardcode unless the user insists. HF_HOME must match the cluster mount for shared models; if the user specifies another path, use that instead.)

2.1 Background / resilient shell (recommended when tests run for a long time)

• Preferred: open tmux new -s nightly (or screen) before srun / docker exec, run the full §2 command inside that session, then detach (Ctrl-b d in tmux). Re-attach later with tmux attach -t nightly to watch progress.
• Alternative: inside docker exec … bash -lc '…', start the nightly script with nohup and append logs to a file under $REPO_ROOT/logs/ (see copy-paste in references/nightly-local-environment.md).

3. Sync logs off-cluster

Follow references/nightly-local-log-fetch.md (scp / rsync / tarball). Include logs/nightly_perf_manual.xlsx when it exists on the server.

---

Agent workflow

1. Collect SSH name, Slurm user, container name, and optional X empty GPUs for CUDA_VISIBLE_DEVICES; do not guess JOBID when ambiguous.
2. Run sections 1–2; inside the container, run source /rebase/.venv/bin/activate first; before the nightly/local test script, set HF / vLLM env vars (§2 step 1). Then, if X is set (or the user asks for GPU selection), run nvidia-smi, export CUDA_VISIBLE_DEVICES, and run the script in the same environment. For multi-hour jobs, use §2.1 (tmux / nohup) so a dropped SSH session does not kill the run.
3. Verify files under logs/nightly_jobs (and logs/nightly_perf_manual.xlsx when your workflow produces it).
4. For fetch to a laptop: (a) copy nightly_perf_manual.xlsx → nightly_perf_manual.prev.xlsx when baselines are needed; (b) rm -rf local logs/nightly_jobs before sync (see references/nightly-local-log-fetch.md); (c) then pull. For HTML report, point to vllm-omni-test-report nightly_local_log_report.py and layout ../vllm-omni-test-report/references/nightly-local-log-layout.md.

References

• Fetch logs (scp / rsync / tar): references/nightly-local-log-fetch.md
• Environment notes: references/nightly-local-environment.md