BBuf/sglang-glm-vlm-ocr-optimization
skills/model-optimization/sglang/sglang-glm-vlm-ocr-optimization/SKILL.md
PR-backed and current-main optimization manual for GLM-4V, GLM-4.1V, GLM-4.5V, GLM-4.6V, GLM-Glyph, and GLM-OCR in SGLang. Use when an engineer needs to recover, extend, or audit GLM vision processors, GLM4V MoE, vision encoder DP/PP, GLM-OCR/NextN loading, transformers compatibility, Conv3D-to-linear projection, AMD/NPU validation, or VLM/OCR cookbook recipes.
$ install --global
skillsauthnpx skillsauth add BBuf/AI-Infra-Auto-Driven-SKILLS sglang-glm-vlm-ocr-optimizationInstall this skill globally with one command. Works with Claude Code, Cursor, and Windsurf.
Security Scan Results
3 of 9 scanners reported clean
Some scanners were skipped, did not run, or reported a non-clean status. Review each row below.
3
Scanners Passed
9
Scanners in report
Clean
TrivyContainer and dependency vulnerability scanner
95%
Clean
SemgrepStatic code analysis for vulnerabilities
95%
Clean
mcp-scan (Snyk)Model Context Protocol security validation
95%
Skipped
Snyk (dep)Open source security scanning
50%
Skipped
Socket.devSupply chain security analysis
50%
Skipped
VirusTotalMulti-engine malware detection
50%
Skipped
CrowdStrikeAdvanced threat intelligence
50%
Skipped
OSV-ScannerOpen Source Vulnerability database check
50%
Skipped
OWASP Dep-Check
50%
Last scanned: May 2, 2026, 2:32 AM104.1s3 files scanned
SKILL.md
- name:
- sglang-glm-vlm-ocr-optimization
- description:
- PR-backed and current-main optimization manual for GLM-4V, GLM-4.1V, GLM-4.5V, GLM-4.6V, GLM-Glyph, and GLM-OCR in SGLang. Use when an engineer needs to recover, extend, or audit GLM vision processors, GLM4V MoE, vision encoder DP/PP, GLM-OCR/NextN loading, transformers compatibility, Conv3D-to-linear projection, AMD/NPU validation, or VLM/OCR cookbook recipes.
SGLang GLM VLM/OCR Optimization
Overview
This skill covers GLM visual and OCR models: GLM-4V, GLM-4.1V, GLM-4.5V, GLM-4.6V, GLM-Glyph, and GLM-OCR. It is separate from GLM text because the processor, vision encoder, image token placement, transformers compatibility, and OCR-specific NextN behavior dominate debugging.
Current evidence snapshot:
- SGLang
origin/main:b3e6cf60aon2026-04-22 - sgl-cookbook
origin/main:816bad5on2026-04-21 - Detailed PR diff dossier:
references/pr-history.md - Runtime files:
glm4v.py,glm4v_moe.py,glm_ocr.py,glm_ocr_nextn.py - Processor files:
multimodal/processors/glm4v.py,glmasr.pywhen audio-adjacent - Docs/snippets: GLM-4.5V, GLM-4.6V, GLM-Glyph, GLM-OCR cookbook pages
Non-Negotiable Evidence Rule
Use skills/model-optimization/model-pr-diff-dossier/SKILL.md as the production bar.
Every PR cited for this family must be based on diff reading, not only PR titles.
Before You Change Anything
Capture:
- model: GLM-4V/4.1V/4.5V/4.6V/Glyph/OCR
- input type: image, OCR page, multi-image, or video-like visual sequence
- encoder DP/PP enabled or disabled
- quantization: BF16, FP8, compressed-tensors, or vendor format
- transformers version and whether config fields are optional/missing
- NPU/AMD/CPU backend
- whether OCR NextN is active
Core Principle
Start at the multimodal boundary.
- If text-only GLM passes, inspect processor registration, image feature projection, position IDs, and vision model DP before text MoE.
- GLM-OCR often fails because of processor/config field drift or NextN omission, not only because of kernels.
- GLM-4.5V/4.6V share text MoE work but need separate vision validation.
- Treat
#20463and#20740as a pair:#20463documents a real Conv3D-to-linear loader regression mechanism, while#20740defines the current-main state after revert.
Main Runtime Surfaces
python/sglang/srt/models/glm4v.pypython/sglang/srt/models/glm4v_moe.pypython/sglang/srt/models/glm_ocr.pypython/sglang/srt/models/glm_ocr_nextn.pypython/sglang/srt/multimodal/processors/glm4v.pydocs_new/cookbook/autoregressive/GLM/GLM-4.5V.mdxdocs_new/cookbook/autoregressive/GLM/GLM-4.6V.mdxdocs_new/cookbook/autoregressive/GLM/GLM-Glyph.mdxdocs_new/cookbook/autoregressive/GLM/GLM-OCR.mdx
Optimization Order
- Validate processor registration and image feature tensor shapes.
- Validate text-only fallback if the model supports it.
- Validate encoder DP/PP and TP assumptions.
- Validate quantized startup.
- Validate OCR/Glyph examples.
- Update cookbook and registered tests.
Open PRs to Track
- #9349: GLM-4.5V FP8.
- #14662: GLM4.6V ktransformers.
- #19728: ROCm GLM-4.5V-FP8 startup.
- #22961: NPU GLM-4.5V.
PR Dossier Rule
When producing or updating model optimization history, do not summarize a PR as one sentence. Read the source diff and write:
- motivation/root cause
- key implementation idea
- a short code fragment that proves the implementation surface
- validation evidence
- current-main status and risk, especially for reverted or open PRs
Validation Lanes
- GLM-4.5V/4.6V image prompts.
- GLM-OCR page recognition.
- GLM-Glyph examples.
- Vision encoder DP/PP.
- Transformers compatibility.
- AMD/NPU startup and accuracy.
Related Skills
BBuf/sglang-humanize-review
development
VerifiedTrustedCommunity
Perform SGLang code review in the style of human maintainers by consulting the full non-agent PR review episode corpus from project start through the latest refresh (June 2026), including inline review threads, top-level PR comments, review submissions, original multilingual text, and multi-round discussions. Use when reviewing SGLang PRs, diffs, patches, or local changes for correctness, tests, performance, GPU/runtime risks, API compatibility, and maintainability.
531SKILL.mdUpdated May 21, 2026
BBuf/model-pr-history-knowledge
documentation
VerifiedTrustedCommunity
Use when an SGLang, vLLM, or TensorRT-LLM serving/model optimization task needs prior model-family PR evidence. Query and read the PR-driven history docs under model-pr-optimization-history before choosing source paths, fast paths, kernel/fusion ideas, regression risks, or validation lanes.
531SKILL.mdUpdated May 17, 2026
BBuf/vllm-sota-humanize-loop
development
VerifiedTrustedCommunity
Run an autonomous Humanize-governed vLLM SOTA performance loop for one LLM model: first perform the fixed fair vLLM/SGLang/TensorRT-LLM deployment search and benchmark, then start one RLCR loop that repeatedly decides the gap, profiles the current bottleneck, runs layer/kernel pipeline analysis, patches vLLM code, optionally uses ncu-report-skill for kernel evidence, and revalidates until vLLM matches or beats the best observed framework under the same workload and SLA.
423SKILL.mdUpdated May 27, 2026
BBuf/llm-pipeline-analysis
devops
VerifiedTrustedCommunity
Inspect LLM torch profiler traces at forward-pass, layer, and kernel level. Use when you need layer timings, anchor-kernel boundaries, representative kernel flows, or Perfetto time ranges.
423SKILL.mdUpdated May 21, 2026