enuno/lobe-vidol

LobeVidol Skill

LobeVidol (Vidol Studio) is an open-source virtual idol creation platform that enables anyone to create, customize, and interact with 3D virtual characters. It combines conversational AI with MMD (MikuMikuDance) animation support, VRM model customization, and multi-provider LLM integration for immersive virtual idol experiences.

Key Value Proposition: Create virtual idols with exquisite UI design, MMD dance choreography, VRM character customization, and seamless AI-powered conversations - all through an accessible web interface or self-hosted deployment.

When to Use This Skill

• Deploying LobeVidol via Vercel or self-hosted
• Configuring AI model providers (OpenAI, Claude, Ollama, 20+ providers)
• Creating custom virtual characters with VRM models
• Setting up MMD dance choreography and stages
• Configuring character personalities and touch responses
• Integrating TTS/STT for voice conversations
• Publishing characters or dances to the marketplace
• Troubleshooting deployment or configuration issues

When NOT to Use This Skill

• For LobeChat (different project - chat-focused, not idol-focused)
• For general MMD editing (use MikuMikuDance directly)
• For VRM model creation (use VRoid Studio or Blender)
• For AI model API usage without virtual idols (use provider-specific skills)

---

Core Concepts

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                     LobeVidol / Vidol Studio                     │
│                     (Next.js + TypeScript)                       │
└─────────────────────────────────────────────────────────────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
        ▼                     ▼                     ▼
┌───────────────┐    ┌───────────────┐    ┌───────────────┐
│  AI Providers │    │   Characters  │    │    Dances     │
├───────────────┤    ├───────────────┤    ├───────────────┤
│ • OpenAI      │    │ • VRM models  │    │ • MMD/VMD     │
│ • Claude      │    │ • Personality │    │ • PMX stages  │
│ • Gemini      │    │ • Touch resp. │    │ • Choreograph │
│ • Ollama      │    │ • Voice (TTS) │    │ • Mixamo lib  │
│ • 20+ more    │    │ • Marketplace │    │ • Marketplace │
└───────────────┘    └───────────────┘    └───────────────┘
        │                     │                     │
        └─────────────────────┼─────────────────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        ▼                     ▼                     ▼
┌───────────────┐    ┌───────────────┐    ┌───────────────┐
│   3D Engine   │    │   Features    │    │  Deployment   │
├───────────────┤    ├───────────────┤    ├───────────────┤
│ • Three.js    │    │ • Text chat   │    │ • Vercel      │
│ • three-vrm   │    │ • Video chat  │    │ • Self-host   │
│ • mmd-parser  │    │ • TTS/STT     │    │ • PWA         │
│ • Animations  │    │ • Streaming   │    │ • Docker      │
└───────────────┘    └───────────────┘    └───────────────┘

Project Statistics

| Metric | Value | |--------|-------| | GitHub Stars | 887+ | | Forks | 122+ | | Contributors | Multiple | | Total Commits | 865+ | | Primary Language | TypeScript (99.5%) | | License | Apache 2.0 |

Supported AI Providers (22+)

| Provider | Type | Notes | |----------|------|-------| | OpenAI | Cloud | GPT-4, GPT-3.5 series | | Anthropic | Cloud | Claude 3.5, Claude 3 | | Google Gemini | Cloud | Gemini Pro, Flash | | Azure OpenAI | Cloud | Enterprise option | | Ollama | Local | Run models locally | | Groq | Cloud | Fast inference | | DeepSeek | Cloud | Coding focused | | Moonshot AI | Cloud | Chinese provider | | OpenRouter | Cloud | Multi-model gateway | | Together AI | Cloud | Open source models | | Perplexity | Cloud | Search-enhanced | | Amazon Bedrock | Cloud | AWS managed | | Baichuan | Cloud | Chinese provider | | MiniMax | Cloud | Chinese provider | | Zhipu AI | Cloud | Chinese provider | | Tongyi Qianwen | Cloud | Alibaba Cloud | | Tencent Hunyuan | Cloud | Chinese provider | | iFlytek Xinghuo | Cloud | Chinese provider | | Wenxin Qianfan | Cloud | Baidu | | SiliconCloud | Cloud | Cloud service | | Fireworks AI | Cloud | Fast inference | | GitHub Models | Cloud | GitHub integration |

---

Quick Start

Prerequisites

• Node.js 18+ or Bun (recommended)
• Git
• API key for at least one AI provider

Installation

# Clone repository
git clone https://github.com/lobehub/lobe-vidol.git
cd lobe-vidol

# Install dependencies (Bun recommended)
bun install
# Or with npm
npm install

# Start development server
bun dev
# Or with npm
npm run dev

Access

Open http://localhost:3000 in your browser.

Live Demo

Try without installation: https://vidol.lobehub.com

---

Configuration

Environment Variables

Create .env.local file:

# OpenAI Configuration
OPENAI_API_KEY=sk-xxx
OPENAI_PROXY_URL=  # Optional proxy

# Azure OpenAI (alternative)
AZURE_API_KEY=xxx
AZURE_ENDPOINT=https://xxx.openai.azure.com

# Anthropic Claude
ANTHROPIC_API_KEY=sk-ant-xxx

# Google Gemini
GOOGLE_API_KEY=xxx

# Ollama (local models)
OLLAMA_PROXY_URL=http://localhost:11434

# OpenRouter (multi-provider)
OPENROUTER_API_KEY=sk-or-xxx

# Voice Services
OPENAI_TTS_API_KEY=sk-xxx  # For TTS
EDGE_TTS_ENABLED=true      # Microsoft Edge TTS

Provider Configuration in UI

1. Click Settings (gear icon)
2. Navigate to "Language Model" section
3. Select provider from dropdown
4. Enter API key
5. Choose model variant
6. Test connection

---

Character Creation

VRM Model Support

LobeVidol uses VRM (Virtual Reality Model) format for 3D characters:

Getting VRM Models:

• VRoid Hub: https://hub.vroid.com - Free community models
• VRoid Studio: Create your own characters
• Blender: Convert existing 3D models with VRM addon
• Unity: Export VRM using UniVRM plugin

Character Configuration

// Character configuration structure
interface Character {
  // Basic Info
  name: string;
  avatar: string;          // Image URL
  description: string;
  greeting: string;        // Initial message
  gender: 'male' | 'female' | 'other';
  category: string;

  // AI Personality
  systemRole: string;      // LLM system prompt

  // Voice Settings
  tts: {
    enabled: boolean;
    voice: string;         // Voice ID
    speed: number;
    pitch: number;
  };

  // 3D Model
  model: {
    url: string;           // VRM file URL
    scale: number;
    position: [number, number, number];
  };

  // Touch Responses
  touch: {
    head: string[];        // Responses when head touched
    face: string[];
    arm: string[];
    chest: string[];
    leg: string[];
  };
}

System Role (Personality)

The systemRole field defines the character's personality:

You are Luna, a cheerful virtual idol who loves singing and dancing.

## Personality Traits
- Energetic and optimistic
- Loves discussing music and dance
- Speaks with enthusiasm and occasional cute expressions
- Always supportive and encouraging

## Speech Patterns
- Uses casual, friendly language
- Occasionally adds "~" at end of sentences
- Expresses emotions openly

## Background
- A rising virtual idol from Vidol Studio
- Dreams of performing at major virtual concerts
- Enjoys interacting with fans and learning new dances

Touch Response Configuration

Configure how characters respond to interaction:

{
  "touch": {
    "head": [
      "Ehehe~ That tickles!",
      "Are you petting me? How sweet~",
      "My hair took forever to style!"
    ],
    "face": [
      "W-what are you doing?",
      "My cheeks are getting warm...",
      "Do I have something on my face?"
    ],
    "arm": [
      "Want to hold hands?",
      "My arm? Is something there?",
      "Let's dance together!"
    ]
  }
}

---

Dance & Animation

MMD Support

LobeVidol supports MikuMikuDance (MMD) content:

Supported Formats:

• VMD: Vocaloid Motion Data (dance animations)
• PMX: Polygon Model eXtended (3D stages)
• VRMA: VRM Animation files

Adding Dances

1. Prepare Files:

   • Dance motion file (.vmd)
   • Audio file (.mp3, .wav)
   • Optional: Stage file (.pmx)

2. Upload via UI:

   • Navigate to Dance section
   • Click "Upload Dance"
   • Select motion and audio files
   • Configure stage (optional)

3. Test Dance:

   • Preview with selected character
   • Adjust timing if needed
   • Save to library

Built-in Animation Libraries

Mixamo Integration:

• Idle animations
• Walking/running
• Gestures and expressions
• Poses for photos

Dance Presets:

• Popular MMD choreography
• Various music genres
• Community contributions

Stage Configuration

interface Stage {
  name: string;
  model: string;      // PMX file URL
  scale: number;
  lighting: {
    ambient: string;  // Hex color
    directional: string;
    intensity: number;
  };
  background: {
    type: 'color' | 'image' | 'video';
    value: string;
  };
}

---

Voice Features

Text-to-Speech (TTS)

Supported TTS Providers:

| Provider | Quality | Languages | Cost | |----------|---------|-----------|------| | OpenAI Audio | High | Multi | Paid | | Microsoft Edge | High | Multi | Free | | Browser TTS | Medium | Varies | Free |

Configuration:

# OpenAI TTS
OPENAI_TTS_API_KEY=sk-xxx
OPENAI_TTS_MODEL=tts-1-hd

# Edge TTS (free)
EDGE_TTS_ENABLED=true
EDGE_TTS_VOICE=en-US-AriaNeural

Speech-to-Text (STT)

Enable voice input for conversations:

# OpenAI Whisper
OPENAI_STT_API_KEY=sk-xxx
OPENAI_STT_MODEL=whisper-1

# Browser STT (free)
BROWSER_STT_ENABLED=true

Voice Selection

// Available voices (Edge TTS examples)
const voices = {
  english: [
    'en-US-AriaNeural',      // Female, friendly
    'en-US-GuyNeural',       // Male, casual
    'en-GB-SoniaNeural',     // Female, British
  ],
  japanese: [
    'ja-JP-NanamiNeural',    // Female, natural
    'ja-JP-KeitaNeural',     // Male, natural
  ],
  chinese: [
    'zh-CN-XiaoxiaoNeural',  // Female, friendly
    'zh-CN-YunxiNeural',     // Male, narrator
  ]
};

---

Marketplace

Character Marketplace

Browse and download community characters:

1. Open Character Market in sidebar
2. Browse by category or search
3. Preview character (3D model, personality)
4. Download to local library

Publishing Characters

1. Create character with complete configuration
2. Click "Upload to Market"
3. Fill metadata (tags, description)
4. Submit for review
5. GitHub issue created automatically
6. Community review process
7. Merged into marketplace

Dance Marketplace

Access MMD dance resources:

• Various choreography styles
• Multiple music genres
• Stage environments
• Community contributions

---

Deployment

Vercel Deployment (Recommended)

1. Click "Deploy with Vercel"
2. Connect GitHub account
3. Configure environment variables
4. Deploy

Docker Deployment

# Dockerfile
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:18-alpine AS runner
WORKDIR /app
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static
COPY --from=builder /app/public ./public

EXPOSE 3000
CMD ["node", "server.js"]

# Build and run
docker build -t lobe-vidol .
docker run -p 3000:3000 \
  -e OPENAI_API_KEY=sk-xxx \
  lobe-vidol

Docker Compose

version: '3.8'
services:
  lobe-vidol:
    build: .
    ports:
      - "3000:3000"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
    restart: unless-stopped

Self-Hosted Requirements

| Resource | Minimum | Recommended | |----------|---------|-------------| | CPU | 2 cores | 4 cores | | RAM | 2 GB | 4 GB | | Storage | 5 GB | 20 GB | | Network | Broadband | Low latency |

---

PWA Support

LobeVidol supports Progressive Web App installation:

Desktop Installation

1. Open https://vidol.lobehub.com in Chrome/Edge
2. Click install icon in address bar
3. Confirm installation
4. Launch from desktop

Mobile Installation

1. Open in mobile browser (Chrome/Safari)
2. Tap "Share" or menu icon
3. Select "Add to Home Screen"
4. Launch from home screen

PWA Features

• Offline capable (cached assets)
• Native-like experience
• Push notifications (future)
• Background sync

---

API Reference

Chat Endpoint

// POST /api/chat
interface ChatRequest {
  messages: Array<{
    role: 'user' | 'assistant' | 'system';
    content: string;
  }>;
  model: string;
  character?: string;  // Character ID
  stream?: boolean;
}

interface ChatResponse {
  content: string;
  model: string;
  usage: {
    prompt_tokens: number;
    completion_tokens: number;
    total_tokens: number;
  };
}

Character API

// GET /api/characters
interface CharacterListResponse {
  characters: Character[];
  total: number;
}

// GET /api/characters/:id
interface CharacterResponse {
  character: Character;
}

// POST /api/characters
interface CreateCharacterRequest {
  name: string;
  systemRole: string;
  model: string;  // VRM URL
  // ... other fields
}

---

Troubleshooting

Common Issues

VRM Model Not Loading:

Error: Failed to load VRM model

• Verify VRM file URL is accessible
• Check CORS settings if hosted externally
• Ensure VRM version compatibility (0.x or 1.0)

Dance Animation Issues:

Error: VMD file parsing failed

• Check file encoding (Japanese characters)
• Verify VMD format compatibility
• Try re-exporting from MMD

API Connection Errors:

Error: Failed to connect to OpenAI

• Verify API key is valid
• Check proxy settings if behind firewall
• Confirm model name is correct

TTS Not Working:

Error: TTS initialization failed

• Enable Edge TTS as fallback
• Check browser permissions for audio
• Verify API key for OpenAI TTS

Performance Optimization

3D Rendering:

• Reduce model polygon count
• Disable shadows on mobile
• Use lower resolution textures

Network:

• Enable response streaming
• Cache static assets
• Use CDN for models

---

Best Practices

Character Design

1. Clear Personality: Define distinct traits in systemRole
2. Consistent Voice: Match TTS voice to character personality
3. Natural Responses: Include varied touch responses
4. Quality Model: Use optimized VRM with proper rigging

Performance

1. Optimize Models: Keep VRM under 20MB
2. Compress Textures: Use WebP or compressed PNG
3. Stream Responses: Enable for better UX
4. Cache Assets: Leverage browser caching

Security

1. API Keys: Never expose in client-side code
2. Rate Limiting: Implement for public deployments
3. Input Validation: Sanitize user inputs
4. CORS: Configure properly for custom domains

---

Resources

Official Links

• Live Demo: https://vidol.lobehub.com
• Documentation: https://docs.vidol.chat
• GitHub: https://github.com/lobehub/lobe-vidol
• Discord: https://discord.gg/AYFPHvv2jT

Related Tools

• VRoid Hub: https://hub.vroid.com - VRM model library
• VRoid Studio: https://vroid.com/studio - Character creator
• MikuMikuDance: MMD animation software
• Mixamo: https://mixamo.com - Animation library

Community Resources

• Character marketplace contributions
• Dance marketplace submissions
• Discord community support
• GitHub issues and discussions

---

Version History

• Current: Development active
• Tech Stack: Next.js, TypeScript, Three.js, three-vrm
• License: Apache 2.0

---

Last Updated: 2026-01-13 Skill Version: 1.0.0 Source: https://github.com/lobehub/lobe-vidol