sschepis/comfyui-media-generator

ComfyUI Media Generator Skill

Generates images, videos, and animations using a local ComfyUI instance. Programmatically constructs workflow DAGs, submits them to ComfyUI's API, and retrieves outputs. Optimized for M4 Mac with 128GB RAM — supports SDXL, SVD, Flux, and AnimateDiff at full resolution.

Pipeline Position

Trend Detection → Campaign Planning → [ComfyUI Media Generator] → [Video Ad Creator] → Media Activation
                                            ↓                            ↓
                                    Product images/video           Rendered MP4 ad
                                    Scene assets (by role)         with voiceover

This skill generates the visual assets that feed into the video-ad-creator as product.images and product.videos.

Prerequisites

ComfyUI must be running locally:

cd ~/ComfyUI && python main.py --listen 127.0.0.1 --port 8188

Config is stored in SQLite. Set defaults:

import { setConfig } from './src/db.mjs';
setConfig('COMFYUI_HOST', 'http://127.0.0.1:8188');
setConfig('COMFYUI_CHECKPOINT', 'sd_xl_base_1.0.safetensors');
setConfig('COMFYUI_STEPS', '30');
setConfig('COMFYUI_CFG', '7.0');
setConfig('COMFYUI_SAMPLER', 'euler');

Input Contract — MediaGenerationRequest

{
  "type": "image | video | animation | product-assets",
  "prompt": "descriptive text prompt",
  "negativePrompt": "things to avoid",
  "style": "editorial beauty photography",
  "constraints": {
    "width": 1024,
    "height": 1024,
    "duration": "4s",
    "fps": 24,
    "frames": 25
  },
  "product": {
    "name": "Complete Lip Kit",
    "description": "Luxury lip care set with gloss, liner, and balm",
    "assetRoles": ["hero_product", "lifestyle", "ingredient"]
  },
  "options": {
    "checkpoint": "sd_xl_base_1.0.safetensors",
    "steps": 35,
    "cfg": 7.5,
    "sampler": "dpmpp_2m",
    "scheduler": "karras",
    "seed": -1,
    "batchSize": 1,
    "denoise": 0.7,
    "template": "beauty-hero-shot"
  }
}

Output Contract — MediaGenerationResult

{
  "status": "done",
  "type": "image",
  "outputs": [
    {
      "role": "hero_product",
      "filename": "instinctsai-hero_00001.png",
      "url": "http://127.0.0.1:8188/view?filename=...",
      "prompt": "the prompt used",
      "seed": 123456
    }
  ],
  "promptId": "comfyui-prompt-id",
  "workflowRecord": "data/comfyui-workflows/{id}.json"
}

Execution Steps

Step 1: Check ComfyUI Health

import { checkHealth, getQueueStatus } from './src/comfyui.mjs';

const health = await checkHealth();
if (!health.online) {
  throw new Error('ComfyUI is not running. Start it with: cd ~/ComfyUI && python main.py');
}
const queue = await getQueueStatus();
// queue.running, queue.pending

Step 2: Select Generation Strategy

Based on request.type:

| Type | Function | Use Case | |------|----------|----------| | image | generateImage(params) | Text-to-image generation | | variation | generateVariation(params) | Image-to-image refinement | | video | generateVideo(params) | Image-to-video via SVD | | animation | generateAnimation(params) | Text-to-animation via AnimateDiff | | product-assets | generateProductAssets(params) | Multi-role asset generation for campaigns |

Step 3: Build & Execute Workflow

Option A: Direct generation

import { generateImage } from './src/comfyui.mjs';

const result = await generateImage({
  prompt: 'Professional product photography of Complete Lip Kit, luxury beauty, 8k',
  negativePrompt: 'ugly, blurry, low quality, deformed, watermark',
  width: 1024,
  height: 1024,
  steps: 35,
  cfg: 7.5,
  sampler: 'dpmpp_2m',
  scheduler: 'karras',
  filenamePrefix: 'instinctsai-hero'
});
// result.status, result.images[], result.seed, result.promptId

Option B: Template-based generation

import { buildFromTemplate, listTemplates } from './src/comfyui-templates.mjs';
import { submitWorkflow, pollWorkflow } from './src/comfyui.mjs';

// List available templates
const templates = listTemplates();

// Build from template with variable substitution
const workflow = buildFromTemplate('beauty-hero-shot', {
  product: 'Complete Lip Kit with gloss, liner, and balm'
});

const { promptId } = await submitWorkflow(workflow);
const result = await pollWorkflow(promptId);

Option C: Full product asset generation

import { generateProductAssets } from './src/comfyui.mjs';

const { assets } = await generateProductAssets({
  productName: 'Complete Lip Kit',
  productDescription: 'Luxury lip care set with gloss, liner, and balm',
  style: 'editorial beauty photography',
  brandAesthetic: 'luxury, clean, premium, dark background with gold accents',
  assetRoles: ['hero_product', 'lifestyle', 'ingredient', 'flatlay'],
  width: 1024,
  height: 1024
});

// assets = [
//   { role: 'hero_product', status: 'done', images: [...], prompt: '...', seed: 123 },
//   { role: 'lifestyle', status: 'done', images: [...], prompt: '...', seed: 456 },
//   ...
// ]

Step 4: Generate Video from Image (optional)

import { generateVideo } from './src/comfyui.mjs';

// Take the hero product image and animate it
const video = await generateVideo({
  inputImage: assets[0].images[0].filename,
  width: 1024,
  height: 576,
  frames: 25,
  fps: 8,
  motionBucketId: 80,
  steps: 25,
  cfg: 2.5,
  filenamePrefix: 'product-reveal'
});
// video.status, video.videos[]

Step 5: Return Results to Pipeline

Package outputs for the video-ad-creator skill:

const pipelineOutput = {
  status: 'done',
  type: request.type,
  outputs: assets.map(a => ({
    role: a.role,
    filename: a.images[0]?.filename,
    url: a.images[0]?.url,
    prompt: a.prompt,
    seed: a.seed
  })),
  // These feed directly into video-ad-creator's product.images
  imageUrls: assets.flatMap(a => a.images.map(i => i.url)),
  videoUrls: videos?.map(v => v.url) || []
};

Available Templates

| Template | Type | Resolution | Description | |----------|------|------------|-------------| | beauty-hero-shot | text-to-image | 1024×1024 | Hero product shot, studio lighting | | beauty-lifestyle | text-to-image | 1024×1536 | Lifestyle with model, golden hour | | beauty-ingredient-macro | text-to-image | 1024×1024 | Macro ingredient close-up | | beauty-flatlay | text-to-image | 1024×1024 | Overhead flat-lay arrangement | | product-reveal-video | image-to-video | 1024×576 | Slow product reveal (SVD) | | texture-flow-video | image-to-video | 1024×576 | Flowing texture motion (SVD) | | tiktok-vertical | text-to-image | 768×1344 | Vertical format for TikTok/Reels | | instagram-square | text-to-image | 1024×1024 | Square format for Instagram |

Workflow Builders

Low-level builders for custom workflows:

| Builder | Function | Description | |---------|----------|-------------| | Text-to-Image | buildTextToImage(params) | Checkpoint → CLIP → Latent → KSampler → VAEDecode → Save | | Image-to-Image | buildImageToImage(params) | LoadImage → VAEEncode → KSampler → VAEDecode → Save | | Image-to-Video | buildImageToVideo(params) | LoadImage → SVD Conditioning → KSampler → VAEDecode → VideoCombine | | AnimateDiff | buildAnimateDiff(params) | Checkpoint → MotionModule → KSampler → VAEDecode → VideoCombine | | Raw | executeRawWorkflow(graph) | Submit any custom DAG |

Parameter Tuning Guide (M4 128GB)

With 128GB unified memory, you can run large models efficiently:

| Parameter | Conservative | Balanced | Maximum Quality | |-----------|-------------|----------|-----------------| | Resolution | 512×512 | 1024×1024 | 2048×2048 | | Steps | 20 | 30-35 | 50 | | CFG | 5.0 | 7.0-7.5 | 10.0 | | Batch Size | 1 | 2-4 | 8 | | Video Frames | 16 | 25 | 48 | | Sampler | euler | dpmpp_2m | dpmpp_2m_sde | | Scheduler | normal | karras | karras |

Recommended Models for M4

• SDXL Base: sd_xl_base_1.0.safetensors — general purpose, excellent quality
• SDXL Refiner: sd_xl_refiner_1.0.safetensors — detail enhancement pass
• SVD XT 1.1: svd_xt_1_1.safetensors — image-to-video, 25 frames
• Flux: High quality, needs more VRAM but M4 128GB handles it
• AnimateDiff v2: mm_sd_v15_v2.ckpt — text-to-animation

Iteration Strategies

If output quality is insufficient:

1. Adjust CFG — too high = oversaturated, too low = unfocused
2. Increase steps — diminishing returns past 40
3. Change sampler — try dpmpp_2m_sde with karras scheduler
4. Refine with img2img — use generateVariation() with denoise 0.3-0.5
5. Add ControlNet — for pose/composition control (use raw workflow builder)
6. Fix seed — lock seed for consistent iterations, change prompt only

Error Handling

| Error | Cause | Fix | |-------|-------|-----| | Connection refused | ComfyUI not running | Start ComfyUI server | | Model not found | Checkpoint missing | Download model to ComfyUI/models/checkpoints/ | | Out of memory | Resolution too high | Reduce resolution or batch size | | Invalid graph | Node mismatch | Check node class_type names | | Timeout | Complex workflow | Increase pollWorkflow maxAttempts |

Utility Functions

import { checkHealth, getQueueStatus, listCheckpoints } from './src/comfyui.mjs';

// Check if ComfyUI is running
const health = await checkHealth();
// { online: true, system: {...}, devices: [...] }

// Check queue
const queue = await getQueueStatus();
// { running: 1, pending: 3 }

// List available models
const models = await listCheckpoints();
// ['sd_xl_base_1.0.safetensors', 'flux_dev.safetensors', ...]

Full Pipeline Example

import { generateProductAssets, generateVideo } from './src/comfyui.mjs';

// 1. Generate product images for all asset roles
const { assets } = await generateProductAssets({
  productName: 'Complete Lip Kit',
  productDescription: 'Luxury lip care set',
  style: 'editorial beauty photography',
  brandAesthetic: 'dark luxury, gold accents, premium',
  assetRoles: ['hero_product', 'lifestyle', 'ingredient'],
  width: 1024,
  height: 1024
});

// 2. Generate product reveal video from hero shot
const heroImage = assets.find(a => a.role === 'hero_product');
const video = await generateVideo({
  inputImage: heroImage.images[0].filename,
  frames: 25,
  fps: 8,
  motionBucketId: 80
});

// 3. Feed into video-ad-creator pipeline
const adRequest = {
  campaign: { /* from upstream */ },
  product: {
    images: assets.flatMap(a => a.images.map(i => i.url)),
    videos: video.videos.map(v => v.url)
  },
  brand: { /* from brand kit */ },
  copy: { /* from campaign planning */ }
};
// → Pass to video-ad-creator skill