JosiahSiegel/fal-video-to-video

Quick Reference

| Task | Endpoint | Key Parameter | |------|----------|---------------| | Style/Edit | fal-ai/kling-video/o1/video-to-video/edit | edit_type | | Remix | fal-ai/sora/remix | prompt | | Upscale | fal-ai/video-upscaler | scale: 2,4 | | Interpolate | fal-ai/frame-interpolation | target_fps |

| Edit Type | Use Case | |-----------|----------| | general | Color, lighting, quality | | style | Anime, painting, noir | | object | Replace/remove elements |

| Style Keyword | Effect | |---------------|--------| | "anime style" | Japanese animation | | "oil painting" | Textured brushstrokes | | "noir" | High contrast B&W | | "cyberpunk" | Neon futuristic |

| Upscale | Size Impact | |---------|-------------| | 2x | 4x file size | | 4x | 16x file size |

When to Use This Skill

Use for video editing and transformation:

• Applying style transfer to videos
• Upscaling low-resolution videos
• Increasing frame rate for smooth motion
• Replacing or removing objects
• Enhancing video quality

Related skills:

• For text-to-video: see fal-text-to-video
• For image-to-video: see fal-image-to-video
• For model selection: see fal-model-guide

---

fal.ai Video-to-Video Models

Complete reference for video editing, transformation, and upscaling models on fal.ai.

Kling O1 Video Editing

Kling O1 Video-to-Video Edit

Endpoint: fal-ai/kling-video/o1/video-to-video/edit Best For: Comprehensive video editing and transformation

import { fal } from "@fal-ai/client";

const result = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
  input: {
    prompt: "Transform into an anime style with vibrant colors",
    video_url: "https://example.com/original-video.mp4",
    edit_type: "style",
    negative_prompt: "blurry, distorted, low quality"
  }
});

console.log(result.video.url);

import fal_client

result = fal_client.subscribe(
    "fal-ai/kling-video/o1/video-to-video/edit",
    arguments={
        "prompt": "Transform into anime style",
        "video_url": "https://example.com/video.mp4",
        "edit_type": "style"
    }
)
print(result["video"]["url"])

Edit Types:

| Type | Description | Use Case | |------|-------------|----------| | general | General purpose editing | Any transformation | | style | Style transfer | Artistic effects | | object | Object replacement/editing | Change specific elements |

Parameters:

| Parameter | Type | Description | |-----------|------|-------------| | prompt | string | Editing instruction | | video_url | string | Source video URL | | edit_type | string | "general", "style", "object" | | negative_prompt | string | What to avoid |

Kling O1 Examples

Style Transfer:

const result = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
  input: {
    prompt: "Transform into a watercolor painting style with soft brushstrokes",
    video_url: sourceVideoUrl,
    edit_type: "style",
    negative_prompt: "harsh edges, digital artifacts"
  }
});

Object Editing:

const result = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
  input: {
    prompt: "Change the red car to a blue sports car",
    video_url: sourceVideoUrl,
    edit_type: "object"
  }
});

General Transformation:

const result = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
  input: {
    prompt: "Make it look like a sunny summer day instead of overcast",
    video_url: sourceVideoUrl,
    edit_type: "general"
  }
});

Sora 2 Remix

Sora Remix (Video Transformation)

Endpoint: fal-ai/sora/remix Best For: Creative video remixing and transformation

const result = await fal.subscribe("fal-ai/sora/remix", {
  input: {
    prompt: "Transform this into a cinematic sci-fi scene with futuristic elements",
    video_url: "https://example.com/original.mp4",
    aspect_ratio: "16:9"
  }
});

console.log(result.video.url);

Sora Remix Parameters:

| Parameter | Type | Description | |-----------|------|-------------| | prompt | string | Transformation description | | video_url | string | Source video URL | | aspect_ratio | string | "16:9", "9:16", "1:1" |

Video Upscaling

Video Upscaler

Endpoint: fal-ai/video-upscaler Best For: Enhance video resolution

const result = await fal.subscribe("fal-ai/video-upscaler", {
  input: {
    video_url: "https://example.com/low-res-video.mp4",
    scale: 2  // 2x or 4x upscale
  }
});

console.log(result.video.url);

result = fal_client.subscribe(
    "fal-ai/video-upscaler",
    arguments={
        "video_url": "https://example.com/video.mp4",
        "scale": 2
    }
)

Parameters:

| Parameter | Type | Values | Description | |-----------|------|--------|-------------| | video_url | string | - | Source video URL | | scale | number | 2, 4 | Upscale factor |

TopazVideoAI (if available)

Endpoint: fal-ai/topaz-video-ai Best For: Professional video enhancement

const result = await fal.subscribe("fal-ai/topaz-video-ai", {
  input: {
    video_url: "https://example.com/video.mp4",
    enhancement_type: "upscale",
    target_resolution: "4k"
  }
});

Video Frame Interpolation

Frame Interpolation

Endpoint: fal-ai/frame-interpolation Best For: Smooth motion, increase frame rate

const result = await fal.subscribe("fal-ai/frame-interpolation", {
  input: {
    video_url: "https://example.com/24fps-video.mp4",
    target_fps: 60,
    interpolation_mode: "smooth"
  }
});

Parameters:

| Parameter | Type | Description | |-----------|------|-------------| | video_url | string | Source video | | target_fps | number | Target frame rate (30, 60, 120) | | interpolation_mode | string | "smooth", "natural" |

Video Style Transfer

General Style Transfer Pipeline

// Apply artistic style to video
async function applyVideoStyle(videoUrl: string, style: string) {
  const result = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
    input: {
      prompt: `Transform into ${style} style`,
      video_url: videoUrl,
      edit_type: "style"
    }
  });
  return result.video.url;
}

// Usage
const animeVideo = await applyVideoStyle(originalVideo, "anime");
const oilPaintingVideo = await applyVideoStyle(originalVideo, "oil painting");
const pixelArtVideo = await applyVideoStyle(originalVideo, "pixel art");

Style Keywords

| Style | Prompt Example | |-------|----------------| | Anime | "Japanese anime style, cel shading" | | Watercolor | "Soft watercolor painting with flowing colors" | | Oil Painting | "Classical oil painting, textured brushstrokes" | | Pixel Art | "8-bit pixel art retro game style" | | Noir | "Black and white film noir, high contrast" | | Cyberpunk | "Neon cyberpunk aesthetic, futuristic" | | Sketch | "Hand-drawn pencil sketch style" | | Comic | "Comic book style with bold outlines" |

Video Enhancement Techniques

Denoising

const result = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
  input: {
    prompt: "Clean and enhance video quality, reduce noise and grain",
    video_url: noisyVideoUrl,
    edit_type: "general",
    negative_prompt: "grainy, noisy, artifacts"
  }
});

Color Correction

const result = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
  input: {
    prompt: "Enhance colors to be more vibrant and natural, improve contrast",
    video_url: flatVideoUrl,
    edit_type: "general"
  }
});

Lighting Enhancement

const result = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
  input: {
    prompt: "Improve lighting, add warm golden hour glow",
    video_url: darkVideoUrl,
    edit_type: "general"
  }
});

Video Object Editing

Object Replacement

const result = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
  input: {
    prompt: "Replace the bicycle with a motorcycle",
    video_url: sourceVideoUrl,
    edit_type: "object"
  }
});

Object Removal

const result = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
  input: {
    prompt: "Remove the person in the background, fill with natural scenery",
    video_url: sourceVideoUrl,
    edit_type: "object"
  }
});

Object Addition

const result = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
  input: {
    prompt: "Add falling snow throughout the scene",
    video_url: sourceVideoUrl,
    edit_type: "object"
  }
});

Complete Parameter Reference

interface VideoToVideoInput {
  // Required
  video_url: string;

  // For editing models
  prompt?: string;
  edit_type?: "general" | "style" | "object";
  negative_prompt?: string;

  // For upscaling
  scale?: number;  // 2 or 4

  // For frame interpolation
  target_fps?: number;
  interpolation_mode?: string;

  // Output
  aspect_ratio?: string;
}

Workflow Examples

Complete Video Enhancement Pipeline

async function enhanceVideo(sourceUrl: string) {
  // 1. Upscale resolution
  const upscaled = await fal.subscribe("fal-ai/video-upscaler", {
    input: {
      video_url: sourceUrl,
      scale: 2
    }
  });

  // 2. Enhance quality
  const enhanced = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
    input: {
      prompt: "Enhance video quality, improve sharpness and colors",
      video_url: upscaled.video.url,
      edit_type: "general"
    }
  });

  // 3. Smooth motion (optional)
  const smoothed = await fal.subscribe("fal-ai/frame-interpolation", {
    input: {
      video_url: enhanced.video.url,
      target_fps: 60
    }
  });

  return smoothed.video.url;
}

Style Transfer Workflow

async function styleTransferVideo(
  sourceUrl: string,
  targetStyle: string
) {
  // Apply style transformation
  const styled = await fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
    input: {
      prompt: `Transform into ${targetStyle} style, maintain motion consistency`,
      video_url: sourceUrl,
      edit_type: "style",
      negative_prompt: "flickering, inconsistent, artifacts"
    }
  });

  return styled.video.url;
}

// Usage
const animeVersion = await styleTransferVideo(original, "anime");
const paintingVersion = await styleTransferVideo(original, "impressionist oil painting");

Batch Video Processing

async function processVideosBatch(
  videos: string[],
  transformation: string
) {
  const results = await Promise.all(
    videos.map(video =>
      fal.subscribe("fal-ai/kling-video/o1/video-to-video/edit", {
        input: {
          prompt: transformation,
          video_url: video,
          edit_type: "general"
        }
      })
    )
  );

  return results.map(r => r.video.url);
}

Model Comparison

| Model | Editing | Style | Upscale | Speed | |-------|---------|-------|---------|-------| | Kling O1 | Full | Full | No | Medium | | Sora Remix | Limited | Full | No | Slow | | Video Upscaler | No | No | Full | Fast |

Best Practices

For Style Transfer

1. Be consistent - Use specific style descriptions
2. Avoid flickering - Add "consistent" to negative prompt
3. Test on clips - Process short segments first
4. Maintain identity - Don't over-transform

For Object Editing

1. Be specific - Name exact objects to change
2. Provide context - Describe replacement clearly
3. Consider motion - Ensure edits work with movement
4. Check boundaries - Object edges should blend naturally

For Upscaling

1. Source quality matters - Clean input = better output
2. Don't over-upscale - 2x usually sufficient
3. Consider file size - Higher res = larger files
4. Test first - Preview with short clip

For General Enhancement

1. Incremental changes - Don't try to fix everything at once
2. Preserve intent - Keep original mood/style
3. Check artifacts - Look for introduced errors
4. Iterate - Multiple passes may be needed

Common Issues and Solutions

| Issue | Cause | Solution | |-------|-------|----------| | Flickering | Inconsistent frames | Add "consistent, stable" to prompt | | Color shifts | Over-aggressive editing | Use gentler prompts | | Lost details | Too much transformation | Reduce edit intensity | | Artifacts | Compression issues | Use higher quality source | | Motion blur | Frame interpolation | Use "natural" mode |

File Size Considerations

| Operation | Size Impact | |-----------|-------------| | 2x Upscale | 4x larger | | 4x Upscale | 16x larger | | Frame interpolation | 2-4x larger | | Style transfer | Similar size |

Always consider storage and bandwidth when processing videos.