narumiruna/kabigon

How to load URL content

Use the automatic path unless you are debugging or intentionally comparing Loaders. Kabigon matches the URL to a source-specific Pipeline, builds an Execution plan from Targeted loaders plus allowed Fallback loaders, and returns the first successful Loader result.

# CLI usage
uvx kabigon https://www.youtube.com/watch?v=dQw4w9WgXcQ
uvx kabigon https://x.com/howie_serious/status/1917768568135115147
uvx kabigon https://reddit.com/r/python/comments/xyz/...
uvx kabigon https://github.com/user/repo/blob/main/README.md
uvx kabigon https://example.com/document.pdf

import kabigon

text = kabigon.load_url_sync("https://www.youtube.com/watch?v=dQw4w9WgXcQ")

import asyncio
import kabigon

text = await kabigon.load_url("https://x.com/user/status/123")

# List supported loader names.
uvx kabigon --list

import kabigon

# Inspect the Pipeline, Targeted loaders, Execution plan, and requirements.
plan = kabigon.explain_plan("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
loaders = kabigon.available_loaders()

Advanced loader selection

Use --loader only when debugging, comparing Loaders, or intentionally bypassing automatic Pipeline planning. A comma-separated loader list runs in the exact order provided.

uvx kabigon --loader playwright https://example.com
uvx kabigon --loader httpx https://example.com
uvx kabigon --loader firecrawl https://example.com
uvx kabigon --loader bbc https://www.bbc.com/news/articles/example
uvx kabigon --loader cnn https://www.cnn.com/2025/01/01/world/example/index.html
uvx kabigon --loader youtube https://www.youtube.com/watch?v=dQw4w9WgXcQ
uvx kabigon --loader youtube-ytdlp https://www.youtube.com/watch?v=dQw4w9WgXcQ
uvx kabigon --loader ytdlp https://www.youtube.com/watch?v=dQw4w9WgXcQ
uvx kabigon --loader twitter https://x.com/howie_serious/status/1917768568135115147
uvx kabigon --loader truthsocial https://truthsocial.com/@realDonaldTrump/posts/115830428767897167
uvx kabigon --loader reddit https://reddit.com/r/confession/comments/1q1mzej/im_a_developer_for_a_major_food_delivery_app_the/
uvx kabigon --loader ptt https://www.ptt.cc/bbs/Gossiping/M.1746078381.A.FFC.html
uvx kabigon --loader reel https://www.instagram.com/reel/CuA0XYZ1234/
uvx kabigon --loader github https://github.com/anthropics/claude-code/blob/main/plugins/ralph-wiggum/README.md
uvx kabigon --loader pdf https://example.com/document.pdf

# Run loaders in explicit order. Example: try YouTube transcripts first,
# then fall back to `youtube-ytdlp` audio transcription if captions are missing.
uvx kabigon --loader youtube,youtube-ytdlp https://www.youtube.com/watch?v=dQw4w9WgXcQ

Supported sources

• YouTube videos: youtube, then youtube-ytdlp audio transcription when needed.
• Social posts: twitter, truthsocial, reddit, ptt, and reel.
• News articles: bbc and cnn article-aware extraction.
• Code and documents: github file/page content and pdf text extraction.
• Generic web pages: playwright, httpx, or firecrawl.
• Audio/video URLs: ytdlp generic audio transcription.

Failure behavior

• A Loader that cannot handle the input raises a not-applicable attempt so the Load chain can continue.
• Content extraction, timeout, and configuration failures are reported in the Load chain failure details.
• Prefer kabigon.explain_plan(url) when you need to understand why a URL will use a specific Pipeline or Loader order.

Configuration notes

• FIRECRAWL_API_KEY is required for the firecrawl loader.
• FFMPEG_PATH can point to a custom FFmpeg binary for Whisper and yt-dlp transcription loaders.

Troubleshooting

• Install uv if uvx is not found:

  https://docs.astral.sh/uv/getting-started/installation/