anhvth/llm-integration

LLM Integration Guide

This skill provides comprehensive guidance for using the LLM utilities in speedy_utils.

When to Use This Skill

Use this skill when you need to:

• Make OpenAI API calls with automatic caching (memoization) to save costs and time.
• Transform chat messages between different formats (ChatML, ShareGPT, Text).
• Prepare prompts for local LLM inference.

Prerequisites

• speedy_utils installed.
• openai package installed for API clients.

Core Capabilities

Memoized OpenAI Clients (MOpenAI, MAsyncOpenAI)

• Drop-in replacements for OpenAI and AsyncOpenAI.
• Automatically caches post (chat completion) requests.
• Uses speedy_utils caching backend (disk/memory).
• Configurable per-instance caching.

Chat Format Transformation (transform_messages)

• Converts between:
  • chatml: List of {"role": "...", "content": "..."} dicts.
  • sharegpt: Dict with {"conversations": [{"from": "...", "value": "..."}]}.
  • text: String with <|im_start|> tokens.
  • simulated_chat: Human/AI transcript format.
• Supports applying tokenizer templates.

Usage Examples

Example 1: Memoized OpenAI Call

Make repeated calls without hitting the API twice.

from llm_utils.lm.openai_memoize import MOpenAI

# Initialize just like OpenAI client
client = MOpenAI(api_key="sk-...")

# First call hits the API
response1 = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello"}]
)

# Second call returns cached result instantly
response2 = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello"}]
)

Example 2: Async Memoized Call

Same as above but for async workflows.

from llm_utils.lm.openai_memoize import MAsyncOpenAI
import asyncio

async def main():
    client = MAsyncOpenAI(api_key="sk-...")
    response = await client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": "Hi"}]
    )

Example 3: Transforming Chat Formats

Convert ShareGPT format to ChatML.

from llm_utils.chat_format.transform import transform_messages

sharegpt_data = {
    "conversations": [
        {"from": "human", "value": "Hi"},
        {"from": "gpt", "value": "Hello there"}
    ]
}

# Convert to ChatML list
chatml_data = transform_messages(sharegpt_data, frm="sharegpt", to="chatml")
# Result: [{'role': 'user', 'content': 'Hi'}, {'role': 'assistant', 'content': 'Hello there'}]

# Convert to Text string
text_data = transform_messages(chatml_data, frm="chatml", to="text")
# Result: "<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\nHello there<|im_end|>\n<|im_start|>assistant\n"

Guidelines

1. Caching Behavior:

   • The cache key is generated from the arguments passed to create.
   • If you change any parameter (e.g., temperature, model), it counts as a new request.
   • Cache is persistent if configured (default behavior of memoize).

2. Format Detection:

   • transform_messages tries to auto-detect input format, but it's safer to specify frm explicitly.

3. Tokenizer Support:

   • You can pass a HuggingFace tokenizer to transform_messages to use its specific chat template.

Limitations

• Streaming: Memoization does NOT work with streaming responses (stream=True).
• Side Effects: If your LLM calls rely on randomness (high temperature) and you want different results each time, disable caching or change the seed/input.