enuno/langsmith-agent-builder

LangSmith Agent Builder

LangSmith Agent Builder enables users to create helpful AI agents without code. Start from a template, connect your accounts, and let the agent handle routine work while you stay in control.

When to Use

• Creating AI agents without writing code
• Automating email management and drafting
• Building Slack-integrated AI assistants
• Connecting agents to Google Calendar, Gmail, GitHub, Linear
• Setting up approval workflows for agent actions
• Deploying agents to Slack workspaces
• Invoking agents programmatically via Python/JavaScript
• Integrating custom tools via MCP servers
• Building multi-agent workflows with sub-agents

Core Concepts

Agents

Autonomous AI assistants that can access tools, perform tasks, and interact with external services on your behalf.

Templates

Pre-built agent configurations for common use cases (email assistant, team updates) that can be customized.

Tools

Built-in integrations with services like Gmail, Slack, GitHub, Linear, and more. Agents use tools to perform actions.

Triggers

Automation rules that invoke agents based on events (new email, Slack message, schedule).

Sub-agents

Specialized agents that can be called by a parent agent for complex multi-step workflows.

Approval Workflows

Human-in-the-loop checkpoints where agents pause for user confirmation before taking sensitive actions.

---

Getting Started

Prerequisites

• LangSmith account (smith.langchain.com)
• Gmail/Google account (for email/calendar tools)
• OpenAI or Anthropic API key

Step 1: Obtain API Key

OpenAI:

1. Visit platform.openai.com/api-keys
2. Create a new secret key
3. Name it "Agent Builder"
4. Save securely (begins with sk-)

Anthropic:

1. Go to console.anthropic.com/settings/keys
2. Create a new key
3. Store safely (starts with sk-ant-)

Step 2: Configure API Key in LangSmith

1. Navigate to smith.langchain.com
2. Go to Settings → Secrets tab
3. Click "Add secret"
4. Enter key name: OPENAI_API_KEY or ANTHROPIC_API_KEY
5. Paste your API key
6. Save

Step 3: Create Your Agent

1. Switch to Agent Builder in LangSmith
2. Select Templates from navigation
3. Choose a template (e.g., Email Assistant)
4. Click "Use this template"
5. Authorize your accounts when prompted

Step 4: Test and Configure

In the Test Chat panel:

"Apply a 'Review' label to emails requiring my attention."

Monitor proposed actions and approve each step using Continue.

---

Built-in Tools

Communication & Collaboration

| Tool | Capabilities | |------|-------------| | Gmail | Read/send emails, create drafts, manage labels | | Slack | Send DMs, post to channels, read message history | | LinkedIn | Publish posts with images or links |

Productivity & Planning

| Tool | Capabilities | |------|-------------| | Google Calendar | List events, get details, create new events | | Google Sheets | Create spreadsheets, read ranges | | Linear | Manage issues, list teams, create/update/delete issues | | Pylon | List and update issues |

Data & Analytics

| Tool | Capabilities | |------|-------------| | BigQuery | Execute SQL queries |

Code & Development

| Tool | Capabilities | |------|-------------| | GitHub | Manage PRs/issues, create PRs, comment, read repo files |

Content Discovery

| Tool | Capabilities | |------|-------------| | Exa | Web search with content fetching, LinkedIn profile search | | Tavily | Web search functionality |

General Utilities

| Tool | Capabilities | |------|-------------| | Webpage Reader | Read webpage text content | | Image Extractor | Extract image URLs and metadata | | Notifications | Send user confirmations |

---

Authentication Methods

OAuth-Based Tools

These tools use OAuth for secure authentication:

• Google (Gmail, Calendar, Sheets)
• Slack
• Linear
• GitHub
• LinkedIn

When adding these tools, you'll be prompted to authorize via OAuth flow.

API Key-Based Tools

These tools require workspace secrets (API keys):

| Tool | Secret Name | |------|-------------| | Exa Search | EXA_API_KEY | | Tavily Search | TAVILY_API_KEY | | Twitter/X | TWITTER_API_KEY, TWITTER_API_KEY_SECRET |

---

Workspace Setup

Required Secrets

Configure at least one LLM API key:

OPENAI_API_KEY=sk-...
# OR
ANTHROPIC_API_KEY=sk-ant-...

Agent Builder-Specific Keys (optional, for better tracking):

AGENT_BUILDER_OPENAI_API_KEY=sk-...
AGENT_BUILDER_ANTHROPIC_API_KEY=sk-ant-...

These take precedence over standard workspace secrets.

Setting Up Secrets

1. Navigate to Settings in LangSmith
2. Select the Secrets tab
3. Click "Add secret"
4. Enter the key name (e.g., OPENAI_API_KEY)
5. Paste the API key value
6. Save

Optional Tool Secrets

EXA_API_KEY=...
TAVILY_API_KEY=...
TWITTER_API_KEY=...
TWITTER_API_KEY_SECRET=...

---

Agent Configuration

Adding Tools

1. Open your agent in edit mode
2. Click "+ Add tool"
3. Select from available integrations:
   • Gmail, Slack, GitHub, Linear, etc.
4. Complete OAuth authorization or enter API keys
5. Configure tool-specific settings

Creating Sub-agents

1. Click "+ Add sub-agent"
2. Configure the sub-agent's:
   • Name and description
   • Available tools
   • Custom instructions
3. Reference sub-agent in parent agent's prompt

Configuring Approval Pauses

For sensitive operations:

1. Edit agent configuration
2. Enable "Request approval" for specific tools
3. Agent will pause and notify before executing

Modifying Tool Settings

1. Navigate to agent settings
2. Locate the integration
3. Adjust configuration or click "Disconnect" to remove

---

Triggers and Automation

Available Trigger Types

| Trigger | Description | |---------|-------------| | Email | Activate on new emails matching criteria | | Slack | Respond to messages in specific channels | | Schedule | Run at specified times (cron-like) |

Setting Up Triggers

1. Open agent configuration
2. Navigate to Triggers section
3. Select trigger type
4. Configure conditions:
   • Email: sender, subject filters
   • Slack: channel, mention requirements
   • Schedule: frequency, time

---

Calling Agents from Code

Prerequisites

• LangSmith account with Agent Builder agent
• Personal Access Token (PAT)
• LangGraph SDK

Installation

Python:

pip install langgraph-sdk python-dotenv

TypeScript:

npm install @langchain/langgraph-sdk

Configuration

Create .env file:

LANGSMITH_API_KEY=lsv2_pt_...
AGENT_ID=your-agent-id
AGENT_URL=https://your-agent-url

Python Example

from langgraph_sdk import get_client
from dotenv import load_dotenv
import os

load_dotenv()

# Initialize client
client = get_client(
    url=os.getenv("AGENT_URL"),
    api_key=os.getenv("LANGSMITH_API_KEY")
)

# Get agent info
agent = await client.assistants.get(os.getenv("AGENT_ID"))
print(f"Agent: {agent['name']}")

# Create thread and invoke
thread = await client.threads.create()

response = await client.runs.create(
    thread_id=thread["thread_id"],
    assistant_id=os.getenv("AGENT_ID"),
    input={
        "messages": [
            {"role": "user", "content": "Summarize my unread emails"}
        ]
    }
)

# Stream response
async for chunk in client.runs.stream(
    thread_id=thread["thread_id"],
    assistant_id=os.getenv("AGENT_ID"),
    input={"messages": [{"role": "user", "content": "What's on my calendar today?"}]}
):
    print(chunk)

TypeScript Example

import { Client } from "@langchain/langgraph-sdk";
import dotenv from "dotenv";

dotenv.config();

const client = new Client({
  apiUrl: process.env.AGENT_URL,
  apiKey: process.env.LANGSMITH_API_KEY,
});

// Get agent
const agent = await client.assistants.get(process.env.AGENT_ID);
console.log(`Agent: ${agent.name}`);

// Create thread
const thread = await client.threads.create();

// Invoke agent
const response = await client.runs.create(
  thread.thread_id,
  process.env.AGENT_ID,
  {
    input: {
      messages: [
        { role: "user", content: "Check my GitHub notifications" }
      ]
    }
  }
);

Authentication Headers

When using raw API:

X-API-Key: lsv2_pt_...
X-Auth-Scheme: langsmith-api-key

---

MCP Server Integration

Overview

Agent Builder integrates with Model Context Protocol (MCP) servers for custom tool access beyond built-in options.

Configuring MCP Servers

1. Navigate to workspace settings
2. Add MCP server configuration
3. Configure authentication headers if required
4. System auto-discovers available tools

Using MCP Tools

Once configured, MCP tools appear in the tool selection panel. Agent Builder:

• Automatically discovers available tools
• Applies configured auth headers during invocation
• Handles tool responses like built-in tools

---

Slack Integration

Setting Up Slack App

1. Navigate to Agent Builder settings
2. Select Slack integration
3. Click "Add to Slack"
4. Authorize in your Slack workspace
5. Configure allowed channels

Using Agents in Slack

Once integrated:

• Mention the agent in channels: @AgentName help me with...
• Send DMs directly to the agent
• Agent can post updates and notifications

Slack-Specific Features

• Read message history for context
• Post to specific channels
• Send direct messages
• React to messages (limited)

---

Agent Visibility

Private Agents

• Only visible to creator
• Personal use and testing
• Cannot be shared with workspace

Workspace Agents

• Visible to all workspace members
• Can be invoked by team members
• Read-only access for non-owners
• Shared tool configurations

Changing Visibility

1. Open agent settings
2. Navigate to Visibility section
3. Toggle between Private and Workspace
4. Save changes

---

Templates

Available Templates

| Template | Description | |----------|-------------| | Email Assistant | Manage inbox, draft responses, apply labels | | Team Updates | Aggregate updates from Linear, GitHub, Slack | | Research Assistant | Search web, summarize findings | | Calendar Manager | Schedule meetings, check availability |

Using Templates

1. Browse templates in Agent Builder
2. Click "Use this template"
3. Authorize required services
4. Customize prompts and settings
5. Test in chat panel

Customizing Templates

After selecting a template:

• Modify agent instructions/prompt
• Add or remove tools
• Configure triggers
• Adjust approval requirements
• Add sub-agents

---

Best Practices

Security

• Store API keys in workspace secrets, not prompts
• Enable approval pauses for destructive actions
• Use Agent Builder-specific keys for usage tracking
• Regularly rotate API keys

Agent Design

• Start with templates, customize incrementally
• Use clear, specific prompts
• Test thoroughly before deploying triggers
• Implement approval workflows for sensitive operations

Performance

• Minimize unnecessary tool calls in prompts
• Use sub-agents for specialized tasks
• Configure appropriate trigger conditions
• Monitor agent runs in LangSmith

Debugging

• Use Test Chat panel for interactive debugging
• Review tool call logs in LangSmith
• Check workspace secrets configuration
• Verify OAuth connections are active

---

Troubleshooting

"API key not found"

Ensure workspace secret is configured:
Settings → Secrets → Add OPENAI_API_KEY or ANTHROPIC_API_KEY

OAuth Tool Not Working

1. Go to agent settings
2. Click "Disconnect" on the tool
3. Re-authorize via OAuth flow
4. Test tool access

Agent Not Responding to Triggers

1. Verify trigger conditions
2. Check workspace secrets
3. Ensure tool authorizations are active
4. Review trigger logs in LangSmith

MCP Tools Not Available

1. Check MCP server configuration
2. Verify authentication headers
3. Ensure MCP server is running
4. Check tool discovery logs

---

Pricing and Limits

Agent Builder usage is tied to your LangSmith plan:

• Free tier: Limited agent runs
• Plus tier: Increased limits
• Enterprise: Custom limits

Check docs.langchain.com/langsmith/pricing for current details.

---

Resources

• Documentation: https://docs.langchain.com/langsmith/agent-builder
• LangSmith: https://smith.langchain.com
• LangGraph SDK (Python): https://pypi.org/project/langgraph-sdk/
• LangGraph SDK (JS): https://www.npmjs.com/package/@langchain/langgraph-sdk
• MCP Framework: https://modelcontextprotocol.io

---

Example Workflows

Email Management Agent

Agent: Email Assistant
Tools: Gmail
Triggers: New email in inbox

Prompt:
"You are an email assistant. For each new email:
1. Categorize by priority (urgent, normal, low)
2. Apply appropriate labels
3. Draft responses for urgent items
4. Summarize action items daily"

GitHub PR Reviewer

Agent: PR Review Assistant
Tools: GitHub, Slack
Triggers: New PR opened

Prompt:
"When a new PR is opened:
1. Read the PR description and changed files
2. Summarize the changes
3. Check for common issues
4. Post summary to #dev-updates Slack channel
5. If changes look significant, request approval before commenting"

Daily Standup Aggregator

Agent: Standup Bot
Tools: Linear, GitHub, Slack
Triggers: Daily at 9 AM

Prompt:
"Every morning:
1. Fetch yesterday's completed Linear issues
2. Get merged PRs from GitHub
3. Compile into a standup summary
4. Post to #standup Slack channel"