OpenAEC-Foundation/frappe-impl-integrations

Frappe Integrations

Step-by-step workflows for OAuth, Webhooks, Payment Gateways, Data Import/Export, and external API calls.

Version: v14/v15/v16

---

Decision Tree: Which Integration Pattern?

WHAT ARE YOU INTEGRATING?
│
├─► External service needs to call YOUR Frappe site?
│   ├─► On document events → Webhook (push to external)
│   ├─► External sends data to you → Whitelisted API endpoint
│   └─► External needs user auth → OAuth 2.0 Provider
│
├─► YOUR Frappe site calls an external service?
│   ├─► Needs user-level OAuth consent → Connected App
│   ├─► Server-to-server with API key → make_request / requests
│   └─► Recurring sync → Scheduler + API calls
│
├─► Bulk data in/out?
│   ├─► Import CSV/XLSX → Data Import DocType
│   ├─► Export data → Report Builder / export-csv / API
│   └─► Programmatic bulk → frappe.get_doc().insert()
│
├─► Payment processing?
│   └─► Payment Request + Payment Gateway controller
│
└─► Real-time vs batch?
    ├─► Real-time → Webhook or API endpoint
    ├─► Near real-time → frappe.enqueue() after event
    └─► Batch → Scheduler task (hourly/daily)

---

Workflow 1: OAuth 2.0: Frappe as Provider

Use when external applications need "Sign in with Frappe" or API access on behalf of users.

Step 1: Configure OAuth Provider Settings

Navigate to Setup > Integrations > OAuth Provider Settings:

• Force: ALWAYS asks user for confirmation
• Auto: Asks only if no active token exists

Step 2: Create OAuth Client

Navigate to Setup > Integrations > OAuth Client:

| Field | Value | |-------|-------| | App Name | External app identifier | | Scopes | Space-separated (e.g., openid all) | | Redirect URIs | Space-separated callback URLs | | Default Redirect URI | Primary callback URL | | Grant Type | Authorization Code (RECOMMENDED) or Implicit | | Response Type | Code (for Auth Code) or Token (for Implicit) | | Skip Authorization | Check for trusted first-party apps only |

Step 3: Use the Generated Endpoints

| Endpoint | URL | |----------|-----| | Authorize | /api/method/frappe.integrations.oauth2.authorize | | Token | /api/method/frappe.integrations.oauth2.get_token | | Profile | /api/method/frappe.integrations.oauth2.openid_profile |

Step 4: Configure External App

# Example: Grafana generic_oauth config
client_id = <generated_client_id>
client_secret = <generated_client_secret>
auth_url = https://your-frappe.com/api/method/frappe.integrations.oauth2.authorize
token_url = https://your-frappe.com/api/method/frappe.integrations.oauth2.get_token
api_url = https://your-frappe.com/api/method/frappe.integrations.oauth2.openid_profile
scopes = openid all

Critical Rules

• NEVER use Implicit grant type for server-side apps — use Authorization Code
• ALWAYS use HTTPS in production for all OAuth endpoints
• NEVER expose client_secret in client-side JavaScript

---

Workflow 2: Connected App: Frappe as OAuth Consumer

Use when your Frappe instance needs to access external services (Google, Microsoft, etc.) on behalf of users.

Step 1: Create Connected App DocType

| Field | Purpose | |-------|---------| | Name | Identifier for the connection | | OpenID Configuration URL | Auto-fetches endpoints (e.g., /.well-known/openid-configuration) | | Authorization URI | Consent screen URL (auto-filled from OpenID) | | Token URI | Token exchange URL (auto-filled from OpenID) | | Redirect URI | Auto-generated — copy this to external provider | | Client ID | From external provider | | Client Secret | From external provider | | Scopes | Permissions needed (e.g., https://mail.google.com/) |

Step 2: Register Redirect URI with Provider

Copy the auto-generated Redirect URI and register it in the external provider's OAuth console.

Step 3: Add Extra Parameters (if needed)

access_type=offline    # Google: enables refresh tokens
prompt=consent         # Google: forces re-consent for refresh token

Step 4: Use in Code

import frappe

connected_app = frappe.get_doc("Connected App", "My Google App")
# Initiates OAuth flow — user clicks "Connect to..." button
# After consent, tokens are stored automatically

# Making authenticated calls:
session = connected_app.get_oauth2_session()
response = session.get("https://www.googleapis.com/gmail/v1/users/me/messages")

Critical Rules

• ALWAYS add access_type=offline for Google APIs to get refresh tokens
• NEVER store tokens manually — Connected App manages token lifecycle
• ALWAYS handle TokenExpiredError — call session.refresh_token() or reconnect

---

Workflow 3: Webhooks: Push Notifications to External Services

Step 1: Create Webhook DocType

Navigate to Integrations > Webhook:

| Field | Value | |-------|-------| | DocType | Target document type | | Doc Event | on_update, after_insert, on_submit, on_cancel, on_trash | | Request URL | External endpoint | | Request Method | POST (default) | | Conditions | Optional Jinja filter (e.g., doc.status == "Approved") | | Enabled | Check to activate |

Step 2: Configure Headers

Add custom headers for authentication:

Authorization: Bearer <api_token>
Content-Type: application/json

Step 3: Configure Data: Choose Format

Form URL-encoded: Select specific fields from a table.

JSON: Use Jinja templates for structured payloads:

{
  "id": "{{ doc.name }}",
  "total": "{{ doc.grand_total }}",
  "items": {{ doc.items | tojson }},
  "event": "{{ event }}"
}

Step 4: Enable Webhook Secret (HMAC Verification)

Set a Webhook Secret — Frappe adds X-Frappe-Webhook-Signature header with base64-encoded HMAC-SHA256 hash of the payload.

Receiver verification (Python example):

import hmac, hashlib, base64

def verify_webhook(payload_body, secret, signature_header):
    expected = base64.b64encode(
        hmac.new(secret.encode(), payload_body, hashlib.sha256).digest()
    ).decode()
    return hmac.compare_digest(expected, signature_header)

Critical Rules

• ALWAYS enable Webhook Secret for production webhooks
• NEVER rely on webhooks for guaranteed delivery — implement idempotency on the receiver
• ALWAYS use | tojson filter for child table data in JSON payloads
• Webhook logs are created for every delivery — check Webhook Request Log for debugging

---

Workflow 4: External API Calls from Frappe

Using frappe.integrations.utils

from frappe.integrations.utils import make_get_request, make_post_request

# GET request
response = make_get_request(
    "https://api.example.com/data",
    headers={"Authorization": "Bearer token123"}
)

# POST request
response = make_post_request(
    "https://api.example.com/submit",
    data={"key": "value"},
    headers={"Content-Type": "application/json"}
)

Using requests Library Directly

import requests
import frappe

def sync_to_external():
    try:
        response = requests.post(
            "https://api.example.com/endpoint",
            json={"data": "value"},
            timeout=30
        )
        response.raise_for_status()
        return response.json()
    except requests.exceptions.RequestException as e:
        frappe.log_error(f"API call failed: {e}", "Integration Error")
        raise

Critical Rules

• ALWAYS set a timeout on external requests (30s recommended)
• ALWAYS wrap external calls in try/except and log errors with frappe.log_error()
• NEVER call external APIs inside validate or before_save — use on_update + frappe.enqueue()
• ALWAYS use frappe.enqueue() for slow external calls to avoid blocking the web request

---

Workflow 5: Data Import

Via UI (Data Import DocType)

1. Navigate to Home > Data Import > New
2. Select DocType and Import Type (Insert or Update)
3. Download template CSV/XLSX
4. Fill in data following the template format
5. Upload and preview
6. Start Import

CSV Format Rules

ID,Item Name,Item Group,Stock UOM
,Widget A,Products,Nos
,Widget B,Raw Material,Kg

• First row: field labels or API field names
• Leave ID/name empty for Insert (auto-generated)
• For Update: ID column MUST contain existing document names
• Child tables: repeat parent row data, add child fields as extra columns

Programmatic Import

import frappe
from frappe.core.doctype.data_import.data_import import DataImport

# Create Data Import document
di = frappe.get_doc({
    "doctype": "Data Import",
    "reference_doctype": "Item",
    "import_type": "Insert New Records",
    "import_file": "/path/to/file.csv"
})
di.insert()
di.start_import()

Critical Rules

• ALWAYS download and use the template — column order and names must match exactly
• NEVER import more than 5,000 rows at once — split into batches
• ALWAYS test with 5-10 rows first before bulk import
• ALWAYS check Import Log for row-level errors after import completes

---

Workflow 6: Data Export

Via Report Builder

1. Open any DocType list view
2. Apply filters
3. Menu > Export (CSV/Excel)

Via CLI

bench --site mysite export-csv "Sales Invoice"
bench --site mysite export-doc "Sales Invoice" "INV-001"
bench --site mysite export-json "Sales Invoice" "INV-001"
bench --site mysite export-fixtures --app myapp

Programmatic Export

import frappe

# Export filtered data
data = frappe.get_all("Sales Invoice",
    filters={"status": "Paid", "posting_date": [">", "2024-01-01"]},
    fields=["name", "customer", "grand_total", "posting_date"],
    order_by="posting_date desc",
    limit_page_length=0  # No limit
)

# Convert to CSV
import csv, io
output = io.StringIO()
writer = csv.DictWriter(output, fieldnames=["name", "customer", "grand_total", "posting_date"])
writer.writeheader()
writer.writerows(data)
csv_content = output.getvalue()

---

Workflow 7: Frappe REST API Authentication

API Key + Secret (Server-to-Server)

# Generate via User > API Access > Generate Keys
curl -H "Authorization: token api_key:api_secret" \
  https://your-site.com/api/resource/Sales%20Invoice

OAuth Bearer Token

curl -H "Authorization: Bearer access_token" \
  https://your-site.com/api/resource/Sales%20Invoice

Session-Based (Login)

# Login first
curl -X POST https://your-site.com/api/method/login \
  -d "[email protected]&pwd=password"
# Subsequent requests use session cookie

---

Integration Patterns: Sync vs Async

| Pattern | When to Use | Implementation | |---------|-------------|----------------| | Synchronous | Response needed immediately | Direct API call in controller | | Async (enqueue) | External call > 5s | frappe.enqueue("myapp.api.sync_record", doc_name=doc.name) | | Webhook | Push on event | Webhook DocType configuration | | Scheduled sync | Periodic batch | scheduler_events in hooks.py | | Real-time | Live updates | Socket.IO + frappe.publish_realtime() |

Retry Pattern

import frappe
from frappe.utils.background_jobs import get_jobs

def sync_with_retry(doc_name, retry_count=0, max_retries=3):
    try:
        result = call_external_api(doc_name)
        frappe.db.set_value("Sales Invoice", doc_name, "sync_status", "Success")
        frappe.db.commit()
    except Exception as e:
        if retry_count < max_retries:
            frappe.enqueue(
                "myapp.integrations.sync_with_retry",
                doc_name=doc_name,
                retry_count=retry_count + 1,
                queue="short",
                enqueue_after_commit=True
            )
        else:
            frappe.log_error(f"Sync failed after {max_retries} retries: {e}")
            frappe.db.set_value("Sales Invoice", doc_name, "sync_status", "Failed")
            frappe.db.commit()

---

Version Differences

| Feature | V14 | V15 | V16 | |---------|:---:|:---:|:---:| | Webhook DocType | Yes | Yes | Yes | | Connected App | Yes | Yes | Yes | | OAuth 2.0 Provider | Yes | Yes | Yes | | Data Import (new UI) | Yes | Yes | Yes | | Print Designer | No | Yes | Yes | | make_get_request | Yes | Yes | Yes | | Webhook HMAC | Yes | Yes | Yes |

---

Reference Files

| File | Contents | |------|----------| | workflows.md | Complete integration workflow patterns | | examples.md | Working code examples for all integration types | | anti-patterns.md | Common integration mistakes and fixes | | decision-tree.md | Extended decision trees for integration choice |