openclaw/surreal-sync

Surreal-Sync -- Data Migration and Synchronization

Surreal-Sync is a CLI tool for migrating data from various database sources to SurrealDB with full and incremental synchronization via Change Data Capture (CDC).

Supported Sources

| Source | Full Sync | Incremental CDC | Method | |--------|-----------|----------------|--------| | MongoDB | Yes | Yes | Change streams | | MySQL | Yes | Yes | Trigger-based CDC + sequence checkpoints | | PostgreSQL (triggers) | Yes | Yes | Trigger-based CDC + sequence checkpoints | | PostgreSQL (wal2json) | Yes | Yes | Logical replication with wal2json plugin | | Neo4j | Yes | Yes | Timestamp-based tracking | | JSONL Files | Yes | N/A | Batch import from JSON Lines | | Apache Kafka | Yes | Yes | Consumer subscriptions with deduplication |

Quick Start

# Install surreal-sync (Rust binary)
cargo install surreal-sync

# Full sync from PostgreSQL (trigger-based)
surreal-sync from postgres trigger-full \
  --connection-string "postgresql://user:pass@localhost/mydb" \
  --surreal-endpoint "http://localhost:8000" \
  --surreal-username root \
  --surreal-password root \
  --to-namespace prod \
  --to-database main

# Incremental CDC from PostgreSQL (wal2json)
surreal-sync from postgres wal2json \
  --connection-string "postgresql://user:pass@localhost/mydb" \
  --surreal-endpoint "http://localhost:8000" \
  --surreal-username root \
  --surreal-password root \
  --to-namespace prod \
  --to-database main

# Full sync from MongoDB
surreal-sync from mongo full \
  --connection-string "mongodb://localhost:27017/mydb" \
  --surreal-endpoint "http://localhost:8000" \
  --surreal-username root \
  --surreal-password root \
  --to-namespace prod \
  --to-database main

# Batch import from JSONL
surreal-sync from jsonl import \
  --file data.jsonl \
  --surreal-endpoint "http://localhost:8000" \
  --surreal-username root \
  --surreal-password root \
  --to-namespace prod \
  --to-database main

# Consume from Kafka
surreal-sync from kafka consume \
  --bootstrap-servers "localhost:9092" \
  --topic my-events \
  --surreal-endpoint "http://localhost:8000" \
  --surreal-username root \
  --surreal-password root \
  --to-namespace prod \
  --to-database main

CLI Pattern

surreal-sync from <SOURCE> <COMMAND> \
  --connection-string [CONNECTION STRING] \
  --surreal-endpoint [SURREAL ENDPOINT] \
  --surreal-username [SURREAL USERNAME] \
  --surreal-password [SURREAL PASSWORD] \
  --to-namespace <NS> \
  --to-database <DB>

Key Features

• Automatic schema inference and SurrealDB table creation
• Record ID mapping from source primary keys
• Relationship extraction and graph edge creation
• Configurable batch sizes and parallelism
• Resumable sync with checkpoint tracking
• Deduplication for Kafka consumers

Full Documentation

See the main skill's rule file for complete guidance:

• rules/surreal-sync.md -- source configuration, schema mapping, CDC setup, conflict resolution, and production deployment
• surrealdb/surreal-sync -- upstream repository