stahura/domo-data-generator

Domo Sample Data Generator

Generate realistic, cross-referenced sample data for Domo using the datagen CLI.

Repository: https://github.com/brrink/domo_data_generator

---

Overview

The generator creates sample data mirroring major business platforms with consistent cross-source entity integrity, then uploads it to Domo. It includes:

• 18 pre-built datasets across 6 source categories (Salesforce, Google Analytics, Financial, Marketing, Health, AdPoint)
• YAML-driven catalog for easy dataset additions
• Shared entity pool (companies, people, products, sales reps, campaigns)
• Date rolling to keep data looking current
• Direct Domo integration (create datasets, upload, set connector icons)
• Structured JSON output by default (AI-agent-friendly)
• pipx installable -- runs from any directory

---

Setup

# Install globally with pipx (recommended)
pipx install git+https://github.com/brrink/domo_data_generator.git

# Initialize a working directory
mkdir my-domo-data && cd my-domo-data
datagen init

# Edit .env with your Domo credentials

If .env.example is missing or you want a clean start, create .env in the working directory with:

cat > .env <<'EOF'
DOMO_CLIENT_ID=your_client_id_here
DOMO_CLIENT_SECRET=your_client_secret_here
DOMO_API_HOST=api.domo.com
DOMO_INSTANCE=your_instance_name
DOMO_SET_CONNECTOR_TYPE=false
EOF

Required Environment Variables

| Variable | Purpose | |----------|---------| | DOMO_CLIENT_ID | OAuth client identifier | | DOMO_CLIENT_SECRET | OAuth client secret | | DOMO_API_HOST | API endpoint hostname | | DOMO_INSTANCE | Domo instance name | | DOMO_SET_CONNECTOR_TYPE | Enable connector icon customization (optional, default: false) |

Auth boundary note: domo_data_generator uses its own public-API/OAuth credential flow and does not run through community-domo-cli or ryuu session auth.

Current tooling boundary: most Product API automation should use community-domo-cli, but datagen dataset create/upload in this skill currently depends on python -m datagen with .env OAuth credentials (DOMO_CLIENT_ID / DOMO_CLIENT_SECRET).

---

CLI Reference

Entry point: datagen [OPTIONS] COMMAND [ARGS]

Global Options

| Option | Description | |--------|-------------| | --verbose / -v | Enable verbose logging | | --output / -o TEXT | Output format: json (default), table, yaml | | --yes / -y | Skip confirmation prompts |

All commands emit structured JSON by default for easy machine parsing.

Init Command

init -- Initialize a working directory

datagen init                  # Initialize current directory
datagen init /path/to/dir     # Initialize a specific directory

Copies bundled catalog YAML files to ./catalog/, creates .env template, and creates ./data/ directory. Run this once before using the CLI in a new directory.

Core Commands

generate -- Generate sample data

datagen generate --all                    # Generate all datasets
datagen generate salesforce_opportunities  # Generate one dataset
datagen generate --all --seed 42          # Reproducible generation
datagen generate --all --dry-run          # Preview without writing

Requires entity pool initialization first. Run python -m datagen pool regenerate before generate even if your schema has no explicit entity_ref columns.

| Option | Description | |--------|-------------| | name | Dataset name (YAML filename stem), optional | | --all | Generate all datasets | | --seed INTEGER | Random seed for reproducibility | | --catalog-dir PATH | Catalog directory override | | --data-dir PATH | Data directory override | | --dry-run | Preview without writing files |

upload -- Upload data to Domo (full replace)

datagen upload --all
datagen upload salesforce_opportunities

Requires DOMO_CLIENT_ID and DOMO_CLIENT_SECRET.

| Option | Description | |--------|-------------| | name | Dataset name, optional | | --all | Upload all datasets | | --catalog-dir PATH | Catalog directory override | | --data-dir PATH | Data directory override |

create-dataset -- Create dataset(s) in Domo from catalog

datagen create-dataset --all --skip-existing
datagen create-dataset salesforce_opportunities

Requires DOMO_CLIENT_ID and DOMO_CLIENT_SECRET. The domo_id is persisted locally (in the catalog YAML if writable, otherwise in data/domo_ids.json).

| Option | Description | |--------|-------------| | name | Dataset name, optional | | --all | Create all datasets | | --skip-existing | Skip datasets that already have a domo_id | | --catalog-dir PATH | Catalog directory override |

roll-dates -- Shift rolling date columns to stay current

datagen roll-dates
datagen roll-dates --anchor-date 2026-04-01

| Option | Description | |--------|-------------| | --anchor-date TEXT | Target date (YYYY-MM-DD), defaults to today | | --catalog-dir PATH | Catalog directory override | | --data-dir PATH | Data directory override |

Informational Commands

list -- List catalog dataset definitions

datagen list                    # JSON output (default)
datagen --output table list     # Rich table for humans
datagen list --verbose          # Include column/schema details

status -- Display generation status for all datasets

datagen status

Connector Icon Commands

Require DOMO_DEVELOPER_TOKEN and DOMO_INSTANCE.

discover-types -- Search Domo connector/provider types

datagen discover-types salesforce

set-type -- Set connector icon on a Domo dataset

datagen set-type salesforce_opportunities
datagen set-type salesforce_opportunities --provider-key custom_key

set-type-all -- Set connector icon on all datasets with a domo_id

datagen set-type-all

Entity Pool Commands

pool regenerate -- Regenerate the shared entity pool

datagen pool regenerate
datagen pool regenerate --seed 99
datagen pool regenerate --company-count 500 --person-count 1000

| Option | Default | |--------|---------| | --seed INTEGER | 42 | | --company-count INTEGER | 200 | | --person-count INTEGER | 500 | | --product-count INTEGER | 50 | | --sales-rep-count INTEGER | 20 | | --campaign-count INTEGER | 30 |

pool show -- Display entity pool summary

datagen pool show

---

Common Workflows

Full setup for a new Domo instance

datagen init
# Edit .env with credentials
datagen pool regenerate
datagen generate --all
datagen create-dataset --all
datagen upload --all
datagen set-type-all

Daily refresh via cron

# Crontab entry: roll dates and re-upload daily at 6 AM
0 6 * * * cd /path/to/project && datagen roll-dates && datagen upload --all

Generate a single dataset end-to-end

datagen generate salesforce_opportunities
datagen create-dataset salesforce_opportunities
datagen upload salesforce_opportunities
datagen set-type salesforce_opportunities

---

Included Datasets

| Category | Dataset Name | Key | Rows | |----------|-------------|-----|------| | Salesforce | Salesforce - Accounts | salesforce_accounts | 500 | | Salesforce | Salesforce - Contacts | salesforce_contacts | 1,500 | | Salesforce | Salesforce - Opportunities | salesforce_opportunities | 2,500 | | Google Analytics | Google Analytics - Sessions | ga_sessions | 5,000 | | Google Analytics | Google Analytics - Page Views | ga_pageviews | 10,000 | | Financial | QuickBooks - Invoices | financial_invoices | 3,000 | | Financial | NetSuite - General Ledger | financial_gl_entries | 5,000 | | Marketing | Google Ads - Campaign Performance | marketing_google_ads | 3,000 | | Marketing | Facebook Ads - Campaign Performance | marketing_facebook_ads | 2,500 | | Marketing | HubSpot - Contacts | marketing_hubspot_contacts | 2,000 | | Marketing | Marketing - Market Leads | marketing_market_leads | 2,500 | | Marketing | Marketo - Leads | marketing_marketo_leads | 3,000 | | Health | Health Portal - Demographics | health_demographics | 15 | | Health | Health Portal - Lab Results | health_lab_results | 1,470 | | Health | Health Portal - Vitals | health_vitals | 5,250 | | AdPoint | AdPoint - Orders | adpoint_orders | 150 | | AdPoint | AdPoint - Line Items | adpoint_line_items | 500 | | AdPoint | AdPoint - Flights | adpoint_flights | 2,000 |

---

Entity Pool

The shared entity pool provides consistent cross-dataset references. Entities are generated once and reused across all datasets.

| Entity Type | Default Count | Key Fields | |-------------|---------------|------------| | company | 200 | id, account_id, name, domain, industry, size, city, state, annual_revenue, employee_count | | person | 500 | id, contact_id, first_name, last_name, full_name, email, company_id, company_name, title, phone | | product | 50 | id, name, category, unit_price, sku | | sales_rep | 20 | id, rep_id, first_name, last_name, full_name, email, region | | campaign | 30 | id, name, channel, budget, status |

---

Adding New Dataset Definitions

Dataset definitions live in the catalog/ directory as YAML files. Each YAML file defines metadata, columns, and generator configurations.

YAML Structure

dataset:
  name: My Custom Dataset
  domo_id: null
  source_type: custom
  description: "Description of the dataset"
  row_count: 1000
  tags:
    - custom
    - demo

schema:
  - name: id
    type: STRING
    generator: uuid4

  - name: company_name
    type: STRING
    generator: entity_ref
    entity: company
    field: name

  - name: amount
    type: DOUBLE
    generator: random_decimal
    min: 100.0
    max: 10000.0
    precision: 2

  - name: created_date
    type: DATE
    generator: date_range
    start_days_ago: 365
    end_days_ahead: 0
    rolling: true

Available Column Types

STRING, LONG, DOUBLE, DECIMAL, DATETIME, DATE

Available Generators

Generic: uuid4, random_choice, weighted_choice, random_int, random_decimal, date_range, entity_ref, compound, sequence, constant, derived_from_date, stage_derived, faker

Salesforce: sf_id, sf_opportunity_name, sf_case_subject, sf_lead_rating

Google Analytics: ga_session_id, ga_page_path, ga_source, ga_medium, ga_campaign, ga_browser, ga_device_category, ga_country, ga_bounce_rate, ga_session_duration, ga_pageviews, ga_landing_page

Financial: gl_account_code, gl_account_name, invoice_number, payment_terms, payment_method, invoice_status, journal_type, department, fiscal_period, debit_credit

Marketing/Ads: ad_platform, campaign_objective, ad_format, ad_headline, ad_keyword, targeting_type, impressions, clicks_from_impressions, ctr, cost_per_click, ad_spend, conversions_from_clicks, hubspot_lifecycle, hubspot_lead_status, ad_group_id

Health: health_lab_init, health_lab_field, health_vital_init, health_vital_field, health_demographics

Generator Column Options

| Option | Used With | Description | |--------|-----------|-------------| | entity | entity_ref | Entity pool type to reference | | field | entity_ref | Field to pull from the entity | | choices | random_choice, weighted_choice | List of possible values | | min / max | random_int, random_decimal | Value range | | precision | random_decimal | Decimal places | | template | compound | String template with {field} placeholders | | refs | compound | Column references for template substitution — must be a YAML list of column name strings (e.g. ["sku", "line_id"]), not a dict/object. A dict triggers ValidationError: schema.N.refs — Input should be a valid list. | | start_days_ago / end_days_ahead | date_range | Date range relative to today | | rolling | date_range | Enable date rolling for freshness | | mapping | stage_derived | Map source values to derived values | | source_column | stage_derived, derived_from_date | Column to derive from | | format | derived_from_date | Date format string | | faker_method | faker | Faker library method name | | faker_args | faker | Arguments for the Faker method |

weighted_choice YAML format:

generator: weighted_choice
choices:
  "Tier 1": 0.40
  "Tier 2": 0.35
  "Tier 3": 0.25

compound refs vs formatted random strings: For values like PO-12345, prefer faker with bothify instead of abusing compound / refs:

- name: purchase_order_ref
  type: STRING
  generator: faker
  faker_method: bothify
  faker_args:
    text: "PO-#####"

---

Rules

1. Run datagen init first -- Initialize a working directory before using any other commands. This copies the catalog and creates .env.
2. Always generate before uploading -- Run generate (or generate --all) before upload to ensure CSV data files exist.
3. Create datasets before first upload -- Run create-dataset before upload for new datasets. The domo_id is persisted locally.
4. Use --skip-existing -- When running create-dataset --all, use --skip-existing to avoid duplicating datasets that already have a domo_id.
5. Entity pool consistency -- Regenerating the pool (pool regenerate) invalidates all previously generated data. Re-generate all datasets afterward.
6. Date rolling -- Use roll-dates before upload to keep date columns current. Only columns with rolling: true are affected.
7. Credentials -- DOMO_CLIENT_ID and DOMO_CLIENT_SECRET are required for upload and create-dataset. DOMO_DEVELOPER_TOKEN is required for set-type and discover-types. Offline commands need no credentials.
8. Reproducibility -- Use --seed for reproducible data generation across runs.
9. Output format -- Default output is JSON. Use --output table for human-readable Rich tables.

---

Checklist

• [ ] CLI installed (pipx install git+https://github.com/brrink/domo_data_generator.git)
• [ ] Working directory initialized (datagen init)
• [ ] .env configured with Domo credentials
• [ ] Entity pool generated (datagen pool regenerate)
• [ ] Datasets generated (datagen generate --all)
• [ ] Datasets created in Domo (datagen create-dataset --all --skip-existing)
• [ ] Data uploaded (datagen upload --all)
• [ ] Connector icons set (datagen set-type-all) if desired
• [ ] Cron configured for daily date rolling and upload if needed