alexeyban/docker-databricks-lab-ops

Docker Databricks Lab Ops

Overview

Use this skill for the operational loop of this repository: bring up Docker services, generate source-table mutations, reset Databricks tables, execute a Databricks notebook or job, and verify whether the notebook run finished successfully.

Prefer the bundled scripts over rewriting shell commands. They encode the repository-specific paths and the expected sequence.

Workflow

1. Inspect the repo inputs first

• Confirm docker-compose.yml, postgres-connector.json, and the target notebook paths exist.
• Read references/repo-workflow.md if you need the repo-specific sequence or parameters.

2. Bring up the local CDC stack

• Use scripts/start_stack.sh.
• This runs docker compose up -d from the repository root.
• If the user asked for verification, follow with docker compose ps or service-specific health checks.
• If Kafka must be reachable from Databricks through an internal network boundary, first use scripts/prepare_ngrok_kafka.py and then start Compose with the discovered KAFKA_EXTERNAL_HOST and KAFKA_EXTERNAL_PORT.

3. Register Debezium connector if CDC ingestion needs to be exercised

• Use scripts/register_connector.sh.
• Only do this after Kafka Connect is accepting requests.
• If the connector already exists, report that clearly instead of treating it as a fatal failure.

4. Run load generators

• Use scripts/run_generators.sh.
• The first argument is film update iterations, the second is rental/payment iterations.
• Prefer bounded runs for verification by passing ITERATIONS; avoid indefinite generators unless the user asked for sustained load.

Example:

skills/docker-databricks-lab-ops/scripts/run_generators.sh 20 40

This runs 20 film mutations and 40 rental/payment mutations.

5. Reset Databricks tables (when starting fresh)

• Use scripts/reset_databricks_tables.py to drop all Bronze/Silver/Gold tables and clear streaming checkpoints before a fresh load.
• Requires --cluster-id or will submit via git source.
• Use --dry-run to preview what would be dropped without dropping.

python3 skills/docker-databricks-lab-ops/scripts/reset_databricks_tables.py \
  --cluster-id <cluster-id> \
  --catalog workspace

# preview only:
python3 skills/docker-databricks-lab-ops/scripts/reset_databricks_tables.py \
  --cluster-id <cluster-id> \
  --dry-run

6. Trigger a Databricks notebook job

• Use scripts/run_databricks_notebook.py.
• Provide either:
  • --job-id to run an existing Databricks job, or
  • --notebook-path and --cluster-id to submit a one-off notebook run.
• For dynamic Kafka exposure, pass --notebook-param KAFKA_BOOTSTRAP=<ngrok-host:port> so the current tunnel endpoint is used at run time instead of a stale static value.
• This script uses DATABRICKS_HOST and DATABRICKS_TOKEN from the environment.

Examples:

python3 skills/docker-databricks-lab-ops/scripts/run_databricks_notebook.py \
  --job-id 123 \
  --notebook-param KAFKA_BOOTSTRAP=0.tcp.eu.ngrok.io:12345

python3 skills/docker-databricks-lab-ops/scripts/run_databricks_notebook.py \
  --notebook-path /Workspace/agent/notebook \
  --cluster-id 0123-456abc-cluster

7. Verify notebook behavior

• Treat the job as successful only when lifecycle is terminal and result is SUCCESS.
• On failure, capture:
  • run_id
  • lifecycle state
  • result state
  • state message
  • notebook path or job id
• If the run succeeded, summarize which notebook or job was exercised and what evidence was collected.

8. Report the outcome

• State what was started locally.
• State whether load generation ran and with what iteration counts.
• State which Databricks job or notebook was executed.
• State whether notebook verification passed or failed.
• If it failed, include the exact failure message and the next corrective step.

9. Use the smoke test when the user wants one-command verification

• Use scripts/smoke_test_notebooks.py.
• It discovers or starts an ngrok tunnel, restarts Docker with the correct advertised Kafka listener, registers the connector, runs bounded load generators, triggers the Databricks job, and waits for terminal results.
• Pass --reset --cluster-id <id> to drop all tables and checkpoints before the smoke run.

# Full smoke test with table reset:
python3 skills/docker-databricks-lab-ops/scripts/smoke_test_notebooks.py \
  --job-id 123 \
  --reset \
  --cluster-id <cluster-id>

# Smoke test without reset (append to existing data):
python3 skills/docker-databricks-lab-ops/scripts/smoke_test_notebooks.py \
  --job-id 123

Scripts

• scripts/start_stack.sh: start Docker Compose services for the lab
• scripts/prepare_ngrok_kafka.py: discover or start an ngrok TCP tunnel for Kafka and print the current public bootstrap
• scripts/register_connector.sh: register the Debezium connector from postgres-connector.json
• scripts/run_generators.sh: run film and rental/payment generators
• scripts/reset_databricks_tables.py: drop all Bronze/Silver/Gold Delta tables and clear streaming checkpoints
• scripts/run_databricks_notebook.py: launch or submit a Databricks run and poll to completion
• scripts/smoke_test_notebooks.py: run the end-to-end smoke test with dynamic ngrok bootstrap handling
• scripts/migrate_and_run.py: full migration script — drops legacy tables, updates the Databricks job via API, resets dvdrental tables, starts Docker+connector+generators, and triggers the job end-to-end

References

• references/repo-workflow.md: repo-specific execution order, assumptions, and parameters