agentworkforce/deploying-to-staging-environment

Deploying to Staging Environment

Overview

The staging environment deployment is a coordinated multi-repo process that synchronizes code across three repositories (relay, relay-dashboard, relay-cloud) and automatically triggers deployment via GitHub Actions. Staging deployments ensure feature verification before production while keeping main branch clean during development.

When to Use

• Integrating features across repos - Multiple components depend on changes in different repos
• Testing feature branches together - Verify feature branch changes don't break integration
• Promoting main to staging - Standard sync to keep staging up-to-date with latest main
• Verifying deployment readiness - Test infrastructure changes or deployment workflows
• Cross-team coordination - Share feature branches for integration testing

When NOT to use:

• Emergency hotfixes (use production deployment instead)
• Single-repo changes only (push directly if not blocking other repos)
• Testing without intent to deploy (use local environment instead)

Architecture

The staging deployment system consists of:

Three Repos (relay, relay-dashboard, relay-cloud)
    ↓
Staging Branches (synced via git push)
    ↓
relay-cloud staging push triggers GitHub Actions
    ↓
Deploy-Staging Workflow (deploy-staging.yml)
    ↓
Fly.io Staging Environment (agent-relay-staging)
    ↓
Automatic health check verification

Key details:

• Each repo has independent staging branch
• relay-cloud staging branch push triggers automatic deployment
• Workflow accepts optional relay/dashboard branch overrides
• Falls back to main if specified branch doesn't exist
• Workspace image builds in parallel with API deployment

Quick Reference

Standard Workflow (All Repos)

# 1. Create git worktree for staging work (BEST PRACTICE)
cd /data/repos/relay
git worktree add .worktrees/staging-push main
cd .worktrees/staging-push

# 2. Push latest main to staging (fetch fresh)
git fetch origin main:main
git push origin main:staging

# 3. Or push current feature branch to staging (if desired)
git fetch origin feature/your-branch:feature/your-branch
git push origin feature/your-branch:staging

# 4. Clean up worktree
cd /data/repos/relay
git worktree remove .worktrees/staging-push

Relay-Dashboard

cd /data/repos/relay-dashboard
git worktree add .worktrees/staging-push main
cd .worktrees/staging-push
git fetch origin main:main
git push origin main:staging
cd /data/repos/relay-dashboard
git worktree remove .worktrees/staging-push

Relay-Cloud (Triggers Deployment)

cd /data/repos/relay-cloud
git worktree add .worktrees/staging-push main
cd .worktrees/staging-push
git fetch origin main:main
git push origin main:staging
# ⚠️ This push triggers deploy-staging.yml workflow
cd /data/repos/relay-cloud
git worktree remove .worktrees/staging-push

Implementation

Why Git Worktrees?

Git worktrees provide several benefits for staging workflows:

1. Isolation - Work on staging without changing main working directory state
2. Safety - Feature branch checked out in worktree won't affect your current work
3. Cleanliness - No lingering state changes after pushing
4. Best practice - Standard DevOps approach for multi-branch operations

Step-by-Step Deployment Process

Prerequisites

• Access to all three repos with push permission
• Current working directory: one of the repos
• No uncommitted changes blocking worktree creation

Execute Staging Push Across All Repos

Step 1: Relay

cd /data/repos/relay
git worktree add .worktrees/staging-push main
cd .worktrees/staging-push
git fetch origin main:main
git push origin main:staging
cd /data/repos/relay
git worktree remove .worktrees/staging-push

Expected output:

Updated 'main' to 'origin/main'
remote: Create pull request for staging...
To github.com:AgentWorkforce/relay.git
 [new branch]      main -> staging
or
 * [up-to-date]    main -> staging

Step 2: Relay-Dashboard

cd /data/repos/relay-dashboard
git worktree add .worktrees/staging-push main
cd .worktrees/staging-push
git fetch origin main:main
git push origin main:staging
cd /data/repos/relay-dashboard
git worktree remove .worktrees/staging-push

Step 3: Relay-Cloud (Triggers Deployment)

cd /data/repos/relay-cloud
git worktree add .worktrees/staging-push main
cd .worktrees/staging-push
git fetch origin main:main
git push origin main:staging
# Deployment starts automatically!
cd /data/repos/relay-cloud
git worktree remove .worktrees/staging-push

Verify Deployment

After pushing relay-cloud, GitHub Actions starts automatically:

1. Watch workflow progress:

   • Go to: https://github.com/AgentWorkforce/relay-cloud/actions
   • Find "Deploy (Staging)" workflow run
   • Check that both jobs pass:
     • "Deploy to Staging" (Fly.io deployment)
     • "Build Staging Workspace" (Docker image build)

2. Verify health check:

   • Workflow runs curl https://agent-relay-staging.fly.dev/health
   • Expected: 200 response within 150 seconds
   • Retries 30 times with 5-second intervals

3. Check deployment summary:

   • Click workflow run
   • View "Deployment Summary" in step summary
   • Confirms deployment to agent-relay-staging

Pushing Feature Branches to Staging

Instead of main, push a specific feature branch:

cd /data/repos/relay
git worktree add .worktrees/staging-push main
cd .worktrees/staging-push
git fetch origin feature/your-feature:feature/your-feature
git push origin feature/your-feature:staging
cd /data/repos/relay
git worktree remove .worktrees/staging-push

For relay-cloud with custom relay/dashboard branches:

The deploy-staging.yml workflow accepts optional inputs to specify exact branches:

# Push relay-cloud staging (will trigger GitHub Actions with inputs)
cd /data/repos/relay-cloud
git worktree add .worktrees/staging-push main
cd .worktrees/staging-push
git fetch origin feature/custom-branch:feature/custom-branch
git push origin feature/custom-branch:staging
cd /data/repos/relay-cloud
git worktree remove .worktrees/staging-push

Then manually trigger with specific branches:

• Go to: https://github.com/AgentWorkforce/relay-cloud/actions/workflows/deploy-staging.yml
• Click "Run workflow"
• Enter: relay_branch and dashboard_branch (optional)
• Confirm run starts

Common Mistakes

| Mistake | Problem | Fix | |---------|---------|-----| | Pushing from dirty working tree | Worktree creation fails with error | Commit or stash changes before creating worktree | | Forgetting to remove worktree | Accumulates worktree directories | Always run git worktree remove .worktrees/staging-push after pushing | | Pushing staging without relay/dashboard | Deployment uses outdated relay/dashboard | Coordinate all three repos or use GitHub Actions inputs to override | | Not fetching fresh ref | Pushes stale local branch state | Always git fetch origin branch:branch before pushing | | Confusing staging branch direction | Push main→staging, not staging→main | Remember: feature work → staging branch, for promotion/testing | | Assuming all repos auto-sync | relay/dashboard changes don't auto-trigger | Only relay-cloud staging push triggers deployment | | Not checking health status | Deployment may fail silently | Always verify workflow completes and health check passes | | Creating worktrees in wrong directory | Path confusion for multiple repos | Each repo is separate; create worktrees within that repo's .worktrees/ |

Workflow Details

deploy-staging.yml Behavior

Triggers:

• Manual push to staging branch (automatic)
• GitHub Actions workflow_dispatch (manual with optional inputs)

Optional Workflow Inputs:

• relay_branch - Override which relay branch to deploy (defaults to current staging branch)
• dashboard_branch - Override which relay-dashboard branch to deploy (defaults to current staging branch)
• Falls back to main if specified branch doesn't exist in remote

Deployment Steps:

1. Checkout relay-cloud repository
2. Determine relay branch (input or current, fallback to main)
3. Determine dashboard branch (input or current, fallback to main)
4. Setup Fly CLI
5. Deploy to Fly.io with flyctl deploy
6. Verify health check endpoint
7. Create deployment summary

Environment: staging (requires GitHub Actions environment secrets)

Deployment Success Criteria

✅ All checks passed:

• [ ] "Deploy to Staging" job completes
• [ ] "Build Staging Workspace" job completes
• [ ] Health check succeeds (HTTP 200 from /health endpoint)
• [ ] Deployment summary shows all expected values

❌ Deployment failed:

• Workflow run shows red X
• Check step that failed in workflow logs
• Common failures: auth (secrets), health check timeout, Docker build error

Real-World Workflow

Scenario: Feature across all three repos ready for integration testing

# 1. Ensure local main branches are up-to-date
cd /data/repos/relay
git fetch origin main

cd /data/repos/relay-dashboard
git fetch origin main

cd /data/repos/relay-cloud
git fetch origin main

# 2. Push relay main to staging (using worktree)
cd /data/repos/relay
git worktree add .worktrees/staging-push main
cd .worktrees/staging-push
git push origin main:staging
cd ..
git worktree remove .worktrees/staging-push

# 3. Push relay-dashboard main to staging
cd /data/repos/relay-dashboard
git worktree add .worktrees/staging-push main
cd .worktrees/staging-push
git push origin main:staging
cd ..
git worktree remove .worktrees/staging-push

# 4. Push relay-cloud main to staging (triggers deployment)
cd /data/repos/relay-cloud
git worktree add .worktrees/staging-push main
cd .worktrees/staging-push
git push origin main:staging
cd ..
git worktree remove .worktrees/staging-push

# 5. Monitor deployment
echo "Check: https://github.com/AgentWorkforce/relay-cloud/actions"
# Wait for "Deploy (Staging)" workflow to complete
# Verify health check passes
# Access staging at: https://agent-relay-staging.fly.dev

Troubleshooting

Worktree Errors

Error: "fatal: 'main' already exists"

Solution: git worktree remove .worktrees/staging-push
         (if worktree from previous run still exists)

Error: "fatal: cannot checkout branch into locked working tree"

Solution: Ensure main branch isn't checked out in primary working tree
         git status (confirm different branch is checked out)

Deployment Failures

Health check timeout after 150s

Problem: Staging environment took too long to become healthy
Solution: - Check Fly.io logs: flyctl logs --app agent-relay-staging
         - Restart app: flyctl restart --app agent-relay-staging
         - Check for build errors in workflow logs
         - Look for startup errors in deployment step

Docker build fails

Problem: "Build Staging Workspace" job failed
Solution: - Check docker-staging.yml workflow logs
         - Verify Dockerfile exists and is valid
         - Check for resource/dependency issues
         - Rebuild locally to verify docker configuration

Branch not found fallback to main

Problem: Workflow says "using main" for your custom branch
Reason: Specified branch doesn't exist in relay/relay-dashboard
Solution: - Verify branch name is exact (case-sensitive)
         - Ensure branch exists in remote (git ls-remote origin)
         - Use GitHub Actions workflow_dispatch with correct inputs

Dependencies & Requirements

Git: Worktree feature (Git 2.5+, standard on all modern systems)

Permissions: Push access to all three repos:

• github.com:AgentWorkforce/relay
• github.com:AgentWorkforce/relay-dashboard
• github.com:AgentWorkforce/relay-cloud

GitHub Secrets (relay-cloud):

• CROSS_REPO_TOKEN - PAT with repo access across all three repos
• FLY_API_TOKEN or FLY_API_TOKEN_STAGING - Fly.io deployment token

Environment: staging (configured in relay-cloud)

---

Key Principle: Use git worktrees for clean, isolated branch operations. Push coordination across repos, with relay-cloud staging push triggering automatic deployment.