apecloud/kubeblocks-horizontal-scaling

Horizontal Scaling: Add or Remove Replicas and Shards

Overview

Horizontal scaling changes the number of replicas (or shards) in a KubeBlocks database cluster. KubeBlocks supports scale-out (add replicas) and scale-in (remove replicas), including decommissioning specific instances and adjusting shard counts for sharded clusters (Redis Cluster, MongoDB sharded).

Official docs: https://kubeblocks.io/docs/preview/user_docs/kubeblocks-for-mysql/cluster-management/scale-a-mysql-cluster Full doc index: https://kubeblocks.io/llms-full.txt

When to Use Which Scaling Type

• Horizontal scaling (this skill): Add read capacity, improve HA, or distribute sharded data. Each new replica gets its own CPU/memory/storage — you're scaling by adding independent database instances.
• Vertical scaling: Increase CPU/memory on existing replicas when the workload is write-heavy or CPU-bound (adding replicas won't help if the bottleneck is single-primary write throughput).
• Volume expansion: Increase disk space when storage is running low — unrelated to compute capacity.

Minimum Replica Constraints

Consensus-based topologies (MySQL Group Replication, Kafka KRaft controllers) require an odd number of replicas (minimum 3) because Raft/Paxos protocols need a majority quorum to elect a leader. With 3 replicas, the system tolerates 1 failure; with 2 replicas, any single failure breaks quorum and the cluster becomes unavailable.

Pre-Check

Before proceeding, verify the cluster is healthy and no other operation is running:

# Cluster must be Running
kubectl get cluster <cluster-name> -n <namespace> -o jsonpath='{.status.phase}'

# No pending OpsRequests
kubectl get opsrequest -n <namespace> -l app.kubernetes.io/instance=<cluster-name> --field-selector=status.phase!=Succeed

If the cluster is not Running or has a pending OpsRequest, wait for it to complete before proceeding.

Check the current replica count:

kubectl get cluster <cluster-name> -n <namespace> -o yaml | grep replicas

Workflow

- [ ] Step 1: Check current replicas
- [ ] Step 2: Apply horizontal scaling OpsRequest
- [ ] Step 3: Monitor the operation
- [ ] Step 4: Verify new topology

Step 1: Check Current Replicas

kubectl get cluster <cluster-name> -n <namespace> -o yaml | grep replicas

Or list pods:

kubectl get pods -n <namespace> -l app.kubernetes.io/instance=<cluster-name>

Step 2: Apply Horizontal Scaling OpsRequest

Scale Out (Add Replicas)

apiVersion: apps.kubeblocks.io/v1beta1
kind: OpsRequest
metadata:
  name: scaleout-<cluster-name>
  namespace: <namespace>
spec:
  clusterName: <cluster-name>
  type: HorizontalScaling
  horizontalScaling:
    - componentName: <component-name>
      scaleOut:
        replicaChanges: <number-to-add>

Example — add 2 replicas to a MySQL cluster.

Before applying, validate with dry-run:

kubectl apply -f - --dry-run=server <<'EOF'
apiVersion: apps.kubeblocks.io/v1beta1
kind: OpsRequest
metadata:
  name: scaleout-mysql-cluster
  namespace: default
spec:
  clusterName: mysql-cluster
  type: HorizontalScaling
  horizontalScaling:
    - componentName: mysql
      scaleOut:
        replicaChanges: 2
EOF

If dry-run reports errors, fix the YAML before proceeding.

kubectl apply -f - <<'EOF'
apiVersion: apps.kubeblocks.io/v1beta1
kind: OpsRequest
metadata:
  name: scaleout-mysql-cluster
  namespace: default
spec:
  clusterName: mysql-cluster
  type: HorizontalScaling
  horizontalScaling:
    - componentName: mysql
      scaleOut:
        replicaChanges: 2
EOF

Scale In (Remove Replicas)

apiVersion: apps.kubeblocks.io/v1beta1
kind: OpsRequest
metadata:
  name: scalein-<cluster-name>
  namespace: <namespace>
spec:
  clusterName: <cluster-name>
  type: HorizontalScaling
  horizontalScaling:
    - componentName: <component-name>
      scaleIn:
        replicaChanges: <number-to-remove>

Example — remove 1 replica from a PostgreSQL cluster.

Before applying, validate with dry-run:

kubectl apply -f - --dry-run=server <<'EOF'
apiVersion: apps.kubeblocks.io/v1beta1
kind: OpsRequest
metadata:
  name: scalein-pg-cluster
  namespace: default
spec:
  clusterName: pg-cluster
  type: HorizontalScaling
  horizontalScaling:
    - componentName: postgresql
      scaleIn:
        replicaChanges: 1
EOF

If dry-run reports errors, fix the YAML before proceeding.

kubectl apply -f - <<'EOF'
apiVersion: apps.kubeblocks.io/v1beta1
kind: OpsRequest
metadata:
  name: scalein-pg-cluster
  namespace: default
spec:
  clusterName: pg-cluster
  type: HorizontalScaling
  horizontalScaling:
    - componentName: postgresql
      scaleIn:
        replicaChanges: 1
EOF

Decommission Specific Instances (Scale In by Name)

To remove a specific pod instead of the last one, use onlineInstancesToOffline:

apiVersion: apps.kubeblocks.io/v1beta1
kind: OpsRequest
metadata:
  name: scalein-specific-<cluster-name>
  namespace: <namespace>
spec:
  clusterName: <cluster-name>
  type: HorizontalScaling
  horizontalScaling:
    - componentName: <component-name>
      scaleIn:
        replicaChanges: 1
        onlineInstancesToOffline:
          - "<pod-name>"

Example — decommission a specific MongoDB replica.

Before applying, validate with dry-run:

kubectl apply -f - --dry-run=server <<'EOF'
apiVersion: apps.kubeblocks.io/v1beta1
kind: OpsRequest
metadata:
  name: scalein-specific-mongo
  namespace: default
spec:
  clusterName: mongo-cluster
  type: HorizontalScaling
  horizontalScaling:
    - componentName: mongodb
      scaleIn:
        replicaChanges: 1
        onlineInstancesToOffline:
          - "mongo-cluster-mongodb-2"
EOF

If dry-run reports errors, fix the YAML before proceeding.

kubectl apply -f - <<'EOF'
apiVersion: apps.kubeblocks.io/v1beta1
kind: OpsRequest
metadata:
  name: scalein-specific-mongo
  namespace: default
spec:
  clusterName: mongo-cluster
  type: HorizontalScaling
  horizontalScaling:
    - componentName: mongodb
      scaleIn:
        replicaChanges: 1
        onlineInstancesToOffline:
          - "mongo-cluster-mongodb-2"
EOF

Shard Scaling (Redis Cluster / MongoDB Sharded)

For sharded topologies, use the shards field to change the number of shard groups:

apiVersion: apps.kubeblocks.io/v1beta1
kind: OpsRequest
metadata:
  name: scale-shards-<cluster-name>
  namespace: <namespace>
spec:
  clusterName: <cluster-name>
  type: HorizontalScaling
  horizontalScaling:
    - componentName: <component-name>
      scaleOut:
        replicaChanges: <shards-to-add>

Note: For Redis Cluster and MongoDB sharded topologies, each "replica" in the component represents a shard group. Increasing replicas adds new shards and triggers data rebalancing automatically.

kubectl Patch Alternative

For simple replica count changes, you can also patch the cluster directly:

kubectl patch cluster <cluster-name> -n <namespace> \
  --type merge -p '{"spec":{"componentSpecs":[{"name":"<component-name>","replicas":<new-total>}]}}'

Example — set MySQL replicas to 5:

kubectl patch cluster mysql-cluster -n default \
  --type merge -p '{"spec":{"componentSpecs":[{"name":"mysql","replicas":5}]}}'

Step 3: Monitor the Operation

kubectl get ops -n <namespace> -w

Success condition: .status.phase = Succeed | Typical: 1-5min | If stuck >10min: kubectl describe ops <ops-name> -n <namespace>

Expected progression: Pending → Running → Succeed.

Watch pods:

kubectl get pods -n <namespace> -l app.kubernetes.io/instance=<cluster-name> -w

Success condition: .status.phase = Running | Typical: 1-5min | If stuck >10min: kubectl describe pod <pod-name> -n <namespace>

Step 4: Verify New Topology

kubectl get cluster <cluster-name> -n <namespace> -o yaml | grep replicas

Confirm pod count:

kubectl get pods -n <namespace> -l app.kubernetes.io/instance=<cluster-name>

Troubleshooting

Scale-out pods stuck in Pending:

• Insufficient node resources. Check: kubectl describe pod <pod-name> -n <namespace>
• Insufficient PVs. Check StorageClass provisioner: kubectl get sc

Scale-in fails:

• Cannot scale below the minimum replica count required by the topology (e.g., Raft-based MySQL requires at least 3 replicas).
• Check OpsRequest events: kubectl describe ops <ops-name> -n <namespace>

Data rebalancing after shard scaling:

• For Redis Cluster, data resharding happens automatically. Monitor with redis-cli --cluster check.
• For MongoDB, balancer redistributes chunks. This may take time for large datasets.

For general agent safety conventions (dry-run, status confirmation, production protection), see safety-patterns.md.

Additional Resources

For engine-specific scaling behaviors, minimum replica constraints, replicas vs shards comparison, and decommissioning patterns, see reference.md.