aldengolab/k8s-helm-immutable-field-workaround

Kubernetes Immutable Field Workaround for Helm + ArgoCD

Problem

Some Kubernetes resource fields are immutable after creation. If the Helm chart that owns the resource doesn't expose the field as a configurable value, there is no pure-Helm way to set it — and ArgoCD self-heal will revert any manual patch on the next sync.

Common examples:

• Service.spec.loadBalancerClass — controls which LB controller handles the Service
• PersistentVolumeClaim.spec.storageClassName — cannot change after PVC creation
• PersistentVolumeClaim.spec.accessModes — immutable after creation

Context / Trigger Conditions

• kubectl patch returns: ... is invalid: ...: may not change once set or Forbidden: ... is immutable
• Helm chart renders the resource without the field; no values key exists for it
• ArgoCD reverts every manual patch via self-heal

Solution

The general pattern: let the chart own a minimal/neutral version of the resource, and own the field you need in a separate, independently-managed resource.

Option A — Degrade the chart resource + create a replacement

When the chart resource type can be changed to a less specific variant:

1. Change the chart values so it creates a simpler version of the resource (e.g. ClusterIP instead of LoadBalancer for Services)
2. Create a separate manifest in your GitOps repo that creates the specific resource you actually need with the immutable field set correctly

Example — LoadBalancerClass on a Service the chart doesn't support:

# In chart values: switch to clusterIP
expose:
  type: clusterIP
  clusterIP:
    ports:
      httpPort: 80

# In your GitOps apps chart: dedicated LB Service with correct class
apiVersion: v1
kind: Service
metadata:
  name: myapp-lb
  namespace: myapp
  annotations:
    tailscale.com/hostname: myapp   # optional: Tailscale hostname
spec:
  type: LoadBalancer
  loadBalancerClass: tailscale
  selector:
    app: myapp
    component: nginx
  ports:
    - port: 80
      targetPort: 8080

Option B — Delete and recreate via PreSync hook

When the chart resource cannot be degraded, use an ArgoCD PreSync hook that deletes the resource before sync. The chart then recreates it with the correct values on the same sync.

Full PreSync hook template (including ServiceAccount and RBAC): see references/delete-and-recreate.md Variant A in this skill directory.

Note: Option B causes a brief outage during sync. Option A is preferred when feasible.

Option C — ignoreDifferences + accept drift

When the field doesn't affect behavior and you just want to stop the OutOfSync noise, add ignoreDifferences to the ArgoCD Application:

ignoredDifferences:
  - group: ""
    kind: Service
    name: myapp
    namespace: myapp
    jsonPointers:
      - /spec/loadBalancerClass

Use this only when the drift is cosmetic and the live value is correct.

Verification

• The immutable field is set correctly on the live resource
• ArgoCD shows Synced (not reverting the field)
• The controller that depends on the field (e.g. Tailscale operator) picks it up

Notes

• Always prefer Option A (degrade + separate resource) for clean GitOps
• Option B causes downtime; use only when A is not possible
• Option C leaves the field unmanaged by GitOps; use sparingly
• For PVC immutable fields, see also: kubernetes-csi-pv-spec-update skill

See Also

• kubernetes-csi-pv-spec-update — handles immutable PV spec fields specifically
• harbor-argocd-deployment — concrete example using Option A for loadBalancerClass