aldengolab/argocd-outofsync-empty-diff-operator-crd

ArgoCD OutOfSync with Empty Diff: Operator CRD Defaults

Problem

An ArgoCD app reports OutOfSync but argocd app diff shows no output. The resource causing the drift is a CRD managed by a Kubernetes operator that sets post-install default field values ArgoCD never rendered.

Context / Trigger Conditions

• argocd app get <app> shows OutOfSync on a CRD resource
• argocd app diff <app> returns empty output (blank, no lines)
• argocd app get <app> --hard-refresh still shows OutOfSync
• The CRD is owned by an operator (e.g. NVIDIA GPU Operator's ClusterPolicy, Strimzi Kafka, Prometheus Operator, cert-manager Certificate, etc.)
• ignoreDifferences is already set for some fields (e.g. image/version/repository)
• Restarting or upgrading the operator re-triggers the OutOfSync

Root Cause

argocd app diff filters its display through ignoreDifferences — so it can appear empty while the sync-status computation still flags the resource as OutOfSync. The actual differences are in fields the operator sets as defaults after Helm creates the CRD instance, but which Helm leaves as null/absent.

Solution

Step 1: Get the desired and live specs

# Desired state as ArgoCD sees it (what Helm rendered)
argocd app manifests <app-name> | python3 -c "
import sys, yaml, json
content = sys.stdin.read()
for doc in content.split('---'):
    try:
        d = yaml.safe_load(doc)
        if isinstance(d, dict) and d.get('kind') == '<YourCRDKind>' \
                and d.get('apiVersion', '').startswith('<api-group>'):
            print(json.dumps(d.get('spec', {})))
            break
    except: pass
" > /tmp/desired_spec.json

# Live state from cluster
kubectl get <crdkind> <name> -o jsonpath='{.spec}' | \
  python3 -c "import sys,json; print(json.dumps(json.loads(sys.stdin.read())))" \
  > /tmp/live_spec.json

Step 2: Find the actual differences

import json

with open('/tmp/desired_spec.json') as f:
    desired = json.load(f)
with open('/tmp/live_spec.json') as f:
    live = json.load(f)

def diff(d1, d2, path=''):
    diffs = []
    if not isinstance(d1, dict) or not isinstance(d2, dict):
        if d1 != d2:
            return [(path, d1, d2)]
        return []
    for k in set(list(d1.keys()) + list(d2.keys())):
        p = f'{path}.{k}'
        v1, v2 = d1.get(k), d2.get(k)
        if isinstance(v1, dict) or isinstance(v2, dict):
            diffs.extend(diff(v1 or {}, v2 or {}, p))
        elif v1 != v2:
            diffs.append((p, v1, v2))
    return diffs

for p, want, got in diff(desired, live):
    print(f'{p}\n  desired: {want}\n  live:    {got}')

Fields where desired is None and live has a value are operator-set defaults that need to be added to ignoreDifferences.

Step 3: Add operator-defaulted fields to ignoreDifferences

In the ArgoCD Application template, add each diffing field path:

ignoreDifferences:
  - group: <api-group>
    kind: <CRDKind>
    jqPathExpressions:
      # existing entries ...
      - .spec.someField.operatorDefault
      - .spec.anotherField

Commit and push. ArgoCD will pick up the change on next sync.

Verification

argocd app get <app-name> --hard-refresh -o json | \
  python3 -c "import json,sys; d=json.load(sys.stdin); \
  print('sync:', d['status']['sync']['status'])"
# Should show: sync: Synced

Notes

• If argocd app manifests returns an empty desired spec file, the Helm source may use a multi-source Application — check which source index renders the CRD.
• Restarting the operator deployment re-triggers field defaulting, confirming which fields it sets and helping identify any missed by the initial comparison.
• This pattern applies to any operator that mutates its own CRD post-install: NVIDIA GPU Operator (ClusterPolicy), Strimzi (Kafka), Prometheus Operator (Prometheus CR), cert-manager, etc.
• managedFieldsManagers in ignoreDifferences is an alternative approach when the CRD has proper SSA managed fields — but many operator-created CRDs have zero managed fields entries, making jqPathExpressions the only viable option.