gavinmcfall/k8s-operator

k8s-operator

Query before acting. Understand scope, then drill down.

---

Hard Constraints

• Talos Linux has no SSH. Use talosctl for node management.
• Prefer read-only operations. State changes go through GitOps (see home-ops-deployer skill).
• Never kubectl apply or kubectl edit — Flux reverts manual changes.
• Never kubectl delete pods to "fix" things — find the root cause.

---

Cluster: your-cluster

| Node | IP | Roles | OS | |------|-----|-------|----| | your-cluster-01 | 10.90.3.101 | control-plane | Talos v1.11.5 | | your-cluster-02 | 10.90.3.102 | control-plane | Talos v1.11.5 | | your-cluster-03 | 10.90.3.103 | control-plane | Talos v1.11.5 |

• Kubernetes: v1.33.1
• Container runtime: containerd 2.1.5
• CNI: Cilium (eBPF, BGP, DSR mode)
• Control plane: 10.90.3.100:6443 (VIP)

---

Storage

Rook-Ceph

3x NVMe OSDs, ~5.2 TiB total capacity.

| Storage Class | Provisioner | Use | |---------------|-------------|-----| | ceph-block (default) | rook-ceph.rbd.csi.ceph.com | Single-instance apps, databases | | ceph-filesystem | rook-ceph.cephfs.csi.ceph.com | Shared/multi-instance (RWX) | | ceph-bucket | rook-ceph.ceph.rook.io/bucket | S3-compatible object storage | | openebs-hostpath | openebs.io/local | Node-local storage (monitoring, caches) |

Check Ceph Health

kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph status
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph osd status
kubectl -n rook-ceph exec deploy/rook-ceph-tools -- ceph df

---

Networking

Load Balancer IPs (Cilium LB-IPAM via BGP)

| IP | Service | |----|---------| | 10.99.8.201 | External gateway (public via Cloudflare tunnel) | | 10.99.8.202 | Internal gateway (LAN direct) |

Infrastructure

| IP | Device | |----|--------| | 10.90.254.1 | UDM Pro (router/DNS/firewall) | | 10.96.0.10 | CoreDNS (cluster DNS) |

Key Concepts

• Split-horizon DNS: Same hostname resolves differently from LAN vs internet. external-dns manages Cloudflare, external-dns-unifi manages UDM Pro records.
• DSR mode: Direct Server Return. Known limitation: same-node hairpin fails. CoreDNS template rewrites internal hostnames to ClusterIP as workaround.
• BGP: Cilium advertises LoadBalancer IPs to UDM Pro (ASN 65010 → 65001).

---

Databases

| Service | Type | Namespace | Notes | |---------|------|-----------|-------| | postgres18-cluster | PostgreSQL 18 | database | 3 replicas, CloudNative-PG, pgBackRest backups | | postgres18-immich | PostgreSQL 18 | database | 3 replicas, dedicated to Immich | | mariadb | MariaDB | database | Single instance | | dragonfly | Redis-compatible | database | 6 replicas, Dragonfly operator | | mosquitto | MQTT | database | Single instance |

---

Deployed Services by Namespace

See references/service-inventory.md for the full list.

Key namespaces for game development context:

• games: pelican (game server panel), pelican-dev, romm
• plane: project management (Plane)
• home: bookstack (wiki), filebrowser, searxng, manyfold (3D models)
• storage: minio-console, kopia, syncthing
• observability: grafana, prometheus, loki

---

Talos Management

Talos is immutable. No SSH, no package manager, no shell access on nodes.

# Node health
talosctl --nodes 10.90.3.101 health
talosctl --nodes 10.90.3.101 get members

# System logs
talosctl --nodes 10.90.3.101 logs kubelet
talosctl --nodes 10.90.3.101 dmesg

# Node config (read-only)
talosctl --nodes 10.90.3.101 get machineconfig

# Apply config changes (use with caution)
talosctl --nodes 10.90.3.101 apply-config --file <config.yaml>

# Upgrade Talos
task talos:upgrade node=your-cluster-01

# Upgrade Kubernetes
task talos:upgrade-k8s

---

Troubleshooting

See references/troubleshooting.md for detailed workflows.

Quick Checks

# Cluster overview
kubectl get nodes -o wide
kubectl top nodes

# All pods not Running
kubectl get pods -A --field-selector status.phase!=Running

# Recent events
kubectl get events -A --sort-by=.lastTimestamp | tail -20

# Flux status
flux get kustomizations
flux get helmreleases -A | grep -v True

# Specific app
flux get hr <name> -n <namespace>
kubectl logs -n <namespace> -l app.kubernetes.io/name=<app>
kubectl describe pod -n <namespace> -l app.kubernetes.io/name=<app>

---

Taskfile Commands

| Command | Purpose | |---------|---------| | task kubernetes:kubeconform | Validate YAML schemas | | task kubernetes:resources | List all cluster resources | | task kubernetes:sync-secrets | Force ExternalSecret refresh | | task kubernetes:network ns=<ns> | Debug pod networking (spawns netshoot) | | task flux:reconcile | Force full Flux reconciliation | | task flux:hr-restart | Restart failed HelmReleases | | task volsync:snapshot app=<app> ns=<ns> | Snapshot an app PVC | | task volsync:restore app=<app> ns=<ns> | Restore a PVC from snapshot | | task volsync:list ns=<ns> | List snapshots |

---

Deeper

• references/service-inventory.md — Full list of deployed services
• references/troubleshooting.md — Step-by-step diagnostic workflows
• ~/home-ops/docs/ai-context/NETWORKING.md — Full networking architecture
• ~/home-ops/docs/ai-context/ARCHITECTURE.md — Backup strategy, operational limits

---

Understand first. Act through GitOps.