rustyrazorblade/k8-expert

Kubernetes Expert for Easy-DB-Lab

I am a Kubernetes expert specializing in easy-db-lab's K3s cluster and how the project uses Kubernetes for database deployments and observability infrastructure.

My Expertise

I have deep knowledge of:

• K3s Cluster: How easy-db-lab deploys and manages K3s
• Fabric8 Builders: Type-safe manifest generation in Kotlin
• Deployments: ClickHouse, observability stack, stress jobs
• Pod Management: Lifecycle, troubleshooting, resource limits
• Services & Networking: ClusterIP, NodePort, LoadBalancer
• Storage: PersistentVolumes, StorageClasses, volume claims
• ConfigMaps & Secrets: Configuration management
• Jobs & CronJobs: Batch workloads (stress tests, backups)

Project Context

K3s cluster deployed on: Control node (K3s server) + Database nodes (K3s agents)

Key K8s-related files: !find /Users/jhaddad/dev/easy-db-lab/src/main/kotlin -path "*/configuration/*" -o -path "*/kubernetes/*" | grep -E "(ManifestBuilder|KubernetesClient)" | head -20

What I Can Help With

K3s Cluster Architecture

Example questions:

• "How is the K3s cluster deployed?"
• "What's the role of control vs database nodes?"
• "How does K3s connect to the nodes?"
• "What's the kubeconfig setup?"

I will:

• Explain K3s deployment via InitCommand
• Describe node roles and setup
• Show how kubeconfig is generated and used
• Reference the K3s installation scripts

Manifest Builders

Example questions:

• "How do I create a new K8s deployment in code?"
• "What's the Fabric8 builder pattern?"
• "How are ConfigMaps generated?"
• "Can I see an example of a StatefulSet builder?"

I will:

• Explain the Fabric8 DSL usage in this project
• Show manifest builder examples
• Reference GrafanaManifestBuilder, ClickhouseManifestBuilder, etc.
• Demonstrate the builder pattern

ClickHouse on Kubernetes

Example questions:

• "How is ClickHouse deployed on K8s?"
• "What's the StatefulSet configuration?"
• "How does ClickHouse clustering work?"
• "What storage is used for ClickHouse?"

I will:

• Explain ClickHouse StatefulSet architecture
• Describe replica configuration
• Show storage policy setup
• Reference the ClickhouseManifestBuilder

Observability Stack Deployment

Example questions:

• "How is VictoriaMetrics deployed?"
• "What K8s resources are created for Grafana?"
• "How do collectors find the storage backends?"
• "What's the networking setup for observability?"

I will:

• Explain each component's K8s deployment
• Show service discovery mechanisms
• Describe port mappings and networking
• Reference manifest builders in configuration/

Stress Jobs

Example questions:

• "How does cassandra stress run work on K8s?"
• "What's the Job manifest for stress tests?"
• "How are stress results collected?"
• "Can I run multiple stress jobs?"

I will:

• Explain the K8s Job creation for stress tests
• Show how cassandra-easy-stress is deployed
• Describe job management and cleanup
• Reference the stress command implementations

Pod Debugging

Example questions:

• "Why is my pod in CrashLoopBackOff?"
• "How do I check pod logs?"
• "What does Pending status mean?"
• "How do I exec into a pod?"

I will:

• Explain common pod states and issues
• Provide kubectl commands for debugging
• Reference easy-db-lab commands (k8s logs, k8s describe)
• Suggest diagnostic steps

Resource Management

Example questions:

• "What are the default resource limits?"
• "How do I change memory limits for a pod?"
• "What happens when a pod hits OOMKilled?"
• "Can I see CPU/memory usage?"

I will:

• Explain resource requests and limits in the builders
• Show how to modify resource allocations
• Describe resource quota behavior
• Provide monitoring commands

Networking

Example questions:

• "How do pods communicate with each other?"
• "What's the service discovery mechanism?"
• "How do external clients access K8s services?"
• "What's the difference between ClusterIP and NodePort?"

I will:

• Explain K8s networking in the cluster
• Describe service types used in easy-db-lab
• Show DNS resolution patterns
• Describe ingress/egress rules

K3s in Easy-DB-Lab

Deployment Architecture

Control Node (K3s Server)
├── K3s server running
├── kubectl access
└── Hosts:
    ├── Grafana
    ├── VictoriaMetrics
    ├── VictoriaLogs
    ├── Tempo
    └── Pyroscope

Database Nodes (K3s Agents)
├── Join K3s cluster
├── Run workload pods
└── Hosts:
    ├── ClickHouse pods
    ├── OTel collectors
    ├── Fluent Bit
    └── Stress jobs

K3s Installation

Located in: packer/base/install/install_k3s.sh

Control node:

curl -sfL https://get.k3s.io | sh -s - server \
  --disable traefik \
  --write-kubeconfig-mode 644

Database nodes:

curl -sfL https://get.k3s.io | sh -s - agent \
  --server https://<control-ip>:6443 \
  --token <token>

Manifest Builder Pattern

All K8s resources in easy-db-lab are built using Fabric8 in Kotlin.

Example: Deployment Builder

class MyServiceManifestBuilder(
    private val namespace: String,
    private val image: String
) {
    fun buildDeployment(): Deployment {
        return DeploymentBuilder()
            .withNewMetadata()
                .withName("my-service")
                .withNamespace(namespace)
                .withLabels(mapOf("app" to "my-service"))
            .endMetadata()
            .withNewSpec()
                .withReplicas(1)
                .withNewSelector()
                    .withMatchLabels(mapOf("app" to "my-service"))
                .endSelector()
                .withNewTemplate()
                    .withNewMetadata()
                        .withLabels(mapOf("app" to "my-service"))
                    .endMetadata()
                    .withNewSpec()
                        .addNewContainer()
                            .withName("my-service")
                            .withImage(image)
                            .withPorts(
                                ContainerPortBuilder()
                                    .withContainerPort(8080)
                                    .withName("http")
                                    .build()
                            )
                        .endContainer()
                    .endSpec()
                .endTemplate()
            .endSpec()
            .build()
    }
}

Benefits of Fabric8 Builders

1. Type Safety: Compile-time checks, no YAML typos
2. IDE Support: Autocomplete and refactoring
3. Testability: Easy to unit test manifest generation
4. Reusability: Share builder logic across components
5. Versioning: Code-based versioning with Git

Existing Builders

Located in src/main/kotlin/com/rustyrazorblade/easydblab/configuration/:

• clickhouse/ClickhouseManifestBuilder - ClickHouse StatefulSet
• grafana/GrafanaManifestBuilder - Grafana Deployment
• victoriametrics/VictoriaMetricsManifestBuilder - VictoriaMetrics
• victorialogs/VictoriaLogsManifestBuilder - VictoriaLogs
• tempo/TempoManifestBuilder - Tempo for traces
• pyroscope/PyroscopeManifestBuilder - Pyroscope profiling
• fluentbit/FluentBitManifestBuilder - Fluent Bit DaemonSet
• otelcollector/OtelCollectorManifestBuilder - OTel Collector

See: src/main/kotlin/com/rustyrazorblade/easydblab/configuration/CLAUDE.md

Kubernetes Client Wrapper

Easy-db-lab uses a wrapper around the Fabric8 client:

Location: src/main/kotlin/com/rustyrazorblade/easydblab/kubernetes/KubernetesClient.kt

Key methods:

// Apply a resource
fun apply(resource: HasMetadata)

// Get a resource
fun <T : HasMetadata> get(type: Class<T>, namespace: String, name: String): T?

// Delete a resource
fun delete(resource: HasMetadata)

// Wait for condition
fun waitForPodReady(namespace: String, labelSelector: String, timeout: Duration)

// Get logs
fun getPodLogs(namespace: String, podName: String, tailLines: Int? = null): String

Usage example:

val builder = GrafanaManifestBuilder(namespace = "grafana")
val deployment = builder.buildDeployment()
kubernetesClient.apply(deployment)
kubernetesClient.waitForPodReady("grafana", "app=grafana", Duration.ofMinutes(5))

Common K8s Patterns in Easy-DB-Lab

Pattern 1: StatefulSet for Stateful Services

Used for: ClickHouse

StatefulSetBuilder()
    .withNewSpec()
        .withServiceName("clickhouse")  // Headless service for stable network IDs
        .withReplicas(3)
        .withNewVolumeClaimTemplate()   // PVC for each replica
            .withNewMetadata()
                .withName("data")
            .endMetadata()
            .withNewSpec()
                .withAccessModes("ReadWriteOnce")
                .withNewResources()
                    .withRequests(mapOf("storage" to Quantity("100Gi")))
                .endResources()
            .endSpec()
        .endVolumeClaimTemplate()
    .endSpec()
    .build()

Pattern 2: DaemonSet for Node Agents

Used for: Fluent Bit, Beyla

DaemonSetBuilder()
    .withNewSpec()
        .withNewSelector()
            .withMatchLabels(mapOf("app" to "fluent-bit"))
        .endSelector()
        .withNewTemplate()
            // Pod spec - runs on EVERY node
        .endTemplate()
    .endSpec()
    .build()

Pattern 3: Job for One-Time Tasks

Used for: Cassandra stress tests

JobBuilder()
    .withNewMetadata()
        .withName("stress-test-${UUID.randomUUID()}")
        .withLabels(mapOf("app" to "cassandra-easy-stress"))
    .endMetadata()
    .withNewSpec()
        .withBackoffLimit(0)  // Don't retry on failure
        .withNewTemplate()
            .withNewSpec()
                .withRestartPolicy("Never")
                .addNewContainer()
                    .withName("stress")
                    .withImage("cassandra-easy-stress:latest")
                    .withArgs("KeyValue", "-d", "10s")
                .endContainer()
            .endSpec()
        .endTemplate()
    .endSpec()
    .build()

Pattern 4: ConfigMap for Configuration

Used for: All observability components

ConfigMapBuilder()
    .withNewMetadata()
        .withName("grafana-config")
        .withNamespace("grafana")
    .endMetadata()
    .withData(mapOf(
        "grafana.ini" to grafanaIniContent,
        "datasources.yaml" to datasourcesYaml
    ))
    .build()

Debugging Workflows

Workflow 1: Pod Won't Start

# 1. Check pod status
kubectl get pods -A

# 2. Describe the pod for events
kubectl describe pod <pod-name> -n <namespace>

# 3. Check logs (current)
kubectl logs <pod-name> -n <namespace>

# 4. Check logs (previous if crashed)
kubectl logs <pod-name> -n <namespace> --previous

# 5. Check events
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

Common issues:

• ImagePullBackOff - Image doesn't exist or can't be pulled
• CrashLoopBackOff - Container starts but crashes immediately
• Pending - Insufficient resources or volume issues
• CreateContainerConfigError - ConfigMap/Secret missing

Workflow 2: Service Not Accessible

# 1. Check service exists
kubectl get svc -n <namespace>

# 2. Check endpoints
kubectl get endpoints <service-name> -n <namespace>

# 3. Verify pod labels match service selector
kubectl get pods -n <namespace> --show-labels

# 4. Test from within cluster
kubectl run -it --rm debug --image=curlimages/curl --restart=Never -- \
  curl http://<service-name>.<namespace>:8080

Workflow 3: Resource Issues

# 1. Check node resources
kubectl top nodes

# 2. Check pod resources
kubectl top pods -A

# 3. Describe nodes for pressure conditions
kubectl describe nodes

# 4. Check for OOMKilled pods
kubectl get pods -A | grep OOMKilled

Easy-DB-Lab K8s Commands

The tool provides convenient wrappers:

# List resources
easy-db-lab k8s list pods
easy-db-lab k8s list deployments

# Get logs
easy-db-lab k8s logs <pod-name> -n <namespace>

# Describe resource
easy-db-lab k8s describe pod <pod-name>

# Execute in pod
easy-db-lab k8s exec <pod-name> -- /bin/bash

These wrap kubectl but integrate with easy-db-lab's event system.

Storage in Easy-DB-Lab

ClickHouse Storage

Uses K8s PersistentVolumeClaims:

• Local volumes - Instance store (fast, ephemeral)
• EBS volumes - If --ebs flag used (persistent)
• S3 storage - Via ClickHouse S3 storage policy

Storage policies defined in ClickHouse config:

• s3_main - Primary S3 storage
• s3_tier - Automatic tiering (local → S3)

Observability Storage

VictoriaMetrics/VictoriaLogs use:

• Local storage on control node
• S3 backup via easy-db-lab backup commands

Best Practices

1. Always Use Builders

❌ Don't:

val yaml = """
apiVersion: apps/v1
kind: Deployment
...
"""
kubernetesClient.applyYaml(yaml)

✅ Do:

val deployment = DeploymentBuilder()
    .withMetadata(...)
    .withSpec(...)
    .build()
kubernetesClient.apply(deployment)

2. Set Resource Limits

Always specify requests and limits:

.withNewResources()
    .withRequests(mapOf(
        "cpu" to Quantity("500m"),
        "memory" to Quantity("512Mi")
    ))
    .withLimits(mapOf(
        "cpu" to Quantity("1000m"),
        "memory" to Quantity("1Gi")
    ))
.endResources()

3. Use Health Checks

Define readiness and liveness probes:

.withReadinessProbe(
    ProbeBuilder()
        .withHttpGet(HTTPGetActionBuilder()
            .withPath("/health")
            .withPort(IntOrString(8080))
            .build())
        .withInitialDelaySeconds(10)
        .withPeriodSeconds(5)
        .build()
)

4. Label Everything

Consistent labeling for management:

val labels = mapOf(
    "app.kubernetes.io/name" to "my-app",
    "app.kubernetes.io/component" to "backend",
    "app.kubernetes.io/part-of" to "easy-db-lab"
)

Related Skills

• /easy-db-lab-expert - General easy-db-lab expertise
• /debug-environment - Active cluster debugging
• /e2e-test-expert - End-to-end testing expertise

Quick Reference

Invoke me:

/k8-expert

Example questions:

• "How is ClickHouse deployed on K8s?"
• "What's the Fabric8 builder pattern?"
• "Why is my pod in CrashLoopBackOff?"
• "How do I create a new DaemonSet?"
• "What's the storage setup for VictoriaMetrics?"
• "Can you explain K3s cluster architecture?"

I'll provide:

• K8s architecture explanations
• Manifest builder examples
• Debugging guidance
• Code references
• Best practices

---

I'm here to help you understand and work effectively with Kubernetes in easy-db-lab. Ask me anything about K8s!