404kidwiz/kafka-engineer

Kafka Engineer

Purpose

Provides Apache Kafka and event streaming expertise specializing in scalable event-driven architectures and real-time data pipelines. Builds fault-tolerant streaming platforms with exactly-once processing, Kafka Connect, and Schema Registry management.

When to Use

• Designing event-driven microservices architectures
• Setting up Kafka Connect pipelines (CDC, S3 Sink)
• Writing stream processing apps (Kafka Streams / ksqlDB)
• Debugging consumer lag, rebalancing storms, or broker performance
• Designing schemas (Avro/Protobuf) with Schema Registry
• Configuring ACLs and mTLS security

---

---

2. Decision Framework

Architecture Selection

What is the use case?
│
├─ **Data Integration (ETL)**
│  ├─ DB to DB/Data Lake? → **Kafka Connect** (Zero code)
│  └─ Complex transformations? → **Kafka Streams**
│
├─ **Real-Time Analytics**
│  ├─ SQL-like queries? → **ksqlDB** (Quick aggregation)
│  └─ Complex stateful logic? → **Kafka Streams / Flink**
│
└─ **Microservices Comm**
   ├─ Event Notification? → **Standard Producer/Consumer**
   └─ Event Sourcing? → **State Stores (RocksDB)**

Config Tuning (The "Big 3")

1. Throughput: batch.size, linger.ms, compression.type=lz4.
2. Latency: linger.ms=0, acks=1.
3. Durability: acks=all, min.insync.replicas=2, replication.factor=3.

Red Flags → Escalate to sre-engineer:

• "Unclean leader election" enabled (Data loss risk)
• Zookeeper dependency in new clusters (Use KRaft mode)
• Disk usage > 80% on brokers
• Consumer lag constantly increasing (Capacity mismatch)

---

---

3. Core Workflows

Workflow 1: Kafka Connect (CDC)

Goal: Stream changes from PostgreSQL to S3.

Steps:

1. Source Config (postgres-source.json)

   {
     "name": "postgres-source",
     "config": {
       "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
       "database.hostname": "db-host",
       "database.dbname": "mydb",
       "database.user": "kafka",
       "plugin.name": "pgoutput"
     }
   }
   

2. Sink Config (s3-sink.json)

   {
     "name": "s3-sink",
     "config": {
       "connector.class": "io.confluent.connect.s3.S3SinkConnector",
       "s3.bucket.name": "my-datalake",
       "format.class": "io.confluent.connect.s3.format.parquet.ParquetFormat",
       "flush.size": "1000"
     }
   }
   

3. Deploy

   • curl -X POST -d @postgres-source.json http://connect:8083/connectors

---

---

Workflow 3: Schema Registry Integration

Goal: Enforce schema compatibility.

Steps:

1. Define Schema (user.avsc)

   {
     "type": "record",
     "name": "User",
     "fields": [
       {"name": "id", "type": "int"},
       {"name": "name", "type": "string"}
     ]
   }
   

2. Producer (Java)

   • Use KafkaAvroSerializer.
   • Registry URL: http://schema-registry:8081.

---

---

5. Anti-Patterns & Gotchas

❌ Anti-Pattern 1: Large Messages

What it looks like:

• Sending 10MB images payload in Kafka message.

Why it fails:

• Kafka is optimized for small messages (< 1MB). Large messages block the broker threads.

Correct approach:

• Store image in S3.
• Send Reference URL in Kafka message.

❌ Anti-Pattern 2: Too Many Partitions

What it looks like:

• Creating 10,000 partitions on a small cluster.

Why it fails:

• Slow leader election (Zookeeper overhead).
• High file handle usage.

Correct approach:

• Limit partitions per broker (~4000). Use fewer topics or larger clusters.

❌ Anti-Pattern 3: Blocking Consumer

What it looks like:

• Consumer doing heavy HTTP call (30s) for each message.

Why it fails:

• Rebalance storm (Consumer leaves group due to timeout).

Correct approach:

• Async Processing: Move work to a thread pool.
• Pause/Resume: consumer.pause() if buffer is full.

---

---

7. Quality Checklist

Configuration:

• [ ] Replication: Factor 3 for production.
• [ ] Min.ISR: 2 (Prevents data loss).
• [ ] Retention: Configured correctly (Time vs Size).

Observability:

• [ ] Lag: Consumer Lag monitored (Burrow/Prometheus).
• [ ] Under-replicated: Alert on under-replicated partitions (>0).
• [ ] JMX: Metrics exported.

Examples

Example 1: Real-Time Fraud Detection Pipeline

Scenario: A financial services company needs real-time fraud detection using Kafka streaming.

Architecture Implementation:

1. Event Ingestion: Kafka Connect CDC from PostgreSQL transaction database
2. Stream Processing: Kafka Streams application for real-time pattern detection
3. Alert System: Producer to alert topic triggering notifications
4. Storage: S3 sink for historical analysis and compliance

Pipeline Configuration: | Component | Configuration | Purpose | |-----------|---------------|---------| | Topics | 3 (transactions, alerts, enriched) | Data organization | | Partitions | 12 (3 brokers × 4) | Parallelism | | Replication | 3 | High availability | | Compression | LZ4 | Throughput optimization |

Key Logic:

• Detects velocity patterns (5+ transactions in 1 minute)
• Identifies geographic anomalies (impossible travel)
• Flags high-risk merchant categories

Results:

• 99.7% of fraud detected in under 100ms
• False positive rate reduced from 5% to 0.3%
• Compliance audit passed with zero findings

Example 2: E-Commerce Order Processing System

Scenario: Build a resilient order processing system with Kafka for high reliability.

System Design:

1. Order Events: Topic for order lifecycle events
2. Inventory Service: Consumes orders, updates stock
3. Payment Service: Processes payments, publishes results
4. Notification Service: Sends confirmations via email/SMS

Resilience Patterns:

• Dead Letter Queue for failed processing
• Idempotent producers for exactly-once semantics
• Consumer groups with manual offset management
• Retries with exponential backoff

Configuration:

# Producer Configuration
acks: all
retries: 3
enable.idempotence: true

# Consumer Configuration
auto.offset.reset: earliest
enable.auto.commit: false
max.poll.records: 500

Results:

• 99.99% message delivery reliability
• Zero duplicate orders in 6 months
• Peak processing: 10,000 orders/second

Example 3: IoT Telemetry Platform

Scenario: Process millions of IoT device telemetry messages with Kafka.

Platform Architecture:

1. Device Gateway: MQTT to Kafka proxy
2. Data Enrichment: Stream processing adds device metadata
3. Time-Series Storage: S3 sink partitioned by device_id/date
4. Real-Time Alerts: Threshold-based alerting for anomalies

Scalability Configuration:

• 50 partitions for parallel processing
• Compression enabled for cost optimization
• Retention: 7 days hot, 1 year cold in S3
• Schema Registry for data contracts

Performance Metrics: | Metric | Value | |--------|-------| | Throughput | 500,000 messages/sec | | Latency (P99) | 50ms | | Consumer lag | < 1 second | | Storage efficiency | 60% reduction with compression |

Best Practices

Topic Design

• Naming Conventions: Use clear, hierarchical topic names (domain.entity.event)
• Partition Strategy: Plan for future growth (3x expected throughput)
• Retention Policies: Match retention to business requirements
• Cleanup Policies: Use delete for time-based, compact for state
• Schema Management: Enforce schemas via Schema Registry

Producer Optimization

• Batching: Increase batch.size and linger.ms for throughput
• Compression: Use LZ4 for balance of speed and size
• Acks Configuration: Use all for reliability, 1 for latency
• Retry Strategy: Implement retries with backoff
• Idempotence: Enable for exactly-once semantics in critical paths

Consumer Best Practices

• Offset Management: Use manual commit for critical processing
• Batch Processing: Increase max.poll.records for efficiency
• Rebalance Handling: Implement graceful shutdown
• Error Handling: Dead letter queues for poison messages
• Monitoring: Track consumer lag and processing time

Security Configuration

• Encryption: TLS for all client-broker communication
• Authentication: SASL/SCRAM or mTLS for production
• Authorization: ACLs with least privilege principle
• Quotas: Implement client quotas to prevent abuse
• Audit Logging: Log all access and configuration changes

Performance Tuning

• Broker Configuration: Optimize for workload type (throughput vs latency)
• JVM Tuning: Heap size and garbage collector selection
• OS Tuning: File descriptor limits, network settings
• Monitoring: Metrics for throughput, latency, and errors
• Capacity Planning: Regular review and scaling assessment

Security:

• [ ] Encryption: TLS enabled for Client-Broker and Inter-broker.
• [ ] Auth: SASL/SCRAM or mTLS enabled.
• [ ] ACLs: Principle of least privilege (Topic read/write).