sawrus/areas/software/data-engineering/skills/streaming-patterns

Skill: Streaming Data Patterns

When to load

When designing Kafka consumers/producers or implementing real-time pipelines.

Producer Best Practices

producer = Producer({
    "bootstrap.servers": settings.KAFKA_BROKERS,
    "acks": "all",               # Wait for all replicas
    "retries": 5,
    "enable.idempotence": True,  # Exactly-once delivery
    "compression.type": "snappy",
})

Consumer Pattern

consumer = Consumer({
    "group.id": "order-processor-v1",  # Version when changing logic
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,        # Manual commit: only after successful processing
})

msg = consumer.poll(timeout=1.0)
try:
    process(deserialize(msg.value()))
    consumer.commit(msg)  # Commit only after success
except ProcessingError as e:
    publish_to_dlq(msg, e)  # Dead-letter, don't block partition
    consumer.commit(msg)

Dead Letter Queue

{
  "original_topic": "orders.created.v1",
  "original_offset": 12345,
  "original_payload": "...",
  "error_message": "Deserialization failed",
  "error_timestamp": "2026-02-16T10:30:00Z",
  "retry_count": 3
}