secondsky/cloudflare-queues

Cloudflare Queues

Status: Production Ready ✅ | Last Verified: 2025-12-27

Dependencies: cloudflare-worker-base (for Worker setup)

Contents: Quick Start • Critical Rules • Top Errors • Use Cases • When to Load References • Limits

---

Quick Start (10 Minutes)

1. Create Queue

bunx wrangler queues create my-queue
bunx wrangler queues list

2. Producer (Send Messages)

wrangler.jsonc:

{
  "name": "my-producer",
  "main": "src/index.ts",
  "queues": {
    "producers": [
      {
        "binding": "MY_QUEUE",
        "queue": "my-queue"
      }
    ]
  }
}

src/index.ts:

import { Hono } from 'hono';

type Bindings = {
  MY_QUEUE: Queue;
};

const app = new Hono<{ Bindings: Bindings }>();

app.post('/send', async (c) => {
  await c.env.MY_QUEUE.send({
    userId: '123',
    action: 'process-order',
    timestamp: Date.now(),
  });

  return c.json({ status: 'queued' });
});

export default app;

3. Consumer (Process Messages)

wrangler.jsonc:

{
  "name": "my-consumer",
  "main": "src/consumer.ts",
  "queues": {
    "consumers": [
      {
        "queue": "my-queue",
        "max_batch_size": 10,
        "max_retries": 3,
        "dead_letter_queue": "my-dlq"
      }
    ]
  }
}

src/consumer.ts:

import type { MessageBatch } from '@cloudflare/workers-types';

export default {
  async queue(batch: MessageBatch): Promise<void> {
    for (const message of batch.messages) {
      console.log('Processing:', message.body);
      // Your logic here
    }
    // Implicit ack: returning successfully acknowledges all messages
  },
};

Deploy:

bunx wrangler deploy

Load: references/setup-guide.md for complete 6-step setup with DLQ configuration

---

Critical Rules

Always Do ✅

1. Configure Dead Letter Queue for production queues
2. Use explicit ack() for non-idempotent operations (DB writes, API calls)
3. Validate message size before sending (<128 KB)
4. Use sendBatch() for multiple messages (more efficient)
5. Implement exponential backoff for retries
6. Set appropriate batch settings based on workload
7. Monitor queue backlog and consumer errors
8. Use ctx.waitUntil() for async cleanup in consumers
9. Handle errors gracefully - log, alert, retry
10. Let concurrency auto-scale (don't set max_concurrency unless needed)

Never Do ❌

1. Never assume message ordering - not guaranteed FIFO
2. Never rely on implicit ack for non-idempotent ops - use explicit ack()
3. Never send messages >128 KB - will fail
4. Never delete queues with active messages - data loss
5. Never skip DLQ configuration in production
6. Never exceed 5000 msg/s per queue - rate limit error
7. Never process messages synchronously in loop - use Promise.all()
8. Never ignore message.attempts - use for backoff logic
9. Never set max_concurrency=1 unless you have a very specific reason
10. Never forget to ack() in explicit acknowledgement patterns

---

Top 3 Critical Errors

Error #1: Message Too Large

Problem: Message exceeds 128 KB limit

Solution: Store large data in R2, send reference

// ❌ Wrong
await env.MY_QUEUE.send({ data: largeArray }); // >128 KB fails

// ✅ Correct
const message = { data: largeArray };
const size = new TextEncoder().encode(JSON.stringify(message)).length;

if (size > 128000) {
  const key = `messages/${crypto.randomUUID()}.json`;
  await env.MY_BUCKET.put(key, JSON.stringify(message));
  await env.MY_QUEUE.send({ type: 'large-message', r2Key: key });
} else {
  await env.MY_QUEUE.send(message);
}

Error #2: Throughput Exceeded

Problem: Exceeding 5000 messages/second per queue

Solution: Use sendBatch() and rate limiting

// ❌ Wrong
for (let i = 0; i < 10000; i++) {
  await env.MY_QUEUE.send({ id: i }); // Too fast!
}

// ✅ Correct
const messages = Array.from({ length: 10000 }, (_, i) => ({
  body: { id: i },
}));

// Send in batches of 100
for (let i = 0; i < messages.length; i += 100) {
  await env.MY_QUEUE.sendBatch(messages.slice(i, i + 100));
}

Error #3: Entire Batch Retried When One Message Fails

Problem: Single message failure causes all messages to retry

Solution: Use explicit acknowledgement

// ❌ Wrong - implicit ack
export default {
  async queue(batch: MessageBatch, env: Env): Promise<void> {
    for (const message of batch.messages) {
      await env.DB.prepare('INSERT INTO orders VALUES (?, ?)').bind(
        message.body.id,
        message.body.amount
      ).run();
    }
    // If any fails, ALL retry!
  },
};

// ✅ Correct - explicit ack
export default {
  async queue(batch: MessageBatch, env: Env): Promise<void> {
    for (const message of batch.messages) {
      try {
        await env.DB.prepare('INSERT INTO orders VALUES (?, ?)').bind(
          message.body.id,
          message.body.amount
        ).run();

        message.ack(); // Only ack on success
      } catch (error) {
        console.error(`Failed: ${message.id}`, error);
        // Don't ack - will retry independently
      }
    }
  },
};

Load references/error-catalog.md for all 10 errors including DLQ configuration, auto-scaling issues, message deletion prevention, and detailed solutions.

---

Common Use Cases

Use Case 1: Basic Message Queue

When: Simple async job processing (emails, notifications)

Quick Pattern:

// Producer
await env.MY_QUEUE.send({ type: 'email', to: '[email protected]' });

// Consumer (implicit ack - for idempotent operations)
export default {
  async queue(batch: MessageBatch): Promise<void> {
    for (const message of batch.messages) {
      await sendEmail(message.body.to, message.body.content);
    }
  },
};

Load: templates/queues-producer.ts + templates/queues-consumer-basic.ts

Use Case 2: Database Writes (Non-Idempotent)

When: Writing to database, must avoid duplicates

Load: templates/queues-consumer-explicit-ack.ts + references/consumer-api.md

Use Case 3: Retry with Exponential Backoff

When: Calling rate-limited APIs, temporary failures

Load: templates/queues-retry-with-delay.ts + references/error-catalog.md (Error #2, #3)

Use Case 4: Dead Letter Queue Pattern

When: Production systems, need to capture permanently failed messages

Load: templates/queues-dlq-pattern.ts + references/setup-guide.md (Step 4)

Use Case 5: High Throughput Processing

When: Processing thousands of messages per second

Quick Pattern:

{
  "queues": {
    "consumers": [{
      "queue": "my-queue",
      "max_batch_size": 100,        // Large batches
      "max_batch_timeout": 5,       // Fast processing
      "max_concurrency": null       // Auto-scale
    }]
  }
}

Load: references/best-practices.md → Optimizing Throughput

---

When to Load References

Load references/setup-guide.md when:

• User needs complete setup walkthrough (queue → producer → consumer → DLQ)
• First time setting up Cloudflare Queues
• Need production configuration examples
• Want complete end-to-end example

Load references/error-catalog.md when:

• Encountering any of the 10 documented errors
• Troubleshooting queue issues
• Messages not being delivered/processed
• Need prevention checklist

Load references/producer-api.md when:

• Need complete producer API reference
• Using send() or sendBatch() methods
• Need message format specifications
• Handling large messages or batches

Load references/consumer-api.md when:

• Need complete consumer API reference
• Using explicit ack(), retry(), or retryAll()
• Need batch processing patterns
• Implementing complex consumer logic

Load references/best-practices.md when:

• Optimizing queue performance
• Production deployment guidance
• Monitoring and observability setup
• Security and reliability patterns

Load references/wrangler-commands.md when:

• Need CLI commands reference
• Managing queues (create, delete, list)
• Controlling delivery (pause, resume)
• Debugging queue issues
• Real-time monitoring and performance analysis

Load references/typescript-types.md when:

• Need complete TypeScript type definitions
• Working with Queue, MessageBatch, or Message interfaces
• Implementing type-safe producers or consumers
• Using generic types for message bodies
• Need type guards for message validation

Load references/production-checklist.md when:

• Preparing for production deployment
• Need pre-deployment verification checklist
• Want detailed explanations of production best practices
• Setting up monitoring, DLQ, or error handling
• Planning load testing or security review

Load references/pull-consumers.md when:

• Need to consume messages from non-Workers environments
• Integrating with existing backend services (Node.js, Python, Go)
• Implementing pull-based polling instead of push-based consumption
• Working with containerized or legacy applications

Load references/http-publishing.md when:

• Publishing messages from external systems via HTTP
• Integrating webhooks from third-party services
• Need to send messages without deploying Workers
• Implementing CI/CD pipeline notifications

Load references/r2-event-integration.md when:

• Triggering queue messages on R2 bucket events
• Implementing automated image/document processing
• Setting up event-driven data pipelines
• Need R2 object upload/delete notifications

---

Agents & Commands

Available Agents:

• queue-debugger - 9-phase diagnostic analysis for queue issues (systematic troubleshooting)
• queue-optimizer - Performance tuning and cost optimization (batch size, concurrency, retries)

Available Commands:

• /queue-setup - Interactive wizard for complete queue setup
• /queue-troubleshoot - Quick diagnostic for common issues
• /queue-monitor - Real-time metrics and status display

---

Limits & Quotas

Critical limits:

• Message size: 128 KB max per message
• Throughput: 5,000 messages/second per queue
• Batch size: 100 messages max per sendBatch()
• Consumer CPU time: 30 seconds (default), 300 seconds (max with config)
• Max retries: Configurable (default 3)
• Queue name: Max 63 characters, lowercase/numbers/hyphens

Load references/best-practices.md for handling limits and optimization strategies.

---

Configuration Reference

Producer: Add queue binding to wrangler.jsonc queues.producers array with binding and queue fields.

Consumer: Configure in wrangler.jsonc queues.consumers array with queue, max_batch_size (1-100), max_batch_timeout (0-60s), max_retries, dead_letter_queue, and optionally max_concurrency (default: auto-scale).

CPU Limits: Increase limits.cpu_ms from default 30,000ms if processing takes longer.

Load references/setup-guide.md for complete configuration examples and templates/wrangler-queues-config.jsonc for production-ready config.

---

Using Bundled Resources

References (references/)

• setup-guide.md - Complete 6-step setup (queue → producer → consumer → DLQ → deploy → production config)
• error-catalog.md - All 10 errors with solutions + prevention checklist
• producer-api.md - Complete producer API (send, sendBatch, message format)
• consumer-api.md - Complete consumer API (ack, retry, batch processing)
• best-practices.md - Performance, monitoring, security, reliability patterns
• wrangler-commands.md - CLI reference (create, delete, list, pause, resume)
• typescript-types.md - Complete TypeScript type definitions for Queue, MessageBatch, Message
• production-checklist.md - Pre-deployment verification and best practices
• pull-consumers.md - Pull-based consumers for non-Workers environments (HTTP polling)
• http-publishing.md - Publishing messages via HTTP API from external systems
• r2-event-integration.md - R2 event notifications triggering queue messages

Templates (templates/)

• queues-producer.ts - Basic producer with single and batch sending
• queues-consumer-basic.ts - Implicit ack consumer (idempotent operations)
• queues-consumer-explicit-ack.ts - Explicit ack consumer (non-idempotent)
• queues-retry-with-delay.ts - Exponential backoff retry pattern
• queues-dlq-pattern.ts - Dead letter queue setup and consumer
• wrangler-queues-config.jsonc - Complete production configuration

---

TypeScript Types

Use @cloudflare/workers-types package for complete type definitions: Queue, MessageBatch<Body>, Message<Body>, QueueSendOptions.

Load references/typescript-types.md for complete type reference with interfaces, generics, type guards, and usage examples.

---

Monitoring & Debugging

Key Commands: wrangler queues info (status), wrangler tail (logs), wrangler queues pause-delivery/resume-delivery (control).

Load references/wrangler-commands.md for complete CLI reference with real-time monitoring, debugging workflows, and performance analysis commands.

---

Production Checklist

12-Point Pre-Deployment Checklist: DLQ configuration, message acknowledgment strategy, size validation, batch optimization, concurrency settings, CPU limits, error handling, monitoring, rate limiting, idempotency, load testing, and security review.

Load references/production-checklist.md for complete checklist with detailed explanations, code examples, and deployment workflow.

---

Related Skills

• cloudflare-worker-base - Worker setup and configuration
• cloudflare-d1 - Database integration for queue consumers
• cloudflare-r2 - Store large message payloads
• cloudflare-workflows - More complex orchestration needs

---

Official Documentation

• Cloudflare Queues: https://developers.cloudflare.com/queues/
• Configuration: https://developers.cloudflare.com/queues/configuration/
• Wrangler Commands: https://developers.cloudflare.com/workers/wrangler/commands/#queues
• Limits: https://developers.cloudflare.com/queues/platform/limits/
• Troubleshooting: https://developers.cloudflare.com/queues/observability/troubleshooting/
• Best Practices: https://developers.cloudflare.com/queues/configuration/best-practices/

---

Questions? Issues?

1. Check references/error-catalog.md for all 10 errors and solutions
2. Review references/setup-guide.md for complete setup walkthrough
3. See references/best-practices.md for production patterns
4. Check official docs: https://developers.cloudflare.com/queues/