eliferjunior/aws-sqs

AWS SQS

Amazon Simple Queue Service (SQS) is a fully managed message queuing service for decoupling microservices. It offers two queue types: Standard (best-effort ordering, at-least-once delivery) and FIFO (exactly-once, ordered).

Core Concepts

• Standard Queue — unlimited throughput, at-least-once delivery, best-effort ordering
• FIFO Queue — exactly-once processing, strict ordering, 3000 msg/s with batching
• Visibility Timeout — period a message is hidden after being received
• Dead-Letter Queue (DLQ) — destination for messages that fail processing
• Message Group ID — FIFO ordering key for parallel processing within a queue
• Long Polling — reduces empty responses and API costs

Creating Queues

# Create a standard queue
aws sqs create-queue \
  --queue-name order-processing \
  --attributes '{
    "VisibilityTimeout": "60",
    "MessageRetentionPeriod": "1209600",
    "ReceiveMessageWaitTimeSeconds": "20"
  }'

# Create a FIFO queue (name must end in .fifo)
aws sqs create-queue \
  --queue-name order-processing.fifo \
  --attributes '{
    "FifoQueue": "true",
    "ContentBasedDeduplication": "true",
    "VisibilityTimeout": "60"
  }'

Dead-Letter Queues

# Create the DLQ first
aws sqs create-queue --queue-name order-processing-dlq

# Get DLQ ARN
DLQ_ARN=$(aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing-dlq \
  --attribute-names QueueArn --query 'Attributes.QueueArn' --output text)

# Configure main queue to use DLQ after 3 failed attempts
aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing \
  --attributes '{
    "RedrivePolicy": "{\"deadLetterTargetArn\":\"'$DLQ_ARN'\",\"maxReceiveCount\":\"3\"}"
  }'

# Redrive messages from DLQ back to source queue
aws sqs start-message-move-task \
  --source-arn "$DLQ_ARN" \
  --destination-arn "arn:aws:sqs:us-east-1:123456789:order-processing"

Sending Messages

# Send a single message
aws sqs send-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing \
  --message-body '{"orderId":"12345","action":"process"}' \
  --message-attributes '{
    "OrderType": {"DataType":"String","StringValue":"premium"}
  }'

# Send to FIFO queue with group ID
aws sqs send-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing.fifo \
  --message-body '{"orderId":"12345","action":"process"}' \
  --message-group-id "customer-789" \
  --message-deduplication-id "order-12345-v1"

Batch Operations

# Send up to 10 messages in a batch
aws sqs send-message-batch \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing \
  --entries '[
    {"Id":"1","MessageBody":"{\"orderId\":\"001\"}"},
    {"Id":"2","MessageBody":"{\"orderId\":\"002\"}"},
    {"Id":"3","MessageBody":"{\"orderId\":\"003\"}"}
  ]'

# Batch consumer with boto3
import boto3
import json

sqs = boto3.client('sqs')
QUEUE_URL = 'https://sqs.us-east-1.amazonaws.com/123456789/order-processing'

def poll_and_process():
    while True:
        response = sqs.receive_message(
            QueueUrl=QUEUE_URL,
            MaxNumberOfMessages=10,  # batch up to 10
            WaitTimeSeconds=20,       # long polling
            MessageAttributeNames=['All']
        )
        messages = response.get('Messages', [])
        if not messages:
            continue

        entries_to_delete = []
        for msg in messages:
            try:
                body = json.loads(msg['Body'])
                process_order(body)
                entries_to_delete.append({
                    'Id': msg['MessageId'],
                    'ReceiptHandle': msg['ReceiptHandle']
                })
            except Exception as e:
                print(f"Failed: {e}")
                # Message returns to queue after visibility timeout

        if entries_to_delete:
            sqs.delete_message_batch(
                QueueUrl=QUEUE_URL,
                Entries=entries_to_delete
            )

Receiving and Deleting

# Receive messages with long polling
aws sqs receive-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing \
  --max-number-of-messages 5 \
  --wait-time-seconds 20 \
  --message-attribute-names All

# Delete a processed message
aws sqs delete-message \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing \
  --receipt-handle "AQEBzL..."

# Change visibility timeout for a message needing more processing time
aws sqs change-message-visibility \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing \
  --receipt-handle "AQEBzL..." \
  --visibility-timeout 120

Monitoring

# Check queue depth
aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing \
  --attribute-names ApproximateNumberOfMessages,ApproximateNumberOfMessagesNotVisible,ApproximateNumberOfMessagesDelayed

# Purge all messages (use with caution)
aws sqs purge-queue \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789/order-processing

Best Practices

• Always enable long polling (WaitTimeSeconds=20) to reduce costs
• Set up dead-letter queues on every queue to catch poison messages
• Use FIFO queues only when ordering matters — standard queues have much higher throughput
• Process and delete messages in batches for efficiency
• Set visibility timeout to 6x your average processing time
• Use message group IDs in FIFO queues for parallel ordered processing
• Monitor ApproximateNumberOfMessages and alert on queue depth spikes
• Implement idempotent consumers — messages may be delivered more than once