itsmostafa/sqs

AWS SQS

Amazon Simple Queue Service (SQS) is a fully managed message queuing service for decoupling and scaling microservices, distributed systems, and serverless applications.

Table of Contents

• Core Concepts
• Common Patterns
• CLI Reference
• Best Practices
• Troubleshooting
• References

Core Concepts

Queue Types

| Type | Description | Use Case | |------|-------------|----------| | Standard | At-least-once, best-effort ordering | High throughput | | FIFO | Exactly-once, strict ordering | Order-sensitive processing |

Key Settings

| Setting | Description | Default | |---------|-------------|---------| | Visibility Timeout | Time message is hidden after receive | 30 seconds | | Message Retention | How long messages are kept | 4 days (max 14) | | Delay Seconds | Delay before message is available | 0 | | Max Message Size | Maximum message size | 256 KB |

Dead-Letter Queue (DLQ)

Queue for messages that failed processing after maxReceiveCount attempts.

Common Patterns

Create a Standard Queue

AWS CLI:

aws sqs create-queue \
  --queue-name my-queue \
  --attributes '{
    "VisibilityTimeout": "60",
    "MessageRetentionPeriod": "604800",
    "ReceiveMessageWaitTimeSeconds": "20"
  }'

boto3:

import boto3

sqs = boto3.client('sqs')

response = sqs.create_queue(
    QueueName='my-queue',
    Attributes={
        'VisibilityTimeout': '60',
        'MessageRetentionPeriod': '604800',
        'ReceiveMessageWaitTimeSeconds': '20'  # Long polling
    }
)
queue_url = response['QueueUrl']

Create FIFO Queue

aws sqs create-queue \
  --queue-name my-queue.fifo \
  --attributes '{
    "FifoQueue": "true",
    "ContentBasedDeduplication": "true"
  }'

Configure Dead-Letter Queue

# Create DLQ
aws sqs create-queue --queue-name my-queue-dlq

# Get DLQ ARN
DLQ_ARN=$(aws sqs get-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue-dlq \
  --attribute-names QueueArn \
  --query 'Attributes.QueueArn' --output text)

# Set redrive policy on main queue
aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue \
  --attributes "{
    \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"${DLQ_ARN}\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\"
  }"

Send Messages

import boto3
import json

sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'

# Send single message
sqs.send_message(
    QueueUrl=queue_url,
    MessageBody=json.dumps({'order_id': '12345', 'action': 'process'}),
    MessageAttributes={
        'MessageType': {
            'DataType': 'String',
            'StringValue': 'Order'
        }
    }
)

# Send to FIFO queue
sqs.send_message(
    QueueUrl='https://sqs.us-east-1.amazonaws.com/123456789012/my-queue.fifo',
    MessageBody=json.dumps({'order_id': '12345'}),
    MessageGroupId='order-12345',
    MessageDeduplicationId='unique-id-12345'
)

# Batch send (up to 10 messages)
sqs.send_message_batch(
    QueueUrl=queue_url,
    Entries=[
        {'Id': '1', 'MessageBody': json.dumps({'id': 1})},
        {'Id': '2', 'MessageBody': json.dumps({'id': 2})},
        {'Id': '3', 'MessageBody': json.dumps({'id': 3})}
    ]
)

Receive and Process Messages

import boto3
import json

sqs = boto3.client('sqs')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-queue'

while True:
    # Long polling (wait up to 20 seconds)
    response = sqs.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=10,
        WaitTimeSeconds=20,
        MessageAttributeNames=['All'],
        AttributeNames=['All']
    )

    messages = response.get('Messages', [])

    for message in messages:
        try:
            body = json.loads(message['Body'])
            print(f"Processing: {body}")

            # Process message...

            # Delete on success
            sqs.delete_message(
                QueueUrl=queue_url,
                ReceiptHandle=message['ReceiptHandle']
            )
        except Exception as e:
            print(f"Error processing message: {e}")
            # Message will become visible again after visibility timeout

Lambda Integration

# Create event source mapping
aws lambda create-event-source-mapping \
  --function-name my-function \
  --event-source-arn arn:aws:sqs:us-east-1:123456789012:my-queue \
  --batch-size 10 \
  --maximum-batching-window-in-seconds 5

Lambda handler:

def handler(event, context):
    for record in event['Records']:
        body = json.loads(record['body'])
        message_id = record['messageId']

        try:
            process_message(body)
        except Exception as e:
            # Raise to put message back in queue
            raise

    return {'batchItemFailures': []}

CLI Reference

Queue Management

| Command | Description | |---------|-------------| | aws sqs create-queue | Create queue | | aws sqs delete-queue | Delete queue | | aws sqs list-queues | List queues | | aws sqs get-queue-url | Get queue URL by name | | aws sqs get-queue-attributes | Get queue settings | | aws sqs set-queue-attributes | Update queue settings |

Messaging

| Command | Description | |---------|-------------| | aws sqs send-message | Send single message | | aws sqs send-message-batch | Send up to 10 messages | | aws sqs receive-message | Receive messages | | aws sqs delete-message | Delete message | | aws sqs delete-message-batch | Delete up to 10 messages | | aws sqs purge-queue | Delete all messages |

Visibility

| Command | Description | |---------|-------------| | aws sqs change-message-visibility | Change timeout | | aws sqs change-message-visibility-batch | Batch change |

Best Practices

Message Processing

• Use long polling (WaitTimeSeconds=20) to reduce API calls
• Delete messages promptly after successful processing
• Configure appropriate visibility timeout (> processing time)
• Implement idempotent consumers for at-least-once delivery

Dead-Letter Queues

• Always configure DLQ for production queues
• Set appropriate maxReceiveCount (usually 3-5)
• Monitor DLQ depth with CloudWatch alarms
• Process DLQ messages manually or with automation

FIFO Queues

• Use message group IDs to partition ordering
• Enable content-based deduplication or provide dedup IDs
• Throughput: 300 msgs/sec without batching, 3000 with

Security

• Use queue policies to control access
• Enable encryption with SSE-SQS or SSE-KMS
• Use VPC endpoints for private access

Troubleshooting

Messages Not Being Received

Causes:

• Short polling returning empty
• All messages in flight (visibility timeout)
• Messages delayed (DelaySeconds)

Debug:

# Check queue attributes
aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names All

# Check approximate message counts
aws sqs get-queue-attributes \
  --queue-url $QUEUE_URL \
  --attribute-names \
    ApproximateNumberOfMessages,\
    ApproximateNumberOfMessagesNotVisible,\
    ApproximateNumberOfMessagesDelayed

Messages Going to DLQ

Causes:

• Processing errors
• Visibility timeout too short
• Consumer not deleting messages

Redrive from DLQ:

# Enable redrive allow policy on source queue
aws sqs set-queue-attributes \
  --queue-url $MAIN_QUEUE_URL \
  --attributes '{"RedriveAllowPolicy": "{\"redrivePermission\":\"allowAll\"}"}'

# Start redrive
aws sqs start-message-move-task \
  --source-arn arn:aws:sqs:us-east-1:123456789012:my-queue-dlq \
  --destination-arn arn:aws:sqs:us-east-1:123456789012:my-queue

Duplicate Processing

Solutions:

• Use FIFO queues for exactly-once
• Implement idempotency in consumer
• Track processed message IDs in database

Lambda Not Processing

# Check event source mapping
aws lambda list-event-source-mappings \
  --function-name my-function

# Check for errors
aws lambda get-event-source-mapping \
  --uuid <mapping-uuid>

References

• SQS Developer Guide
• SQS API Reference
• SQS CLI Reference
• boto3 SQS