⚙️ AWS CLI Configuration
Setting up SNS
1. Create SNS Topic
aws sns create-topic --name OrderNotifications
This command creates a new SNS topic named "OrderNotifications". The topic serves as a communication channel where publishers can send messages and subscribers can receive them. AWS returns a TopicArn that uniquely identifies this topic.
2. Create FIFO Topic (for ordered messages)
aws sns create-topic --name OrderNotifications.fifo --attributes FifoTopic=true,ContentBasedDeduplication=true
Creates a FIFO (First-In-First-Out) topic that ensures message ordering and exactly-once delivery. The .fifo suffix is required. ContentBasedDeduplication=true automatically generates deduplication IDs based on message content to prevent duplicate messages.
3. List Topics
aws sns list-topics
Retrieves all SNS topics in your AWS account for the current region. Returns topic ARNs and basic metadata for each topic.
4. Subscribe Email to Topic
aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:OrderNotifications --protocol email --notification-endpoint user@example.com
Creates an email subscription to the topic. AWS will send a confirmation email to the specified address. The user must click the confirmation link to activate the subscription. Replace the topic-arn with your actual topic ARN.
5. Subscribe SQS Queue to Topic
aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:OrderNotifications --protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:123456789012:OrderProcessingQueue
Subscribes an SQS queue to receive messages from the SNS topic. This enables the fan-out pattern where a single SNS message can trigger multiple SQS queues for parallel processing. Ensure the SQS queue policy allows SNS to deliver messages.
6. Publish Message to Topic
aws sns publish --topic-arn arn:aws:sns:us-east-1:123456789012:OrderNotifications --message "New order received: Order #12345" --subject "Order Notification"
Publishes a message to all subscribers of the topic. The message is delivered immediately to all active subscriptions. The subject parameter is used for email and SMS protocols to provide a message header.
7. Publish with Message Attributes (for filtering)
aws sns publish --topic-arn arn:aws:sns:us-east-1:123456789012:OrderNotifications --message "Premium order received" --message-attributes '{"OrderType":{"DataType":"String","StringValue":"Premium"},"Amount":{"DataType":"Number","StringValue":"150.00"}}'
Publishes a message with custom attributes that can be used for message filtering. Subscribers can filter messages based on these attributes to receive only relevant messages, reducing processing overhead and improving efficiency.
Setting up SQS
1. Create Standard SQS Queue
aws sqs create-queue --queue-name OrderProcessingQueue
Creates a standard SQS queue with default settings. Standard queues offer unlimited throughput and at-least-once delivery but don't guarantee message ordering. They're ideal for high-volume, parallel processing scenarios.
2. Create FIFO Queue
aws sqs create-queue --queue-name OrderProcessingQueue.fifo --attributes FifoQueue=true,ContentBasedDeduplication=true
Creates a FIFO queue that maintains strict message ordering and provides exactly-once processing. The .fifo suffix is mandatory. ContentBasedDeduplication prevents duplicate messages based on message content without requiring explicit deduplication IDs.
3. Configure Queue Attributes
aws sqs set-queue-attributes --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/OrderProcessingQueue --attributes VisibilityTimeoutSeconds=300,MessageRetentionPeriod=1209600,MaxReceiveCount=3
Configures important queue parameters: VisibilityTimeoutSeconds (300) sets how long messages remain invisible after being received, MessageRetentionPeriod (1209600 = 14 days) determines how long messages stay in the queue, and MaxReceiveCount (3) sets retry attempts before moving to dead letter queue.
4. Create Dead Letter Queue
aws sqs create-queue --queue-name OrderProcessingQueue-DLQ
Creates a dead letter queue to capture messages that fail processing after the maximum number of retry attempts. This prevents message loss and allows for manual investigation and reprocessing of problematic messages.
5. Configure Dead Letter Queue
aws sqs set-queue-attributes --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/OrderProcessingQueue --attributes '{"RedrivePolicy":"{\"deadLetterTargetArn\":\"arn:aws:sqs:us-east-1:123456789012:OrderProcessingQueue-DLQ\",\"maxReceiveCount\":3}"}'
Links the main queue to its dead letter queue with a redrive policy. Messages that are received 3 times without being successfully deleted are automatically moved to the dead letter queue for troubleshooting.
6. Send Message to Queue
aws sqs send-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/OrderProcessingQueue --message-body "Process order #12345 for customer John Doe"
Sends a message directly to the SQS queue. The message will remain in the queue until a consumer receives and processes it. For FIFO queues, you would also need to specify MessageGroupId and optionally MessageDeduplicationId.
7. Receive Messages
aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/OrderProcessingQueue --max-number-of-messages 10 --wait-time-seconds 20
Retrieves up to 10 messages from the queue using long polling (wait-time-seconds=20). Long polling reduces the number of API calls and provides more efficient message retrieval by waiting for messages to arrive if the queue is empty.
8. Delete Processed Message
aws sqs delete-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/OrderProcessingQueue --receipt-handle "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a..."
Removes a message from the queue after successful processing. The receipt-handle is provided when receiving the message and acts as a temporary identifier. If you don't delete the message, it will become visible again after the visibility timeout expires.
Queue Policy for SNS Integration
aws sqs set-queue-attributes --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/OrderProcessingQueue --attributes '{
"Policy": "{
\"Version\": \"2012-10-17\",
\"Statement\": [{
\"Effect\": \"Allow\",
\"Principal\": \"*\",
\"Action\": \"sqs:SendMessage\",
\"Resource\": \"arn:aws:sqs:us-east-1:123456789012:OrderProcessingQueue\",
\"Condition\": {
\"ArnEquals\": {
\"aws:SourceArn\": \"arn:aws:sns:us-east-1:123456789012:OrderNotifications\"
}
}
}]
}"
}'
Sets a queue policy that allows the SNS topic to send messages to the SQS queue. This is essential for SNS-to-SQS integration. The policy grants SendMessage permission only to the specific SNS topic, ensuring security through the SourceArn condition.
Message Filtering Setup
aws sns set-subscription-attributes --subscription-arn arn:aws:sns:us-east-1:123456789012:OrderNotifications:12345678-1234-1234-1234-123456789012 --attribute-name FilterPolicy --attribute-value '{"OrderType":["Premium","VIP"],"Amount":[{"numeric":[">=",100]}]}'
Configures message filtering for a subscription. This filter ensures the subscriber only receives messages where OrderType is "Premium" or "VIP" AND the Amount is greater than or equal to 100. Message filtering reduces unnecessary message processing and improves system efficiency.