chrismitchellonline

AWS SQS Message With Attributes

2019-12-12

In this post we’ll look at how to post messages to Amazon Web Services Simple Queue Service - AWS SQS , with NodeJS and the AWS SDK. We’ll look at how to post and recieve messages as well as how to add and read messages with attributes.

Prerequisits

To follow along you’ll need an active AWS account with an SQS queue to send messages too. I have a basic queue setup using the quick create feature in SQS called message-attributes. Additionally, you’ll need to be able to communicate with AWS using the AWS CLI tools installed on your development station.

A Note about AWS SQS

AWS SQS is an easy to use message queue service managed by AWS in the cloud. You can read more about it here: Amazon Simple Queue Service. Amazon offeres a free tier, up to one million messages per month. All tasks outlined in this post are free to play with.

Messages with Attributes

Typically when you send data to SQS you would send the data formatted as JSON data, a simple example would be sending an order number:

{"orderNumber":123456}

With message attributes we can attach more data to our message JSON data. This can be helpful to the system that reads queue messages to know more information about the message. You can even add binary data as message attributes, things like images, Excel files, etc. You can read more about attributes here: Amazon SQS Message Attributes.

Using the orderNumber message example, we’ll add add a string message attribute named orderType with a value of new as well as orderSent with a timestamp value.

Why Message Attributes?

Using binary message attributes are more apparent in their usefulness, but what about string message attributes? At this point one would argue:

“Why use string message attributes when you can include additional data in your JSON object?"

AWS documentation points out:

Your consumer can use message attributes to handle a message in a particular way without having to process the message body first.

Additionally I would like to point out a seperation of concerns. You can add data as message attributes about the message, instead of convoluting the message data itself.

Sending a Message

As mentioned I’ll be using NodeJS and the AWS SDK for working with AWS SQS. To send a message to the queue with attributes, we’ll need the following values, where MessageBody and QueueUrl are required fields:

{
    MessageBody: 'STRING_VALUE',
    QueueUrl: 'STRING_VALUE',
    MessageAttributes:{
        '<String>': {
            DataType: 'STRING_VALUE',
            StringValue: 'STRING_VALUE'
        }
    } 
}

Setting up my NodeJS index.js script with includes and AWS objects:

    var AWS = require('aws-sdk');
    AWS.config.update({region: 'us-west-2'});
    const SQS = new AWS.SQS({apiVersion: '2012-11-05'});

Then build our message body and queue url:

    //Message body is JSON to send
    const messageBody = {
        "orderNumber":123456
    }

    //queue url to post too. NOTE: Not ARN!
    const queueUrl = "https://sqs.us-west-2.amazonaws.com/XXX/message-attributes";

Finally setup our message attributes.

Setting Message Attributes

The JSON object required by Amazon for the message attributes is not an array, instead a comma seperated JSON object, for example:

MessageAttributes: {
    "Title": {
      DataType: "String",
      StringValue: "The Whistler"
    },
    "Author": {
      DataType: "String",
      StringValue: "John Grisham"
    }
}

The following code segment uses an array of message attributes messageAttribs and builds the required format object messageAttribsObj:

    //Array of string message attributes with name and value
    const messageAttribs = [
        {
            "name":"orderType",
            "value":"new"
        },
        {
            "name":"orderSent",
            "value":Date.now().toString()
        }
    ]
    //Build strange attributes object
    let messageAttribsObj = {};
    for (var i = 0; i < messageAttribs.length; i++) {
        messageAttribsObj[messageAttribs[i].name] = {};
        messageAttribsObj[messageAttribs[i].name].DataType = "String";
        messageAttribsObj[messageAttribs[i].name].StringValue = messageAttribs[i].value;
    }

Putting it all together in the SQS_PARAMS object:

    const SQS_PARAMS = {
        MessageBody: JSON.stringify(messageBody),
        QueueUrl: queueUrl,
        MessageAttributes:messageAttribsObj
    }

And finally use the AWS.SQS object to send your message:

SQS.sendMessage(SQS_PARAMS,function(err,result){
    if(err){
        console.log("Error sending SQS");
        console.log(err);
        return;
    }
    console.log(result);
})

If everything is successfull you will get a response from the object with metadata from your message:

{ ResponseMetadata: { RequestId: 'b796835c-5d03-58c1-a258-d530d5e9efe3' },
  MD5OfMessageBody: 'edf19b076ee08e28510eebd4b9ffe075',
  MD5OfMessageAttributes: '2efbcecd7f9869597d1907b4a0e01e0b',
  MessageId: '240007a3-b553-43ff-b12b-36e7e2cad314' }

Here is the final result of index.js:

var AWS = require('aws-sdk');
AWS.config.update({region: 'us-west-2'});
const SQS = new AWS.SQS({apiVersion: '2012-11-05'});

//Message body is JSON to send
const messageBody = {
    "orderNumber":123456
}

//queue url to post too. NOTE: Not ARN!
const queueUrl = "https://sqs.us-west-2.amazonaws.com/XXX/message-attributes";

//Array of string message attributes with name and value
const messageAttribs = [
    {
        "name":"orderType",
        "value":"new"
    },
    {
        "name":"orderSent",
        "value":Date.now().toString()
    }
]
//Build strange attributes object
let messageAttribsObj = {};
for (var i = 0; i < messageAttribs.length; i++) {
    messageAttribsObj[messageAttribs[i].name] = {};
    messageAttribsObj[messageAttribs[i].name].DataType = "String";
    messageAttribsObj[messageAttribs[i].name].StringValue = messageAttribs[i].value;
}

const SQS_PARAMS = {
    MessageBody: JSON.stringify(messageBody),
    QueueUrl: queueUrl,
    MessageAttributes:messageAttribsObj
}

SQS.sendMessage(SQS_PARAMS,function(err,result){
    if(err){
        console.log("Error sending SQS");
        console.log(err);
        return;
    }
    console.log(result);
})

Reading Messages with Attributes

Now we can read the message back from the queue. I’ve setup read.js similar to index.js with includes and AWS objects. Here is the updated SQS parameters object:

const queueUrl = "https://sqs.us-west-2.amazonaws.com/089357728632/message-attributes";
var SQS_PARAMS = {    
    QueueUrl: queueUrl,
    VisibilityTimeout: 20,
    WaitTimeSeconds: 10,
    MessageAttributeNames: [
        "All"
     ],
};

Where VisibilityTimeout is the amount of time a message will be unavailable on the queue because we are reading it here, WaitTimeSeconds is how long to wait for messages before returning an empty response, and MessageAttributeNames is an array of attribute names you want to include in the messages response. All will return all attributes.

A note about receiving messages: Your call to receive messages from the queue will not always return data, depending on message availablity. The message can be invisible due to others reading it, or because AWS SQS is distributed, not all messages make it to all endpoints. Typically if you hit the endpoint once or twice your messages will eventually show in your responses.

Sending the request and handling the response:

SQS.receiveMessage(SQS_PARAMS,function(err,result){
    if(err){
        console.log("Error reading SQS");
        console.log(err);
        return;
    }
    //Full result object including message metadata
    console.log(result);

    //Only the messages
    console.log(result.Messages)

    //List of attributes for an individual message [0]
    console.log(result.Messages[0].MessageAttributes);
})

The result object will contain response metadata and an array of messages (or empty if none were found). In the messages array you’ll see the MessageAttributes object for each message, where you can filter data accordingly. Each of these data points I’ve logged in the console as a demonstration.

Deleting Messages

Once a message is read and is processed, typically it needs to be deleted to avoid duplicate reads. As a side note, its also possible that you might not delete the message from the queue based on the message attributes, or any other conditions for that matter.

Extending our read code, delete the message only if orderType=“new”. To delete the message from the queue, use the ReceiptHandle for the specific message:

var deleteParams = {
      QueueUrl: queueURL,
      ReceiptHandle: data.Messages[0].ReceiptHandle
    };
    sqs.deleteMessage(deleteParams, function(err, data) {
      if (err) {
        console.log("Delete Error", err);
      } else {
        console.log("Message Deleted", data);
      }
});

The full read/delete code looks like this:

var AWS = require('aws-sdk');
AWS.config.update({region: 'us-west-2'});
const SQS = new AWS.SQS({apiVersion: '2012-11-05'});

//queue url to post too. NOTE: Not ARN!
const queueUrl = "https://sqs.us-west-2.amazonaws.com/xxx/message-attributes";
var SQS_PARAMS = {    
    QueueUrl: queueUrl,
    VisibilityTimeout: 20,
    WaitTimeSeconds: 10,
    MessageAttributeNames: [
        "All"
     ],
};

SQS.receiveMessage(SQS_PARAMS,function(err,result){
    if(err){
        console.log("Error reading SQS");
        console.log(err);
        return;
    }
    //Full result object
    console.log(result);

    //Only the messages
    console.log(result.Messages)

    //List of attributes for message [0]
    console.log(result.Messages[0].MessageAttributes);

    // Here is where we would process the messages, or leave it alone based on attributes, etc.

    //Now that we've read the message and have done things, decide if we would like to delete it or not
    //If the orderType is new, delete it from the queue
    if(result.Messages[0].MessageAttributes.orderType.StringValue==="new"){
        var deleteParams = {
            QueueUrl: queueUrl,
            ReceiptHandle: result.Messages[0].ReceiptHandle
        };
        SQS.deleteMessage(deleteParams, function(err, deleteResponse) {
            if (err) {
              console.log("Delete Error", err);
            } else {
              console.log("Message Deleted", deleteResponse);
            }
        });
    }
})

Conclusion

This article shows how we can read and write AWS SQS messages with not only JSON data but attach custom attributes as well. Using queues along with the extra attributes can be a powerful technique to move data around AWS in a clean and percise way.

Do you use queues in your current project, or need help getting started with AWS SQS? Send me an email and we can chat.

Questions or comments about this post, or queues in general? Use the Disqus comments section below.

comments powered by Disqus
Search
Social Media
Sponsor