Overcome X-Ray’s issue for Lambda debugging

Overcome X-Ray’s issue for Lambda debugging

01 April, 2019
Overcome X-Ray’s issue for Lambda debugging

Tracing is one of the pillars of Observebility, according to Twitter. AWS X-Ray helps you to get an overview of your distributed application, to find bottlenecks and to tune its critical path. Also, it provides you with the ability to trace your requests, so you can see which journey, any single request goes through when it enters your application. AWS X-Ray can potentially be a powerful tool for debugging your serverless application, however it’s still immature and there is a lot to improve. One of them is about its debugging ability and how it shows the errors.

I’m going to explain my point it with a simple serverless application as following:

Architecture of my example application

Lambda function downloads an image, saves it into a s3 bucket and records context.awsRequestId into a Dynamodb table. However, the latter part is an error prone operation: randomly the Lambda function chooses a repetitive partition key and subsequently Dynamodb throws ConditionalCheckFailedException. If an error is emitted directly from an AWS resource, normally it is shown in the Exceptions section of the subsegment.

But this is not always the case; X-Ray doesn’t propagate your custom errors into the trace. These types of error can happen often in different parts of your app, and subsequently, you will see that your trace is error-free and the Exceptions part is empty, even though an error has occurred! This error-free trace doesn’t bring much help to you and for debugging, you should go through your logs. I contacted X-Ray support, they are aware of this problem and hopefully they will address this in future. However, I’ve come up with a workaround and currently, this is the only way to overcome this issue: You are able to propagate your custom error, if you create your custom subsegment! Then you can inject an error object or attach an annotation with it. Following is a snippet that you can use:

module.exports.handler = (event, context, callback) => {
  // Errors emiting directly from AWS reosurces, probably will be shown
  // in the X-Ray trace, however you might often see that your errors
  // can't be seen in X-Ray trace and you should look at logs for debugging
  // You can alleviate tihs problem by creating your custom subsegment
  // And enrich it with errors, in case exist, for better debugging
  AWSXRay.captureAsyncFunc('mycustomsub_segment', function(subsegment) {
    dynamodb.putItem(params).promise().then(() => {
    // some business logic. remember to close subsegment at the end of operaiton
      callback(null, response)
    }).catch(error => {
      //You can propagate your errors into the trace, for better debugging
      subsegment.addAnnotation("error", 'You can query traces based on their annotation');
      subsegment.addError(new Error('This custom error object will' +
      'be propagated into X-Ray trace'));
      //You must close the subsegment, eventually
      callback(null, response)

Now you are able to search for buggy traces with help of Filter Expression:

  1. You can filter out traces with specific annotation, for example: annotation.error=”customError 1"

  1. You can filter out traces which have downstream issue / error with help of Complex Keyword, for example: service(“service-name”) { error = true }

You can optionally leave the service-name empty.

This workaround has a drawback; you should always create an extra subsegment to embrace your error(s), this makes it to look like a dirty workaround. But let’s hope AWS address this issue soon!

Please note:

  1. X-Ray categorizes client side (4xx series errors) as Error, and server side errors (5xx series errors) as Fault.
    1. You can look into X-Ray SDK for Nodejs to find out different ways to create your subsegment

You can see my full conversation with X-Ray support teamin here.

Thanks for reading. Please let me know if you have any comment or question.