Adding a DynamoDB TTL with step functions

Step Functions are a powerful service in AWS, they allow us orchestrate state machines to shift complexity from our lambdas onto AWS Infrastructure. We can manipulate the data flowing through the state machines using JSONPath and ASL(Amazon States Language). Unfortunately neither of these have an extremely rich ecosystem of samples online. So finding a solution for a problem that may seem quite simple can be frustrating.

The Use Case

One use case for a Step Function I found recently was picking up events from an EventBus and storing them in an Auditing/Tracing table. The events should only reside in the table for 30 days, so a DynamoDB TTL needs to be enabled.

The events coming would look something like this:

{
    "detail-type": "order-created",
    "source": "orders-service",
    "detail": {
        "metadata": {
            "created_at": 1682259710,
            "immutable": {
                "correlation_id": "4199fb72-e6f7-4c95-a9b3-bf498b92e7df"
            }
        },
        "data" : {
            "id": "14175108091"
        }
    },
    "time" : "2023-04-14T14:21:50Z"
}

The Problem

The DynamoDB Put action requires the payload into AttributeValue map format, and the TTL needs to be 30 days after the current time.

There are a few issues with this:

• The mapping template does not have an intrinsic function for getting the current Unix Epoch.
• The execution context does have current time in ISO-8601 but we do not have a Utility to convert it to Unix Epoch.

The Solution

Using the detail.metadata.created_at timestamp from our events, we can transform this into a Unix Epoch that is 30 days in the future by composing some intrinsic functions.

To map detail.metadata.created_at to an expire_at attribute we can do the following:

States.Format('{}',States.MathAdd($.detail.metadata.created_at, 2592000))

Lets break this down into steps:

• $.detail.metadata.created_at extracts the timestamp field.
• 2592000 is 30 days in seconds.
• States.MathAdd($.detail.metadata.created_at, 2592000) adds 30 days in seconds to our expires at timestamp.
• States.Format('{}',...) and converts our integer into a string

Putting this all together our DynamoDB Put action definition would look something like this:

{
  "TableName": "audit-table",
  "Item": {
    "PK": {
      "S.$": "$.detail.metadata.correlation_id"
    },
    "SK": {
      "S.$": "$.detail-type"
    },
    "data": {
      "M.$": "$.detail.data"
    },
    "metadata": {
      "M.$": "$.detail.metadata"
    },
    "expires_at": {
      "N.$": "States.Format('{}',States.MathAdd($.detail.metadata.created_at, 2592000))"
    }
  }
}

Closing thoughts

I believe Step Functions could have the ability to replace most lambda workflows, but before that is possible a more mature mapping toolkit is required. I would like to see AWS supply more intrinsic functions to improve the developer experience of working with step functions. For example:

• Timestamp manipulation and creation:
  • States.FormatDate("YYYY-MM-dd", $.date)
  • States.Timestamp()
  • States.UnixEpoch()
• AttributeValue map conversion:
  • States.ToAttributeValueMap($.json)
  • States.FromAttributeValueMap($.dynamodb_item)
• Additional Math functions:
  • States.MathFloor($.float_value)
  • States.MathRound($.float_value)
  • States.MathRound($.float_value)
  • States.MathMultiply($.float_value, 10)
  • States.MathDivide($.float_value, 1000)
• Json manipulation:
  • States.JsonRemoveKey($.json.key_to_remove)