Configuring your Cluster to Use Cadence Archival

Creating Cadence Clusters with Cadence Archival on AWS

Cadence Archival is a tiered storage feature of Cadence which enables workflow histories and visibility records to be stored in AWS S3 buckets following a retention period in your primary datastore cluster. More information about the usefulness and capabilities of this feature can be found in the official Cadence documentation, here: https://cadenceworkflow.io/docs/concepts/archival/ . This documentation will guide you through the process of enabling Cadence Archival on your Instaclustr managed Cadence cluster.

Please note that as a consequence of enabling archival, your cluster’s workflow histories and/or visibility records will be retained indefinitely in an S3 bucket within your AWS account. For this reason, before proceeding, we highly recommend implementing the following best practices to minimise risk to your business’s data:

• In keeping with the official advice from the Cadence project, workflows should never operate on plaintext sensitive data or personally identifiable information (PII). Please ensure that your workflows do not operate on unencrypted sensitive data or PII.

• Do not use your created S3 bucket for non-Archival purposes (i.e. storing Cadence-unrelated data).

• We recommend enabling access logging on your Archival S3 bucket. Directions for doing so can be found here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html .

• We recommend enabling Object versioning on your Archival S3 bucket. This will either be done by default if creating a bucket using our CloudFormation template for Cadence clusters in Instaclustr’s account, or should be configured by you during bucket creation for Cadence clusters in your own AWS account.

Storage pre-setup for Cadence Clusters in Instaclustr’s account.

In this section we will be walking through utilizing an AWS CloudFormation template to provision a stack containing the required S3 bucket, Access Policy, User and User Access Keys to leverage Cadence Archival on your cluster. Note that this section of the guide is for Cadence clusters run in Instaclustr’s AWS account; see the section below if running in your own account.

1. Create a CloudFormation template file instaclustr_cadence_archival_aws with the below content.

1. Please log into your AWS console and access https://console.aws.amazon.com/cloudformation
2. Click on the Create Stack dropdown and select With new resources option
3. In the create stack view select  Template is Ready for the Prerequisite – Prepare template section
4. In the ‘Create Stack’ view select Upload a template file for the Specify template section
5. Upload the previously created instaclustr_cadence_archival_aws CloudFormation template file and enter your choice of resource prefix that would be used for naming the resources created.
6. Follow the rest of the create stack wizard according to the requirements of your account.
7. Once the stack creation is complete you will be able to access the S3 bucket name, the created user access key, and the secret key required. This will be in the CloudFormation stack’s  ‘Output’ view.

Storage pre-setup for Cadence Clusters in your own account.

In this documentation we will be walking through leveraging Cadence Archival on your cluster. Note that this section of the guide is for Cadence clusters run in your own AWS account; see the section above if running in Instaclustr’s account.

1. Please log into your AWS console and access https://s3.console.aws.amazon.com/s3/buckets
2. Follow the standard process to provision an S3 bucket as your archival storage in the AWS region of your choosing. Please enable Object versioning on your bucket at this stage. There are no other specific requirements in terms of permissions or access configuration that we require during the bucket configuration process.
3. Make a note of the bucket name and region for your reference when provisioning your Cadence cluster.

Provisioning an Archival-enabled Cadence Cluster

1. Navigate to the Instaclustr console at https://console2.instaclustr.com
2. Follow our guide at https://www.instaclustr.com/support/documentation/cadence/getting-started-with-cadence/creating-a-cadence-cluster/ to get started provisioning a Cadence cluster. The specific steps that you will need to take in order to enable Cadence Archival on this cluster are the following:
   1. On the ‘Options’ page of the Cadence creation wizard, select ‘Enable Cadence Archival’
      
      
   2. On the ‘Data Centre’ page of the Cadence creation wizard, you will be asked to enter the following information to configure your archival setup.
      • The URI that you wish to target for your archival. This is formed by the bucket file path prefixed by ‘s3://’. For example, if you have generated an S3 bucket called example-bucket, you might wish to target the URI s3://example-bucket. Or, if you have a directory in that bucket called ‘my-cluster’, you might target the URI s3://example-bucket/my-cluster. If running in Instaclustr’s account, the name of your bucket can be retrieved from the ‘Outputs’ tab of your CloudFormation stack view.
      • The region that your S3 resource was created in, in lowercase and hyphen-separated e.g. us-east-1, us-west-2, etc. 
      • Clusters in Instaclustr’s account only: The AWS Access Key ID of the created stack’s IAM user (available in the stack ‘Outputs’ tab).
      • Clusters in Instaclustr’s account only: The AWS Secret Access Key of the created stack’s IAM user (available in the stack ‘Outputs’ tab).
        

Utilising Archival

Once your archival-enabled Cadence cluster is in a running state, simply create a domain with history and/or visibility archival enabled in order to begin leveraging this feature, like so.