Originally developed and open sourced by Uber, Cadence® is a workflow engine that greatly simplifies the development of complex, long-running automated business processes at scale. Cadence needs to use OpenSearch/Elasticsearch when the Advanced Visibility feature is turned on. However, some default configurations in Cadence are chosen more for ease of setup than robust production operations. This blog post walks through one such setting that Instaclustr tunes when provisioning a Cadence cluster to ensure production reliability of your Cadence architecture.
To enable the Advanced Visibility feature on a Cadence cluster, an index that has a naming pattern like cadence-visibility-xxxxx needs to be created on the OpenSearch/Elasticsearch cluster. This is achieved in 2 steps:
- Creating an index template by using the schema file (called index_template.json) provided by Cadence.
- Creating the actual index by using the index template created in Step 1. Note, the index name must have the prefix cadence-visibility-.
The above 2 steps can be achieved by issuing 2 cURL commands demonstrated by the following code snippet (Snippet A):
1 2 3 4 5 6 7 8 9 10 11 12 13 |
SCHEMA_FILE=$CADENCE_HOME/schema/elasticsearch/$version/visibility/index_template.json URL="https://$host:$port/_template/cadence-visibility-template" curl -X PUT $URL -H 'Content-Type: application/json' --data-binary "@$SCHEMA_FILE" URL="https://$host:$port/cadence-visibility-production" curl -X PUT $URL |
Closer inspection of the index_template.json file shows that the required index has the number of replicas set to 0, i.e.:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
..................... "settings": { "index": { "number_of_shards": "5", "number_of_replicas": "0" } } ...................... |
The number of shards on an index determines how it can be distributed across the cluster. An index with 1 shard can only be stored on 1 node. An index with 5 shards could be split across 5 nodes. The number of replicas on an index determines how resilient that index is. OpenSearch/Elasticsearch will spread replicas across different nodes in the cluster.
If an index has zero replicas, stopping a node in the cluster will likely cause some documents to become unavailable (if that node holds shards for that index). This can lead to failure of Cadence search queries via the CLI or via Cadence Web. If the node goes down permanently, i.e., due to disk failure, some data for the index will be permanently lost.
If an index has at least 1 replica, then stopping a node in the cluster will not cause documents to become unavailable. Replicas take up about as much disk space as the original shard and have some performance overhead. However, if the availability of your data is at all important, it usually makes sense to have at least 1 replica. This high availability of data will, in turn, improve the reliability of the search queries made via Cadence CLI and Cadence Web.
To update this setting, i.e., set the number of replicas to 1, the following cURL command can be issued immediately after executing Snippet A:
1 2 3 |
URL="https://$host:$port/cadence-visibility-production/_settings" curl -X PUT $URL -H 'Content-Type: application/json' -d '{ "index": { "number_of_replicas" : "1" } }' |
All of Instaclustr’s managed Cadence offerings automatically have this configuration applied.