Originally developed and open sourced by Uber, Cadence® is a workflow engine that greatly simplifies the development of complex, long-running automated business processes at scale. NetApp offers Cadence on the Instaclustr Platform, providing fully automated provisioning and ongoing management of Cadence and its dependent technologies (Apache Cassandra®, Apache Kafka®, and OpenSearch®). This blog provides some background into one aspect of the behind–the–scenes implementation of our Developer Shared Infrastructure offering for Cadence.
Cadence is capable of scaling out horizontally to handle millions of concurrent workflows. However, sometimes it may be necessary to slow down the activities within Cadence services for various reasons like server maintenance and backend database maintenance. The scenario that we were facing was setting up a few small Cadence clusters, each using shared back-end services (Cassandra, Kafka, OpenSearch). We did not want a user of one Cadence cluster to be able to take more than their allocated share of the processing capacity of the shared back-end services.
Achieving this outcome was pretty simple with Cadence once we found the right settings to configure. These settings, found in the https://github.com/uber/cadence/blob/master/common/dynamicconfig/constants.go file, are listed below:
Set A – Internal Service Request Limits
1 2 3 4 5 6 7 | // Request rate per second for each frontend host/service // KeyName: frontend.rps // Value type: Int // Default value: 1200 |
1 2 3 4 5 6 7 | // Request rate per second for each matching host/service // KeyName: matching.rps // Value type: Int // Default value: 1200 |
1 2 3 4 5 6 7 | // Request rate per second for each history host/service // KeyName: history.rps // Value type: Int // Default value: 3000 |
Set B – Back End (Database) Request Limits
1 2 3 4 5 6 7 | // This is the max rate that frontend service can query DB // KeyName: frontend.persistenceMaxQPS // Value type: Int // Default value: 2000 |
1 2 3 4 5 6 7 | // This is the max rate that matching service can query DB // KeyName: matching.persistenceMaxQPS // Value type: Int // Default value: 3000 |
1 2 3 4 5 6 7 | // This is the max rate that history service can query DB // KeyName: history.persistenceMaxQPS // Value type: Int // Default value: 9000 |
1 2 3 4 5 6 7 | // This is the max rate that worker host can query DB // KeyName: worker.persistenceMaxQPS // Value type: Int // Default value: 500 |
Both of the sets above can be used to slow down internal Cadence operations by assigning smaller values to the respective properties. More specifically, Set A can be used to slow down request rates to Cadence internal services (i.e., frontend, matching and history), and Set B can be used to slow down request rates from Cadence internal services to the backend database (i.e., Cassandra).
If you like the sound of this but don’t want to go through the work of figuring out how to set them up yourself, Instaclustr’s Managed Cadence offering already supports this through its Development Shared Infrastructure offering.
Sign up today and experience it yourself with a free trial!