The Instaclustr Prometheus Monitoring API allows users to retrieve metrics in Prometheus format for clusters running in the Instaclustr Managed Platform. The Prometheus API also implements Prometheus service discovery mechanisms to provide a discovery endpoint that can be used to discover newly created clusters and node changes to existing clusters without requiring Prometheus configs to be updated. Metrics scraped from Prometheus can then be integrated with visualization and alerting tools such as Grafana for a complete monitoring/alerting solution. The Instaclustr Cluster Prometheus Monitoring API can be used with any tool that supports the ingestion of Prometheus format metrics. The following steps walk through how to configure and use the API with Prometheus and Grafana.
You will need to configure prometheus.yml to discover your clusters. You can find example prometheus.yml configs for discovering your clusters by going to the Connection Info tab under any of your clusters and then selecting the Prometheus connection example. Here is an example prometheus.yml file:
Replace the fields surrounded by angle brackets such as <your_user_name> etc. Your username and Prometheus monitoring API key can be created in the Instaclustr console, in the Account tab. Make sure to copy your API key as it will only be displayed once.
To specify the clusters you would like Prometheus to scrape metrics from, replace <your-cluster-id> to include the cluster you would like to scrape. You can add additional URLs to http_sd_configs to scrape additional clusters. Alternatively, you can specify https://<your-account-id>.prometheus.monitoring.instaclustr.com/discovery/v1/ as a URL for http_sd_configs to scrape an entire Instaclustr account.
We collect general node metrics that are useful for monitoring general cluster health information such as CPU usage, disk usage, and memory usage on nodes; “ic_node_disk_utilization”, “ic_node_cpu_utilization” and “ic_node_memavailable”. Application-specific metrics, such as table metrics for Cassandra and Topic metrics for Kafka are also available. For available metrics, please refer to: https://instaclustr.redoc.ly/Current/tag/Monitoring-API. All metrics are labeled with the node id they are associated with and are tagged by their cluster id, public IP, and private IP for easy identification.
To integrate Prometheus monitoring with Grafana first go to your Grafana data sources tab in Configuration (http://localhost:3000/datasources) and add a new data source targeting the URL of the Prometheus scraper you set up above. Then create a new dashboard and specify Prometheus as the data source and we can start visualizing our metrics. In the query tab select a metric from the “Metrics Browser” dropdown box and the labels and label values you want to filter on.
As you can see above in Grafana we can specify the Cluster ID, Node ID, Private or Public IP etc. to choose what nodes to monitor. Cluster tags created through the Instaclustr console can also be exposed as labels for easy reference. Cluster tags can be created by clicking the top-right gear and selecting the Cluster Resources menu and then selecting the Tags tab. Note that Prometheus doesn’t support labels with special characters so special characters in labels are replaced with “_”. Other metric labels such as metric type and schema-related labels such as the table for Cassandra metrics are also available. Once the desired metric has been selected click “Use query” to visualize.