Using the Instaclustr Monitoring API with Datadog

Instaclustr’s Monitoring API is designed to allow you to integrate the monitoring information from your Instaclustr managed cluster with the monitoring tool used for the entire application. Datadog (datadoghq.com) is a popular platform for monitoring a range of applications. This help article walks you through how to use the Instaclustr Monitoring API with Datadog.

At a high-level, the approach we will take in this article is to install a script on a server you manage that has the Datadog agent installed. This script calls the Instaclustr Monitoring API at regular intervals and passes the information returned to the Datadog agent which reports it to the central Datadog system.

One of our awesome customers has also come up with and uses an alternative approach using AWS lambdas. You can find details here: https://github.com/manheim/InstaCluster-to-Datadog-Lambda

Table of Contents

Prepare Your Environment

Follow these steps to set up your environment:

1. Set up a cluster with Instaclustr (see https://www.instaclustr.com/support/documentation/getting-started/creating-a-cluster/)
2. Set up a Datadog account (datadoghq.com)
3. Install the Datadog agent on the machine you will use to run the integration script (Install instruction are available in the Datadog console).
4. Install Python on the machine (https://www.python.org/downloads/)
5. Install the pip Python package manager on the machine (https://pip.pypa.io/en/stable/installing/).
6. Install the Datadog DogStatsD API package (pip install datadog).

Set Up The Script

We have created a sample script that calls the Instaclustr API and forwards the data to Datadog. The script is available on GitHub here: https://github.com/instaclustr/ICAPI-DataDog. You can download the ZIP file or clone the repository.

One of our customers has extended our open source script above with some enhancements. Depending on your requirements, you may use that script instead. The script is available on GitHub here:
https://github.com/lendi-au/ICAPI-DataDog.

The script (ic2datadog.py) is fairly straightforward. It retrieves a specified list of metrics for all nodes in the cluster and requires a configuration file (called configuration.json) in the format shown below:

The settings to be added to the configuration file are:

• cluster_id: The Instaclustr cluster ID. Available from the cluster details page on the Instaclustr console
• metrics_list: A comma separated list of metrics to retrieve and pass to Datadog. For a full list of available metrics, see the Instaclustr monitoring API documentation (https://www.instaclustr.com/support/api-integrations/api-reference/monitoring-api/).
• dd_options: Your Datadog API key and Application key. Available from Integrations/APIs in the Datadog console (you may need to create a new app key).
• ic_options: Your Instaclustr user name and API key. These can be created under the Account/API Keys tab of the Instaclustr console. Make sure you copy them as they will only be displayed once. For more information, please view our support documentation page on API Keys.

Run The Script and View The Results

Running the script is a simple matter of ‘python ic2datadog.py’. The script will then run until interrupted.After a minute or so of running, the metrics will be visible in Datadog.

You can see the results by logging into the Datadog console:

To view the gauge metrics (e.g. CPU):

1. Go to Metrics/Explorer
2. In the Graph text box, start typing ‘instaclustr’. You should see a list of available metrics appear in the format “Instaclustr.[Node IP].[metric name]. Choose the metrics you want and Datadog will draw you a graph.

To view the node status information:

1. Go to Monitors/Check Summary
2. You should see the Instaclustr node status checks in the list (filter for “Instaclustr” if necessary).

The Instaclustr metrics are now available to use wherever else you would use them in Datadog (dashboards, monitors, etc).