In March this year, we shared an exciting new project on our Github page—Instacollector for Redis. The inspiration for this project was our (also public) Instacollector project. The idea behind both projects is that they are a collection of scripts that anyone can download and use on their Redis cluster, or Cassandra cluster (for Instacollector) to access diagnostic information. We hope this will prove useful in diagnosing any issues with your own Redis clusters. One of our Senior Redis TechOps Engineers worked on, tested, and released this tool to the open source community.
The tool can be located here: https://github.com/instaclustr/instacollector_redis and has a comprehensive README. It has been released as open source, so feel free to contribute back as well.
In this blog we take you through running the tool in more detail and what to expect.
Currently this tool collects:
- Redis cluster node hostname
- Node current date
- Node host details
- Node logger user details
- Node time details
- Node uptime and load average
- Node Linux Standard Base (LSB) and Distribution
- Node OS (Operating System) release details
- Node system kernel information
- Node CPU details
- Node CPU core usage
- Node process list in descending order of CPU consumption
- Node memory summary
- Node memory details
- Node process list in descending order of MEMORY consumption
- Node swap file info
- Node transparent hugepage status
- Node disk space info
- Redis directory listing
- Redis-server current open files count
- Redis-server process limits
- Node network interfaces
- Node socket stat summary
- Node open ports
- Node system log size
- Node system log (last 200 entries)
- Node kernel parameters
- Node block device details
- Node total number of open files
- Node hardware information from DMI
- Node hardware list
- Redis conf
- Redis log
List of metrics collected from Redis:
- Redis cluster check
- Redis cluster nodes list
- Redis cluster info
- Redis-server info
- Redis-server client list
- Redis-server latency
- Redis-server slowlog
- Redis cluster shards
- Redis cluster slots
Now to use the tool yourself. First things first, you’ll need to clone the GitHub repository to your local machine. Press the green “Code” button on GitHub and download using your preferred method to the folder you wish to run this from on your local machine.
Before running anything, we need to check the following:
- The localhost should have ssh-private key access to all the nodes in the Redis cluster
- The script will ask for the ssh user you wish to connect with and the location of the private key that the instance will accept. This is what it will use to connect to the Redis nodes.
- Sudo access on the Redis nodes is not required. However, if you use a user without sudo access, the amount of data able to be retrieved will be reduced.
- The ssh user has access to the /tmp directory on the Redis nodes
- The /tmp directory is writable and has enough disk (150KB per node)
- redis-cli (6+) is installed on the localhost.
The first thing needed is a list of IP (Internet Protocol) addresses in redis_cluster_nodes.list. Each IP address needs to be on a new line, for example:
Then you can immediately run it if you like. As you can see in the screenshot below, we print out some instructions.
The script then asks you to select whether you want node level metrics, cluster level metrics, or both. If your configuration/security settings allow, we recommend you go for both. This will give you the greatest insight into your cluster.
The script then asks how to connect, and what level access it should try when it connects to the nodes. Details required include username, path to the private key, and whether it should try sudo commands with this user. Then it needs to know a few things about how these nodes are set up and where it will find configuration and log files.
After this the script is away!
Once the script is complete it will ssh back to you a tarball-ed file with all the relevant log files and diagnostic files.
Here are some examples of the files generated:
The redis.log file
Information about the machines that Redis is running on:
Some Redis command line checks that are run against Redis and the output included.
You can also check the source code directly if you require further details of how these details are collected on the Redis nodes. We think this tool will prove helpful when diagnosing any customer issues for support only customers.
As always, feel free to contribute via GitHub. For existing Instaclustr customers, feel free to reach out to our Technical Operations team here: https://support.instaclustr.com/hc/en-us/requests/new