Apache Cassandra 4.0 comes with a host of new features to get excited about, along with serious improvements in performance and stability. In a recent post, we took a look at full query logging in Cassandra 4.0 as one of the valuable new features for users. In this post, we’ll review the new diagnostic events feature that comes with Cassandra 4.0.

What Are Diagnostic Events and Why Do They Matter?

Whereas auditing and full query logging examine Cassandra from the point of view of the user and tracking the actions an individual user takes, diagnostic events are more about what is going on under the hood with Cassandra. The ability to track diagnostic events and view them via external monitoring tools is a new function that comes with Cassandra 4.0. Tracking these events can be helpful for monitoring the ongoing health of your cluster and troubleshooting any issues that arise.

In Cassandra, diagnostic events are tracked in memory, and users can be notified of them via Java Management Extensions (JMX), a Java technology used for monitoring applications and systems. To access the information, you need to subscribe to events so they can be pushed via JMX to the respective notification handlers on the client side.

Say, for example, you want to know what is going on with Hints, a data repair technique that can be applied during writes. You would then subscribe to HintEvent, which will pour records of those events into internal memory storage where you can access them via JMX using Cassandra Reaper or other tools. (Note that tools like Reaper are designed for self-supported Cassandra instances; if you’re an Instaclustr Managed Cassandra customer, you are not able to use a tool like Reaper and will probably never need to worry about diagnostic events as our support team will take care of that type of detailed investigation for you.) 

For Cassandra diagnostic events, events you can track in addition to HintEvent include GossiperEvent, BootstrapEvent, and SchemaEvent, among others.

Getting Started With Diagnostic Events

By default, diagnostic events are not enabled when you spin up your Cassandra 4.0 cluster. To enable them, you must set the diagnostic_events_enabled Java flag (boolean variable) to true. 

Then restart the Cassandra node where you enabled the events. 

If you’re using Reaper, you can then go to the “Add Events Subscription” section to subscribe to the diagnostic events of your choice. (To learn more about setting up diagnostic event monitoring with an open source tool like Cassandra Reaper, check out the Reaper documentation.)

The in-memory implementation can currently hold at most 200 events. Currently, this limit is hard-coded. This means that as the time goes by, there will be basically a sliding window of events, dropping the oldest ones.

It is currently not possible to persist diagnostic events anywhere other than in memory. But Instaclustr’s open source team is looking into developing an extension point for integrators to implement their own diagnostic event sinks in the future. A reference implementation of this effort would, ideally, persist these events into a Chronicle Queue file. This is the same way it is done for full query logging (FQL) and audit logging frameworks.

For easier usage, we also plan to expose these events into a virtual table so they would be accessible via Cassandra Query Language (CQL). Instaclustr is leading this effort and you can track the progress and raise your ideas related to this feature in the CEP-12.

You can also persist audit log events into the diagnostic event persistence layer (into memory) by using DiagnosticEventAuditLogger.

Conclusion

Diagnostic events are a useful new feature in Cassandra 4.0 for exploring what’s going on under the hood with Cassandra so you can address performance or stability issues or optimize your data model. Get started with your own Cassandra 4.0 cluster for free on the Instaclustr Managed Platform.