Instaclustr is pleased to announce the general availability of extensions to our provisioning APIs and Terraform Provider to allow Apache Kafka® topic configuration and ACL management.
What Is Apache Kafka Topic Management and ACL Management?
Topics are the core piece of application configuration for any Apache Kafka usage. Topic management includes operations such as accessing and modifying topics’ configurations, deleting topics, and creating new ones.
ACLs (Access Controls Lists) are how Kafka manages granting and restricting access to Kafka topics. ACL management refers to the set of operations that can be performed for managing ACLs* in Kafka such as creating and deleting ACLs, and listing all existing ACLs.
What’s New With These Changes?
Currently, when provisioning clusters, customers wanting to manage topics have to switch from using the provisioning APIs (for spinning up the Kafka cluster) to using the kafka-topics.sh or ic-kafka-topics tools (to create topics). The new changes introduce extensions to existing functionality in the provisioning APIs to allow operations like creating topics to be carried out without having to download, install, and use other tools. The same applies for customers using the Instaclustr Terraform provider, who now have access to additional options to, for example, streamline their CI/CD pipelines. In short, customers will now find it easier to provision and maintain usable Kafka clusters via a single familiar interface that can be scripted from end to end.
What Topic and ACL Management-related Changes Are Being Introduced?
The new extensions are:
- For topic management:
- List all topics
- Create a topic
- Delete a topic
- Get a topic’s replication factor and number of partitions
- Get a topic’s configuration
- Modify a topic’s configuration
- For ACL management:
- List all ACLs
- Search ACLs
- Create ACL
- Delete ACL
All functionality available through these APIs is also available via our Terraform provider. To learn more about the Instaclustr Terraform provider and how to use it, please read further here.
Examples on Using New APIs
We’ve done our best to make using these APIs simple and straightforward. For example, to create a new topic you would send a POST request to https://api.instaclustr.com/provisioning/v1/{cluster}/kafka/topics (where {cluster} is the IP address of your Kafka cluster), with the following body:
1 2 3 4 5 6 7 8 9 10 | Body { "topic": "test", "partitions": 3, "replicationFactor": 3 } |
The output from this would be something like the following, indicating that a new topic with the specified name has been created:
1 2 3 4 5 | { "message": "Kafka topic test created." } |
To achieve the same result using the Instaclustr Terraform provider, the following code would be needed:
1 2 3 4 5 6 7 8 9 10 11 | resource "instaclustr_kafka_topic" "test"{ cluster_id = xxx topic = "test" replication_factor = 3 partitions = 3 } |
These APIs are available now with Kafka 2.6.1 and later deployments and are in addition to existing methods of topic and ACL management described in our provisioning API documentation.
Existing customers who would like further information are encouraged to get in touch with their Customer Success representative or to consult our support page. New customers should visit this page to sign up for a free consultation with one of our experts and welcome to sign up for a 30-day free trial to try out our Managed Kafka offering.