Managing Apache Kafka ACLs
Warning: It is not recommended that you use the Kafka ACLs to grant higher privileges than the default given by Instaclustr. This could result in your cluster being unrecoverable and not covered under Instaclustr’s SLA’s. If you require any other configuration, please submit a support request before making any changes beyond those outlined below.
Kafka ACL Management via Kafka CLI
Kafka has built in Access Control Lists so that you can control what users and hosts are allowed to do. Instaclustr has enabled the modification of these consoles. Currently, this is accessed through the Kafka CLI.
The default ickafka user is given the ability to modify ACLs by default. If this user is deleted before you have made the required ACL changes please contact the Technical Operations team and they will be able to recover this for you.
Set Up the Kafka CLI
Instructions for setting up the Kafka CLI can be found on the Connection Info page after you have set up a Kafka cluster
List the ACLs
- Once you have set up the Kafka.properties file as described above you will be able to list the ACLs.
- In the bin directory of your kafka cli run:
123{"message": "User test1 has been deleted."}
- This should return a list of ACLs for all users
12345678910111213141516Current ACLs for resource 'ResourcePattern(resourceType=TRANSACTIONAL_ID, name=*,patternType=LITERAL)':(principal=User:testuser, host=*, operation=ALL, permissionType=ALLOW)Current ACLs for resource `ResourcePattern(resourceType=CLUSTER, name=kafka-cluster, patternType=LITERAL)`:(principal=User:testuser, host=*, operation=CLUSTER_ACTION, permissionType=DENY)(principal=User:testuser, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)(principal=User:testuser, host=*, operation=IDEMPOTENT_WRITE, permissionType=ALLOW)(principal=User:testuser, host=*, operation=CREATE, permissionType=ALLOW)(principal=User:testuser, host=*, operation=ALTER_CONFIGS, permissionType=DENY)(principal=User:testuser, host=*, operation=DESCRIBE, permissionType=ALLOW)(principal=User:testuser, host=*, operation=ALTER, permissionType=DENY)Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=*, patternType=LITERAL)`:(principal=User:testuser, host=*, operation=ALL, permissionType=ALLOW)Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:(principal=User:testuser, host=*, operation=ALL, permissionType=ALLOW)
- Results can be further filtered by user or host, for example:
1./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --list --principal User:ickafka
- The ACLs first list the ResourceGroup, then underneath each line lists User:Operation
Alter the ACLs for Users
Instaclustr by default gives all users created through the dashboard or the API the right to produce or consume from any topic.
Kafka ACLs work on the principle that Deny takes precedence over Allow. So if a user has both Deny and Allow to the one operation on the one resource group, that user will be denied.
Therefore, when considering altering the default Instaclustr Kafka ACLs, do not think about allowing users access to resources, think in the mindset of denying them, as by default they are allowed.
Note: Kafka also assumes that unless there is a specific Allow ACL then the default is Deny.
Altering Kafka User ACLs
Examples of adding ACLs for Kafka users are below:
1 |
./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --add --deny-principal User:test --operation Write --topic test --force |
1 |
./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --add --deny-principal User:test --operation All --topic test --force |
1 |
./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --add --allow-principal User:test --operation Read --topic test --force |
Examples of removing ACLs for Kafka users below:
1 |
./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --remove --deny-principal User:test --operation Write --topic test --force |
1 |
./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --remove --deny-principal User:test --operation All --topic test --force |
1 |
./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --remove --allow-principal User:test --operation Read --topic test --forc |
Host ACLs
Kafka ACLs also allow restrictions from hosts with the --allow-host or --deny-host flags. Note: IP addresses are only supported. Hostnames are not supported.
Additional ACL Options
The Kafka-acls.sh tool provides some convenience flags for the most common requirements. These are as follows.
--producer | Convenience option to add/remove ACLs for producer role. This will generate ACLs that allows WRITE, DESCRIBE and CREATE on topic. |
--consumer | Convenience option to add/remove ACLs for consumer role. This will generate ACLs that allows READ, DESCRIBE on topic and READ on consumer-group. |
Different Resource Groups
The examples given so far have been limited to users and topics. However, Kafka can have ACLs added at the cluster or group level also. You can also permit more broadly by specifying * on a resource group.
You can also add ACLs on prefixed resource patterns, the default resource pattern is LITERAL, which performs exact name matching. However, you could do something similar to add an ACL to all topics starting with Instaclustr
1 |
./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --add --deny-principal User:write --operation Read --topic instaclustr --resource-pattern-type prefixed --force |
Kafka ACL Management via Instaclustr API
Instaclustr provides an ACL Management API for Kafka clusters to help you with managing users’ ACL. The ACL Management API supports the following functions:
- List ACLs
- Search (Filter) ACLs
- Create an ACL
- Delete ACLs
These examples show how to use the features of the Instaclustr ACL Management API.
For each endpoint listed below, all requests must include basic authentication details:
1 2 |
Username: <your Instaclustr account username> Password: <your Instaclustr provisioning API key> |
List ACLs
- To retrieve a list of ACLs currently available in the Kafka cluster, make a GET request to:
1https://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka/acls - The API should respond with a 200 status code and a JSON containing the list of ACLs. E.g.:
123456789101112131415{"acls": [{"principal": "User:test","host": "*","resourceType": "TOPIC","resourceName": "*","operation": "ALL","permissionType": "ALLOW","operation": "ALL","permissionType": "ALLOW","patternType": "LITERAL",}]}
Search ACLs
- To retrieve a list of ACLs currently available in the Kafka cluster, make a POST request to:
1https://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka/acls/searches
A request body needs to be included specifying the principal, host, resource type, resource name, operation, permission type, and pattern type. E.g.:
123456789{"principal": "User:test","host": "*","resourceType": "TOPIC","resourceName": "*","operation": "ALL","permissionType": "ALLOW","patternType": "LITERAL",}
Valid values for resource type, operation, permission type, and pattern type are case insensitive:- resourceType: any, cluster, transactional_id, topic, group, delegation_token
- permissionType: any, allow, deny
- patternType: any, match, literal, prefixed. literal and prefixed is the actual pattern type of the ACL, whereas patternType match are some special matching type, refer to https://kafka.apache.org/documentation/ and search for --resource-pattern-type for more information on how match works
- principal: prefixed by User:. Also takes in wildcard * (User:*)
- host: any string. Also takes in wildcard *
- resourceName: any string that fits the resource name, e.g., topic name if the resource type is TOPIC
- operation: any, all, read, write, create, delete, alter, describe, cluster_action, describe_configs, alter_configs, idempotent_write
- All of the keys are optional, if not supplied it will match any values for that key
- On a successful request, the API should respond with a 200 status code and a JSON containing the list of ACLs that fits the search criteria supplied in the request body. E.g.:
12345678910111213{"acls": [{"principal": "User:test","host": "*","resourceType": "TOPIC","resourceName": "*","operation": "ALL","permissionType": "ALLOW","patternType": "LITERAL",}]}
Create an ACL
- To create a new ACL to the Kafka cluster make a POST request to:
1https://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka/acls
A request body needs to be included, specifying the principal, host, resource type, resource name, operation, permission type, and pattern type. E.g.:
1234567891011{"principal": "User:test","host": "*","resourceType": "TOPIC","resourceName": "*","operation": "ALL","permissionType": "ALLOW","operation": "ALL","permissionType": "ALLOW","patternType": "LITERAL",}
Valid values for resource type, operation, permission type, and pattern type are case insensitive :- resourceType: cluster, transactional_id, topic, group, delegation_token
- permissionType: allow, deny
- patternType: literal, prefixed.
- principal: prefixed by User:. Also takes in wildcard * (User:*)
- host: any string. Also takes in wildcard *
- resourceName: any string that fits the resource name, e.g., topic name if the resource type is TOPIC
- operation: all, read, write, create, delete, alter, describe, cluster_action, describe_configs, alter_configs, idempotent_write
- All of the keys are required
- On a successful request, the API should respond with a 200 status code and a short message. E.g.
123{"message": "Kafka ACL (principal=User:test, host=*, resourceType=TOPIC, resourceName=*, operation=ALL, permissionType=ALLOW, patternType=LITERAL) is created."}
Once this message is received, the ACL should be immediately effective in the Kafka cluster.
Delete ACLs
- To delete existing ACLs from the Kafka cluster make a DELETE request to:
1https://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka/acls
A request body needs to be included specifying the principal, host, resource type, resource name, operation, permission type, and pattern type. E.g.:
123456789{"principal": "User:test","host": "*","resourceType": "TOPIC","resourceName": "*","operation": "ALL","permissionType": "ALLOW","patternType": "LITERAL",}
Valid values for resource type, operation, permission type, and pattern type are case insensitive:- resourceType: any, cluster, transactional_id, topic, group, delegation_token
- permissionType: any, allow, deny
- patternType: any, match, literal, prefixed. literal and prefixed is the actual pattern type of the ACL, whereas patternType match are some special matching type, refer to https://kafka.apache.org/documentation/ and search for --resource-pattern-type for more information on how match works
- principal: prefixed by “User:”. Also takes in wildcard * (User:*)
- host: any string. Also takes in wildcard *
- resourceName: any string that fits the resource name, e.g., topic name if the resource type is TOPIC
- operation: any, all, read, write, create, delete, alter, describe, cluster_action, describe_configs, alter_configs, idempotent_write
- All of the keys are required
- On successful request, the ACL that fits the criteria / filter will be deleted, and the API should respond with a 200 status code. Additionally, a short message is given. E.g.
123{"message": "Kafka ACL (principal=User:test, host=*, resourceType=TOPIC, resourceName=*, operation=ALL, permissionType=ALLOW, patternType=LITERAL) is deleted."}
Kafka ACL Management via Instaclustr Terraform Provider
Instaclustr also provides a way to manage Kafka ACL using Terraform. For more information and examples, have a look at
1 |
https://github.com/instaclustr/terraform-provider-instaclustr |
The documentation for the ACL resource is located in docs/resources/kafka_acl.md and the ACL data source is located in docs/data-sources/kafka_acl.md .