Kafka User Management
Instaclustr provides both a User Management UI and User Management API for Kafka clusters to help you with managing users. The User Management UI and API provide four functions:
- List Users
- Add Users
- Change User Passwords
- Delete Users
All Kafka users created or listed in the Users list will have read and write access to all topics in the cluster. There are details below on how to change this in Kafka ACL Management. If you require any other configuration, please submit a support request.
All Kafka clusters will have an ickafka user automatically created as a convenience, which can be deleted. If the ickafka user is deleted or has the password changed the new password will not be stored and the credentials will disappear from the cluster’s Connection Info page. Because changed passwords and new user passwords will not be stored, ensure you make a note of the passwords to user accounts.
For Kafka Schema Registry and Kafka Rest Proxy services the user password can be updated from the dashboard kafka user management UI. We do not support user deletion or creation for these services. Make sure you take into account that the services will be restarted when actioning a password change.
User Management UI
These examples show how to use the features of the Instaclustr console’s User Management UI.
List Users
- To list available Kafka users, first select your Kafka cluster in the Instaclustr console. With your Kafka cluster selected, click Users from the Kafka menu.
- The Users page will list all Kafka users on the cluster. If you have enabled Kafka Schema Registry and/or Kafka Rest Proxy, you will see the default users for those services listed here as well.
Add User
- To add a new Kafka user, first select your Kafka cluster in the Instaclustr console. With your Kafka cluster selected, click Users from the Kafka menu.
- From the Users page, click the New User button. You will be redirected to the New User page.
- On the New User page, enter the desired username and password for the new Kafka user, the desired level of initial permissions, and your Instaclustr password to confirm your identity. The initial permissions setting can be one of three things:
- Standard – The new user will be able to read and write to all topics
- Read-only – The new user will be able to read from all topics, but not write
- None – The new user will have no initial permissions
- Once you’ve entered the required information, click the Create User button. If your Instaclustr password is correct and your username and password are valid, your new user will be created.
- Once the user is created, you will be redirected to the Users page, with a success message displayed at the top of the page. Your new user will be shown in the list of users.
Change User Password
- To change the password of an existing Kafka user, first select your Kafka cluster in the Instaclustr console. With your Kafka cluster selected, click Users from the Kafka menu.
- From the Users page, click the Change Password button of the user whose password you want to change. You will be redirected to the Change User Password page.
- On the Change User Password page, enter the new password for the User, and your Instaclustr password to confirm your identity. Once you’ve entered the required information, click the Change Password button.
- After clicking the Change Password button you will be redirected to the Users page, with a success message displayed at the top of the page.
Delete User
- To delete an existing Kafka user, first select your Kafka cluster in the Instaclustr console. With your Kafka cluster selected, click Users from the Kafka menu.
- From the Users page, click the Delete User button of the user whose password you want to change. You will be redirected to the Delete User page.
- On the Delete User page, enter your Instaclustr password to confirm your identity. Once you’ve entered your password, click the Delete User button to delete the user.
- If the password was entered correctly and validated, you will be redirected to the Users page, with a success message displayed at the top of the page. The deleted user will no longer be shown in the list of users.
User Management API
These examples show how to use the features of the Instaclustr User Management API.
Please note that we currently do not support Kafka schema registry and Kafka rest proxy with add user and delete user functions.
To provide customers with a high level of security, the Kafka User API endpoint will not store or allow customers to retrieve passwords for Kafka users.
For each endpoint listed below, all requests must include basic authentication details:
1 2 | Username: <your Instaclustr account username> Password: <your Instaclustr provisioning API key> |
List Users
- To retrieve a list of users currently enabled in the Kafka cluster, make a GET request to:
https://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka/users - The API should respond with a 200 status code and a JSON list of users. E.g.:123["ickafka"]
- To list the users of Kafka rest proxy or Kafka schema registry, use the bundle name as kafka_rest_proxy or kafka_schema_registry respectively in the GET request. E.g.:123https://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka_schema_registry/usershttps://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka_rest_proxy/users
Add User
- To add a new user to the Kafka cluster (with read/write access to all topics), make a POST request to:
https://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka/users
A request body should be included, specifying the username and password, and initial permissions to use. E.g.:12345{"username": "test1","password": "Test1Test1","initial-permissions": "standard"}
Valid values for “initial-permissions” are : “standard”, “read-only”, “none”. - The API should respond with a 201 status code and a short message. E.g.123{"message": "User test1 created."}
Once this message is received, the user should be immediately available for use when connecting to the Kafka cluster.
Change User Password
- To change an existing user’s password, make a POST request to:
https://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka/users/reset-password
A request body should be included, specifying the username and password to apply. E.g.:1234{"username": "test1","password": "Test2Test2"} - The API should respond with a 200 status code and a short message. E.g.:123{"message": "Updated password for user test1."}
- To change the user passwords of Kafka rest proxy or Kafka schema registry, use the bundle name as kafka_rest_proxy or kafka_schema_registry respectively in the POST request. E.g.:123https://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka_rest_proxy/users/reset-passwordhttps://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka_schema_registry/users/reset-password
Delete User
- To delete a Kafka user, make a DELETE request to:
https://api.instaclustr.com/provisioning/v1/<cluster_id>/kafka/users
A request body should be included, specifying the username and password to use. E.g.:123{"username": "test1"} - The API should respond with a 200 status code and a short message. E.g.123{"message": "User test1 has been deleted."}
Kafka ACL Management
Warning: It is not recommended that you use the Kafka ACLs to grant higher privileges than the default given by Instaclustr. This could result in your cluster being unrecoverable and not covered under Instaclustr’s SLA’s. If you require any other configuration, please submit a support request before making any changes beyond those outlined below.
Kafka has built in Access Control Lists so that you can control what users and hosts are allowed to do. Instaclustr has enabled modification of these consoles. Currently this accessed through the Kafka CLI.
The default ickafka user is given the ability to modify ACLs by default. If this user is deleted before you have made the required ACL changes please contact the Technical Operations team and they will be able to recover this for you.
Set Up the Kafka CLI
Instructions for setting up the Kafka CLI can be found on the Connection Info page after you have set up a Kafka cluster
List the ACLs
- Once you have set up the kafka.properties file as described above you will be able to list the ACLs.
- In the bin directory of your kafka cli run:1./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --list
- This should return a list of ACLs for all users12345678910111213141516Current ACLs for resource 'ResourcePattern(resourceType=TRANSACTIONAL_ID, name=*,patternType=LITERAL)':(principal=User:testuser, host=*, operation=ALL, permissionType=ALLOW)Current ACLs for resource `ResourcePattern(resourceType=CLUSTER, name=kafka-cluster, patternType=LITERAL)`:(principal=User:testuser, host=*, operation=CLUSTER_ACTION, permissionType=DENY)(principal=User:testuser, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)(principal=User:testuser, host=*, operation=IDEMPOTENT_WRITE, permissionType=ALLOW)(principal=User:testuser, host=*, operation=CREATE, permissionType=ALLOW)(principal=User:testuser, host=*, operation=ALTER_CONFIGS, permissionType=DENY)(principal=User:testuser, host=*, operation=DESCRIBE, permissionType=ALLOW)(principal=User:testuser, host=*, operation=ALTER, permissionType=DENY)Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=*, patternType=LITERAL)`:(principal=User:testuser, host=*, operation=ALL, permissionType=ALLOW)Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:(principal=User:testuser, host=*, operation=ALL, permissionType=ALLOW)
- Results can be further filtered by user or host, for example:
1./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --list --principal User:ickafka - The ACLs first list the ResourceGroup, then underneath each line lists User:Operation
Alter the ACLs for Users
Instaclustr by default gives all users created through the dashboard or the API the right to produce or consume from any topic.
Kafka ACLs work on the principal that Deny takes precedence over Allow. So if a user has both Deny and Allow to the one operation on the one resource group, that user will be denied.
Therefore, when considering altering the default Instaclustr Kafka ACLs, do not think about allowing users access to resources, think in the mindset of denying them, as by default they are allowed.
Note: Kafka also assumes that unless there is a specific Allow ACL then the default is Deny.
Altering Kafka User ACLs
Examples of adding ACLs for Kafka users below:
1 | ./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --add --deny-principal User:test --operation Write --topic test --force |
1 | ./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --add --deny-principal User:test --operation All --topic test --force |
1 | ./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --add --allow-principal User:test --operation Read --topic test --force |
Examples of removing ACLs for Kafka users below:
1 | ./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --remove --deny-principal User:test --operation Write --topic test --force |
1 | ./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --remove --deny-principal User:test --operation All --topic test --force |
1 | ./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --remove --allow-principal User:test --operation Read --topic test --force |
Host ACLs
Kafka ACLs also allow restrictions from hosts with the –allow-host or –deny-host flags. Note: IP addresses only supported. Hostnames are not supported.
Additional ACL Options
The kafka-acls.sh tool provides some convenience flags for the most common requirements. These are as follows.
–producer | Convenience option to add/remove ACLs for producer role. This will generate ACLs that allows WRITE, DESCRIBE and CREATE on topic. |
–consumer | Convenience option to add/remove ACLs for consumer role. This will generate ACLs that allows READ, DESCRIBE on topic and READ on consumer-group. |
Different Resource Groups
The examples given so far have been limited to users and topics. However, Kafka can have ACLs added at the cluster or group level also. You can also permit more broadly by specifying * on a resource group.
You can also add ACLs on prefixed resource patterns, the default resource pattern is LITERAL, which performs exact name matching. However you could do something similar to add an ACL to all topics starting with Instaclustr
1 | ./kafka-acls.sh --bootstrap-server public_ip_of_kafka_node:9092 --command-config kafka.properties --add --deny-principal User:write --operation Read --topic instaclustr --resource-pattern-type prefixed --force |
Submit Comment