Provide access to S3 bucket using permission policy
Recommended method for RIYOA account where both cluster instances and S3 bucket are in the same AWS account. This method will add a permission policy to the instance role. It is useful when you want to provide the Kafka Connect cluster with direct access to the S3 bucket.
Follow Custom Kafka Connect Connectors until step 9 to create a Kafka Connect cluster with custom connectors
If you use the Instaclustr Console, in the Custom Connector Configuration section, choose “Add permission policy to instance role later”
If you use the Provisioning API, specify the S3 bucket name without any further access detail in the body of the POST request. For example
Delete the old policy aws iam delete-role-policy--role-name$CDC_ID--policy-name s3-access-policy
Add the edited policy aws iam put-role-policy--role-name$CDC_ID--policy-name new-s3-access-policy--policy-document file://FILE_PATH
After the policies are set up correctly and the cluster hits RUNNING state, head to the Managing Custom Connectors section in Connectors page and press Sync to load the custom connectors.
Once loaded successfully, they should be visible under Available Connectors section.
Provide access to S3 bucket using role
Recommended method for customers who use a RIIA account and have an S3 bucket on their own AWS account, but can also be used for RIYOA clusters. This method uses a separate role with access to the S3 bucket, let’s call it S3 access role, and allows the instance role to assume the S3 access role and gain access to the bucket. It is useful when you want to manage the S3 access role separately from the instance. You can do this using the AWS CLI or the AWS Console.
Using AWS CLI
Create a policy that allow access to the S3 bucket
Prepare a JSON file that contains the policy. It should be similar to:
Then click Next:Tags and optionally provide tags for the policy
Then review, name the policy and create it
Create the S3 access role with the S3 access policy and copy its ARN
Go to the IAM dashboard, switch to Roles and click on Create Role
Select Custom trust policy as trusted entity
Click Next, then find and tick select the S3 access policy we just created
Then click Next to review and create the role
Provision the Kafka Connect cluster using your preferred method with option “Use IAM role” and copy its Data Center Id If you use the Instaclustr Console, in the Custom Connector Configuration section, choose “Use IAM role”. Input the S3 access role ARN If you use the Provisioning API, specify the S3 bucket name with the S3 access role ARN. For example: