This example shows how to use the Karapace Schema Registry to store data schemas for Kafka topics which we will generate using Apache Avro. The example will also demonstrate how to use the Schema Registry to produce and consume generated Apache Avro objects using an Instaclustr Kafka cluster.
Creating an Apache Kafka cluster with the Karapace Schema Registry add-on
Instaclustr now offers Karapace Schema Registry as an add-on for our Instaclustr for Apache Kafka Managed Service. To take advantage of this offering, you can now select ‘Karapace Schema Registry’ as an option when creating a new Apache Kafka cluster.
Note: Karapace Schema Registry cannot be used together with either Kafka Schema Registry (Old) or Kafka REST Proxy (Old). This is enforced through the console as well our API, and Terraform Provider. For example, on the console, enabling Karapace causes Kafka Schema Registry as well as Kafka REST Proxy to be disabled, and vice-versa.
If you wish to add the Karapace Schema Registry to an existing Apache Kafka cluster, or wish to migrate from Kafka Schema Registry to Karapace Schema Registry on an existing cluster, please contact [email protected].
Using the Karapace Schema Registry
Now that the Karapace Schema Registry is up and running, you can use it in your applications to store data schemas for your Kafka topics. The following example is a Java application that uses the Karapace Schema Registry and Apache Avro to produce and consume some simulated product order events.
Allow access to your client application
Before you can access our Karapace schema registry application, you need to open the firewall to your client application IP address. Once your cluster is up and running, go to Firewall Rules and add your client IP address to the Karapace Schema Registry Allowed Addresses.
Add the kafka_2.12, avro, and kafka-avro-serializer packages to your application. These packages are available via Maven (kafka_2.12, avro, kafka-avro-serializer). To add the following dependencies using Maven, add the following to your pom.xml file:
You will also need the avro-tools utility in order to compile the data schema into a Java class. The avro-tools utility is available here.
Create the Avro Schema
Before you can produce or consume messages using Avro and the Karapace Schema Registry you first need to define the data schema. Create a file orderEventSchema.avsc with the following content:
This file specifies a simple OrderEvent data serialization schema for product orders, with each OrderEvent containing an id, timestamp, product name, and price. For more information on the Avro serialization format see the documentation here.
Generate the Avro Object Class
With the schema file created, use the avro-tools utility to compile the schema file into an actual Java class:
Note: The src/main/java file path at the end of the command can be located wherever you want so long as you make sure the generated class will be accessible by your application code. An example file structure is:
Create Kafka Topic
Use the guide here to create a new topic called orders.
Producing Avro Objects
Before creating a Kafka producer client, you first need to define the configuration properties for the producer client to use. In this example we provide only the required properties for the producer client. See here for the full list of configuration options.
The Connection Info page in the Instaclustr Console has these example settings pre-configured with your cluster’s IP addresses, username and password.
If your cluster has client ⇆ broker encryption enabled, create a new file named producer.properties with the following content, ensuring the password, truststore location, and bootstrap servers list are correct:
As in the producer example, before creating a Kafka consumer client you first need to define the configuration properties for the consumer client to use. In this example we provide only the required properties for the consumer client. See here for the full list of configuration options.
If your cluster has client ⇆ broker encryption enabled, create a new file named consumer.properties with the following content, ensuring the password, truststore location, and bootstrap servers list are correct: