Using the Karapace Schema Registry

This example shows how to use the Karapace Schema Registry to store data schemas for Kafka topics which we will generate using Apache Avro. The example will also demonstrate how to use the Schema Registry to produce and consume generated Apache Avro objects using an Instaclustr Kafka cluster.

Creating an Apache Kafka cluster with the Karapace Schema Registry add-on

Instaclustr now offers Karapace Schema Registry as an add-on for our Instaclustr for Apache Kafka Managed Service. To take advantage of this offering, you can now select ‘Karapace Schema Registry’ as an option when creating a new Apache Kafka cluster.

Note: Karapace Schema Registry cannot be used together with either Kafka Schema Registry (Old) or Kafka REST Proxy (Old). This is enforced through the console as well our API, and Terraform Provider. For example, on the console, enabling Karapace causes Kafka Schema Registry as well as Kafka REST Proxy to be disabled, and vice-versa.

If you wish to add the Karapace Schema Registry to an existing Apache Kafka cluster, or wish to migrate from Kafka Schema Registry to Karapace Schema Registry on an existing cluster, please contact [email protected].

Note: You can enable the Karapace Schema Registry add-on on a Kafka cluster with PCI enabled. However, for such clusters, you must design your applications to comply with PCI DSS standards and local regulations. i.e. Cardholder data (CHD), Sensitive Authentication Data (SAD), Personally Identifiable Information (PII) or other sensitive data must not be included in the message schema for your Kafka topics. Keeping sensitive data out of schema helps prevent accidental data leakage or data loss. 

For definitions of what is considered Cardholder Data (CHD) or Sensitive Authentication Data (SAD), or Personally Identifiable Information, please refer to the PCI DSS standard (for CHD and SAD) and local regulations governing PII.

Using the Karapace Schema Registry

Now that the Karapace Schema Registry is up and running, you can use it in your applications to store data schemas for your Kafka topics. The following example is a Java application that uses the Karapace Schema Registry and Apache Avro to produce and consume some simulated product order events.

Allow access to your client application

Before you can access our Karapace schema registry application, you need to open the firewall to your client application IP address. Once your cluster is up and running, go to Firewall Rules and add your client IP address to the Karapace Schema Registry Allowed Addresses.

Client Dependencies

Add the kafka_2.12, avro, and kafka-avro-serializer packages to your application. These packages are available via Maven (kafka_2.12, avro, kafka-avro-serializer). To add the following dependencies using Maven, add the following to your pom.xml file:

You will also need the avro-tools utility in order to compile the data schema into a Java class. The avro-tools utility is available here.

Create the Avro Schema

Before you can produce or consume messages using Avro and the Karapace Schema Registry you first need to define the data schema. Create a file orderEventSchema.avsc with the following content:

This file specifies a simple OrderEvent data serialization schema for product orders, with each OrderEvent containing an id, timestamp, product name, and price. For more information on the Avro serialization format see the documentation here.

Generate the Avro Object Class

With the schema file created, use the avro-tools utility to compile the schema file into an actual Java class:

Note: The src/main/java file path at the end of the command can be located wherever you want so long as you make sure the generated class will be accessible by your application code. An example file structure is:

Create Kafka Topic

Use the guide here to create a new topic called orders.

Producing Avro Objects

Client configuration

Before creating a Kafka producer client, you first need to define the configuration properties for the producer client to use. In this example we provide only the required properties for the producer client. See here for the full list of configuration options.

The Connection Info page in the Instaclustr Console has these example settings pre-configured with your cluster’s IP addresses, username and password.

If your cluster has client ⇆ broker encryption enabled, create a new file named producer.properties with the following content, ensuring the password, truststore location, and bootstrap servers list are correct:

If your cluster does not have client ⇆ broker encryption enabled, create a new file named producer.properties with the following content, ensuring the password and bootstrap servers list are correct:

Make sure the password and bootstrap servers list are correct.

Important Notes:

• To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.
• Instaclustr’s implementation of Karapace Schema Registry is configured with basic authentication credentials in the format ‘user:password@karapace-schema-registry-url:8085’
• basic.auth.credentials.source=URL is necessary for this basic authentication to work correctly.

Java Code

Now that the configuration properties have been set up you can create a Kafka producer.

First, load the properties:

Once you’ve loaded the properties you can create the producer itself:

Next, create some OrderEvents to produce:

Where the getTimestamp() function is:

Now turn each OrderEvent into a ProducerRecord to be produced to the orders topic, and send them:

Finally, use the producer’s flush() method to ensure all messages get sent to Kafka:

Full code example:

Consuming Avro Objects

Client configuration

As in the producer example, before creating a Kafka consumer client you first need to define the configuration properties for the consumer client to use. In this example we provide only the required properties for the consumer client. See here for the full list of configuration options.

If your cluster has client ⇆ broker encryption enabled, create a new file named consumer.properties with the following content, ensuring the password, truststore location, and bootstrap servers list are correct:

If your cluster does not have client ⇆ broker encryption enabled, create a new file named consumer.properties with the following content, ensuring the password and bootstrap servers list are correct:

Make sure the password and bootstrap servers list are correct.

Important Notes:

• To connect to your Kafka cluster over the private network, use port 9093 instead of 9092.
• Instaclustr’s implementation of Karapace Schema Registry is configured with basic authentication credentials in the format ‘user:password@karapace-schema-registry-url:8085’
• basic.auth.credentials.source=URL is necessary for this basic authentication to work correctly.

Java Code

Now that the configuration properties have been setup you can create a Kafka consumer.

First, load the properties:

Once you’ve loaded the properties you can create the consumer itself:

Before you can consume messages, you need to subscribe the consumer to the topic(s) you wish to receive messages from, in this case the orders topic:

Finally, continually poll Kafka for new messages, and print each OrderEvent received:

Full code example:

Putting Them Together

Now that you have a consumer and producer set up, it’s time to combine them.

Start the Consumer

Start the consumer before starting the producer, because by default, consumers only consume messages that were produced after the consumer started.

Start the Producer

Now that the consumer is set up and ready to consume messages, you can start your producer.

If the consumer and producer are set up correctly the consumer should output the messages sent by the producer shortly after they were produced, for example:

Transparent, fair, and flexible pricing for your data infrastructure: See Instaclustr Pricing Here