(Source: Shutterstock)
“All is flux, nothing stays still.”
“You can not step twice into the same river; for other waters are ever flowing on to you.”
Heraclitus, 500 BCE
In previous parts of this blog series, we’ve seen how to introduce schemas into Apache Kafka® to enable smooth serialization and deserialization between producers and consumers, and prevent any unwanted changes in the data format. Plato believed that the forms don’t change, but that change is restricted to the realm of appearances, i.e. the physical world. But Heraclitus (quoted by Plato) believed that “change is inevitable” (and may have invented the concept of streams processing). So what happens when the unchangeable forms (schemas) meet the inevitability of change? Let’s dip our toes in the water and find out.
1. BACKWARD Compatibility Experiments
As we discovered in the last part of this series, the default compatibility level is “BACKWARD”, and you can’t change the schema from a platonic solid to a simple string with this level of compatibility. Let’s try some experiments to determine what can and can’t be changed, using the Karapace REST API and the curl command.
First, let’s check that the compatibility level is still BACKWARD:
curl -X GET -u karapaceuser:karapacepassword karapaceURL:8085/config
{“compatibilityLevel”: “BACKWARD”}
Now let’s create the initial PlatonicSolid schema for a new topic “evolution”:
curl -X POST -H “Content-Type: application/vnd.schemaregistry.v1+json” -u karapaceuser:karapacepassword –data ‘{“schema”: “{\”fields\”: [{\”name\”: \”figure\”, \”type\”: \”string\”}, {\”name\”: \”faces\”, \”type\”: \”int\”}, {\”name\”: \”vertices\”, \”type\”: \”int\”}, {\”name\”: \”volume\”, \”type\”: [\”float\”, \”null\”]}], \”name\”: \”PlatonicSolid\”, \”namespace\”: \”example.avro\”, \”type\”: \”record\”}”}’ karapaceURL:8085/subjects/evolution-value/versions
{“id”: 8}
This apparently worked ok, as any schema is allowed as the first version for a given subject (evolution-value in this case). The result tells us that the schema has been registered successfully and has an ID of 8 (Note: This is not the 1st schema I’ve created in this instance of Karapace).
Let’s double check to see what the latest schema is for evolution-value:
curl -X GET -u karapaceuser:karapacepassword karapaceURL:8085/subjects/evolution-value/versions/latest | jq
{
“id”: 8,
“schema”: “{\”fields\”: [{\”name\”: \”figure\”, \”type\”: \”string\”}, {\”name\”: \”faces\”, \”type\”: \”int\”}, {\”name\”: \”vertices\”, \”type\”: \”int\”}, {\”name\”: \”volume\”, \”type\”: [\”float\”, \”null\”]}], \”name\”: \”PlatonicSolid\”, \”namespace\”: \”example.avro\”, \”type\”: \”record\”}”,
“subject”: “evolution-value”,
“version”: 1
}
This confirms that the schema with ID 8 is the latest version, shows us what the schema is exactly, and also tells us that the version number of the latest schema is 1, i.e. only the initial schema exists for this subject, no schema changes have been made yet.
For simplicity, rather than trying to register different schemas (so we don’t accidentally change the schema), I used a different REST API which checks for compatibility only (but unfortunately doesn’t provide any explanation—just true or false).
First, let’s check that this works and see if the current schema is compatible.
curl -X POST -H “Content-Type: application/vnd.schemaregistry.v1+json” -u karapaceuser:karapacepassword –data ‘{“schema”: “{\”fields\”: [{\”name\”: \”figure\”, \”type\”: \”string\”}, {\”name\”: \”faces\”, \”type\”: \”int\”}, {\”name\”: \”vertices\”, \”type\”: \”int\”}, {\”name\”: \”volume\”, \”type\”: [\”float\”, \”null\”]}], \”name\”: \”PlatonicSolid\”, \”namespace\”: \”example.avro\”, \”type\”: \”record\”}”}’ karapaceURL:8085/compatibility/subjects/evolution-value/versions/latest
{“is_compatible”: true}