A long camel train on the beach in Broome, Western Australia

(Source: Adobe Stock by scottimage)

Recently I wrote a blog called “Running the Apache Camel™ HTTP Kafka Source Connector on Instaclustr Managed Apache Kafka®” (check it out here). There are a lot of Apache Camel Kafka Connectors available, and I wondered how to “get your Camels in a row”? That is, understand what permutations of Apache Camel Kafka connectors are possible and form a useful “camel train”! 

Which technologies are supported by Apache Camel Kafka source connectors only, sink connectors only, or ideally, both source and sink connectors?  

Technologies with both source and sink connectors make it easy to get data into and out of Kafka again, but in practice, the advantage is that it’s easy to use Kafka to write data to the sink/target technology and read it back into Kafka with a source connector.  

Technologies with both types of connectors are more likely to work correctly for a data pipeline, as they take into account the target technology’s native data formats, types, and APIs etc.

Here’s a reminder of how Kafka Connect works with heterogeneous source and sink technologies: 

Here are my initial results (all errors are mine!)—a table showing Kafka source connectors in the left-hand column and sink connectors in the right-hand column. Technologies (53) with both source and sink connectors are highlighted in green:

I’ve left the -source and -sink postfixes in the names to help with understanding the connector roles (but note that some connectors don’t have source/sink in the names—so rather than using a different connector for the source or sink, you configure the connector as either a source or sink connector—officially there are 172 Camel Connectors, but 179 if you count these types as 2 sorts).

The above table is the traditional Kafka-centric view of the source and sink connectors (writing to Kafka from a source technology, reading from Kafka to a sink technology). But as noted above, it’s also useful to put the non-Kafka technologies in the center with Kafka writing to them (Kafka sinks) and reading from them (Kafka sources)—in theory, and in practice, the Kafka sink and source systems can be different (there are many use cases for multiple Kafka clusters):

So, in this table we’ve reversed the columns with the Sinks on the left:

I was also curious to see which of Instaclustr’s Managed Services have matching Apache Camel Kafka connectors (noting that all of them work with Kafka® Connect), and here’s a blue highlighted table with the answers:

This is pretty good coverage (except for OpenSearch®). Interestingly, there are also a couple of Kafka source and sink connectors—this is a bit “odd”, as Kafka connectors are designed to write to, or read from, Kafka and some other technology!

Kafka connector, Kafka source, and sink connectors will presumably work identically and read data from Kafka and write data back to Kafka—actually this could be useful and is what Kafka stream processors do, too.

There could be some use cases for this (e.g., moving data between topics, applying single message transforms to data, etc.). But watch out for infinite event loops (which is why MM2 is normally used for Kafka→Kafka mirroring as it can be configured to prevent them):

(Source: Adobe Stock)

I was trying to think of a nice visualization for this but still haven’t found it—any suggestions are welcome. Also, note that I haven’t tried most of these connectors out yet, so you will need to do your own testing and evaluation.

And just a reminder, when configuring Apache Camel Kafka Connectors, you need to look in 3 places:

1. Camel component documentation
2. Camel Kafka Connect basic configuration documentation, and
3. Camel Kafka Connect specific documentation!