• Dev Rel
  • Apache Kafka
Get Your Apache Camel™ Kafka Connectors in a Row

A long camel train on the beach in Broome, Western Australia

(Source: Adobe Stock by scottimage)

Recently I wrote a blog called “Running the Apache Camel HTTP Kafka Source Connector on Instaclustr Managed Apache Kafka®” (check it out here). There are a lot of Apache Camel Kafka Connectors available, and I wondered how to “get your Camels in a row”? That is, understand what permutations of Apache Camel Kafka connectors are possible and form a useful “camel train! 

Which technologies are supported by Apache Camel Kafka source connectors only, sink connectors only, or ideally, both source and sink connectors?  

Technologies with both source and sink connectors make it easy to get data into and out of Kafka again, but in practice, the advantage is that it’s easy to use Kafka to write data to the sink/target technology and read it back into Kafka with a source connector.  

Technologies with both types of connectors are more likely to work correctly for a data pipeline, as they take into account the target technology’s native data formats, types, and APIs etc.

Here’s a reminder of how Kafka Connect works with heterogeneous source and sink technologies: 

Here are my initial results (all errors are mine!)—a table showing Kafka source connectors in the left-hand column and sink connectors in the right-hand column. Technologies (53) with both source and sink connectors are highlighted in green:

SourceSink
aws-cloudtrail-source
aws-cloudwatch-sink
aws-ddb-experimental-sink
aws-ddb-sink
aws-ddb-streams-source
aws-ec2-sink
aws-eventbridge-sink
aws-kinesis-firehose-sink
aws-kinesis-sourceaws-kinesis-sink
aws-lambda-sink
aws-redshift-sourceaws-redshift-sink
aws-s3-cdc-source
aws-s3-experimental-source
aws-s3-sourceaws-s3-sink
aws-s3-streaming-upload-sink
aws-secrets-manager-sink
aws-ses-sink
Thankaws-sns-fifo-sink
aws-sns-sink
aws-sqs-batch-sink
aws-sqs-fifo-sink
aws-sqs-sourceaws-sqs-sink
aws2-iam
aws2-kms
azure-cosmosdb-source
azure-eventhubs-sourceazure-eventhubs-sink
azure-functions-sink
azure-servicebus-sourceazure-servicebus-sink
azure-storage-blob-cdc-source
azure-storage-blob-changefeed-source
azure-storage-blob-sourceazure-storage-blob-sink
azure-storage-queue-sourceazure-storage-queue-sink
beer-source
bitcoin-source
cassandra-sourcecassandra-sink
ceph-sourceceph-sink
chuck-norris-source
couchbase-sink
cron-source
cxfcxf
cxfrscxfrs
dropbox-sourcedropbox-sink
earthquake-source
elasticsearch-search-sourceelasticsearch-index-sink
exec-sink
fhir-sourcefhir-sink
filefile
file-watch-source
ftp-sourceftp-sink
ftps-sourceftps-sink
github-commit-source
github-event-source
github-pullrequest-comment-source
github-pullrequest-source
github-tag-source
google-bigquery-sink
google-calendar-source
google-functions-sink
google-mail-source
google-pubsub-sourcegoogle-pubsub-sink
google-sheets-source
google-storage-cdc-source
google-storage-sourcegoogle-storage-sink
hdfshdfs
http-secured-sourcehttp-secured-sink
http-sourcehttp-sink
https
infinispan-sourceinfinispan-sink
jdbc
jira-add-comment-sink
jira-add-issue-sink
jira-oauth-source
jira-source
jira-transition-issue-sink
jira-update-issue-sink
jms-amqp-10-sourcejms-amqp-10-sink
jms-apache-activemq-sourcejms-apache-activemq-sink
jms-apache-artemis-sourcejms-apache-artemis-sink
jms-ibm-mq-sourcejms-ibm-mq-sink
kafka-not-secured-sourcekafka-not-secured-sink
kafka-sourcekafka-sink
kafka-ssl-sourcekafka-ssl-sink
kubernetes-namespaces-source
kubernetes-nodes-source
kubernetes-pods-source
log-sink
mail-imap-source
mail-sink
mariadb-sourcemariadb-sink
minio-sourceminio-sink
mongodb-changes-stream-source
mongodb-sourcemongodb-sink
mqtt-sourcemqtt-sink
mqtt5-sourcemqtt5-sink
mysql-sourcemysql-sink
nats-sourcenats-sink
netty-httpnetty-http
nettynetty
oracle-database-sourceoracle-database-sink
postgresql-sourcepostgresql-sink
pulsar-sourcepulsar-sink
rabbitmq-source
redis-sourceredis-sink
rest-openapi-sink
salesforce-create-sink
salesforce-delete-sink
salesforce-sourcesalesforce-update-sink
scp-sink
sftp-sourcesftp-sink
sjms2sjms2
slack-sourceslack-sink
solr-sourcesolr-sink
splunk-hec-sink
splunk-sourcesplunk-sink
sqlserver-sourcesqlserver-sink
ssh-sourcessh-sink
syslogsyslog
telegram-sourcetelegram-sink
timer-source
twitter-directmessage-source
twitter-search-source
twitter-timeline-source
webhook-source
websocket-source
wttrin-source

I’ve left the -source and -sink postfixes in the names to help with understanding the connector roles (but note that some connectors don’t have source/sink in the names—so rather than using a different connector for the source or sink, you configure the connector as either a source or sink connector—officially there are 172 Camel Connectors, but 179 if you count these types as 2 sorts).

The above table is the traditional Kafka-centric view of the source and sink connectors (writing to Kafka from a source technology, reading from Kafka to a sink technology). But as noted above, it’s also useful to put the non-Kafka technologies in the center with Kafka writing to them (Kafka sinks) and reading from them (Kafka sources)—in theory, and in practice, the Kafka sink and source systems can be different (there are many use cases for multiple Kafka clusters):

So, in this table we’ve reversed the columns with the Sinks on the left:

Write to this technology from Kafka (Sink)Read from this technology to Kafka (Source)
aws-cloudtrail-source
aws-cloudwatch-sink
aws-ddb-experimental-sink
aws-ddb-sink
aws-ddb-streams-source
aws-ec2-sink
aws-eventbridge-sink
aws-kinesis-firehose-sink
aws-kinesis-sinkaws-kinesis-source
aws-lambda-sink
aws-redshift-sinkaws-redshift-source
aws-s3-cdc-source
aws-s3-experimental-source
aws-s3-sinkaws-s3-source
aws-s3-streaming-upload-sink
aws-secrets-manager-sink
aws-ses-sink
aws-sns-fifo-sink
aws-sns-sink
aws-sqs-batch-sink
aws-sqs-fifo-sink
aws-sqs-sinkaws-sqs-source
aws2-iam
aws2-kms
azure-cosmosdb-source
azure-eventhubs-sinkazure-eventhubs-source
azure-functions-sink
azure-servicebus-sinkazure-servicebus-source
azure-storage-blob-cdc-source
azure-storage-blob-changefeed-source
azure-storage-blob-sinkazure-storage-blob-source
azure-storage-queue-sinkazure-storage-queue-source
beer-source
bitcoin-source
cassandra-sinkcassandra-source
ceph-sinkceph-source
chuck-norris-source
couchbase-sink
cron-source
cxfcxf
cxfrscxfrs
dropbox-sinkdropbox-source
earthquake-source
elasticsearch-index-sinkelasticsearch-search-source
exec-sink
fhir-sinkfhir-source
filefile
file-watch-source
ftp-sinkftp-source
ftps-sinkftps-source
github-commit-source
github-event-source
github-pullrequest-comment-source
github-pullrequest-source
github-tag-source
google-bigquery-sink
google-calendar-source
google-functions-sink
google-mail-source
google-pubsub-sinkgoogle-pubsub-source
google-sheets-source
google-storage-cdc-source
google-storage-sinkgoogle-storage-source
hdfshdfs
http-secured-sinkhttp-secured-source
http-sinkhttp-source
https
infinispan-sinkinfinispan-source
jdbc
jira-add-comment-sink
jira-add-issue-sink
jira-oauth-source
jira-source
jira-transition-issue-sink
jira-update-issue-sink
jms-amqp-10-sinkjms-amqp-10-source
jms-apache-activemq-sinkjms-apache-activemq-source
jms-apache-artemis-sinkjms-apache-artemis-source
jms-ibm-mq-sinkjms-ibm-mq-source
kafka-not-secured-sinkkafka-not-secured-source
kafka-sinkkafka-source
kafka-ssl-sinkkafka-ssl-source
kubernetes-namespaces-source
kubernetes-nodes-source
kubernetes-pods-source
log-sink
mail-imap-source
mail-sink
mariadb-sinkmariadb-source
minio-sinkminio-source
mongodb-changes-stream-source
mongodb-sinkmongodb-source
mqtt-sinkmqtt-source
mqtt5-sinkmqtt5-source
mysql-sinkmysql-source
nats-sinknats-source
netty-httpnetty-http
nettynetty
oracle-database-sinkoracle-database-source
postgresql-sinkpostgresql-source
pulsar-sinkpulsar-source
rabbitmq-source
redis-sinkredis-source
rest-openapi-sink
salesforce-create-sink
salesforce-delete-sink
salesforce-update-sinksalesforce-source
scp-sink
sftp-sinksftp-source
sjms2sjms2
slack-sinkslack-source
solr-sinksolr-source
splunk-hec-sink
splunk-sinksplunk-source
sqlserver-sinksqlserver-source
ssh-sinkssh-source
syslogsyslog
telegram-sinktelegram-source
timer-source
twitter-directmessage-source
twitter-search-source
twitter-timeline-source
webhook-source
websocket-source
wttrin-source

I was also curious to see which of Instaclustr’s Managed Services have matching Apache Camel Kafka connectors (noting that all of them work with Kafka® Connect), and here’s a blue highlighted table with the answers:

SourceSink
aws-cloudtrail-source
aws-cloudwatch-sink
aws-ddb-experimental-sink
aws-ddb-sink
aws-ddb-streams-source
aws-ec2-sink
aws-eventbridge-sink
aws-kinesis-firehose-sink
aws-kinesis-sourceaws-kinesis-sink
aws-lambda-sink
aws-redshift-sourceaws-redshift-sink
aws-s3-cdc-source
aws-s3-experimental-source
aws-s3-sourceaws-s3-sink
aws-s3-streaming-upload-sink
aws-secrets-manager-sink
aws-ses-sink
aws-sns-fifo-sink
aws-sns-sink
aws-sqs-batch-sink
aws-sqs-fifo-sink
aws-sqs-sourceaws-sqs-sink
aws2-iam
aws2-kms
azure-cosmosdb-source
azure-eventhubs-sourceazure-eventhubs-sink
azure-functions-sink
azure-servicebus-sourceazure-servicebus-sink
azure-storage-blob-cdc-source
azure-storage-blob-changefeed-source
azure-storage-blob-sourceazure-storage-blob-sink
azure-storage-queue-sourceazure-storage-queue-sink
beer-source
bitcoin-source
cassandra-sourcecassandra-sink
ceph-sourceceph-sink
chuck-norris-source
couchbase-sink
cron-source
cxfcxf
cxfrscxfrs
dropbox-sourcedropbox-sink
earthquake-source
elasticsearch-search-sourceelasticsearch-index-sink
exec-sink
fhir-sourcefhir-sink
filefile
file-watch-source
ftp-sourceftp-sink
ftps-sourceftps-sink
github-commit-source
github-event-source
github-pullrequest-comment-source
github-pullrequest-source
github-tag-source
google-bigquery-sink
google-calendar-source
google-functions-sink
google-mail-source
google-pubsub-sourcegoogle-pubsub-sink
google-sheets-source
google-storage-cdc-source
google-storage-sourcegoogle-storage-sink
hdfshdfs
http-secured-sourcehttp-secured-sink
http-sourcehttp-sink
https
infinispan-sourceinfinispan-sink
jdbc
jira-add-comment-sink
jira-add-issue-sink
jira-oauth-source
jira-source
jira-transition-issue-sink
jira-update-issue-sink
jms-amqp-10-sourcejms-amqp-10-sink
jms-apache-activemq-sourcejms-apache-activemq-sink
jms-apache-artemis-sourcejms-apache-artemis-sink
jms-ibm-mq-sourcejms-ibm-mq-sink
kafka-not-secured-sourcekafka-not-secured-sink
kafka-sourcekafka-sink
kafka-ssl-sourcekafka-ssl-sink
kubernetes-namespaces-source
kubernetes-nodes-source
kubernetes-pods-source
log-sink
mail-imap-source
mail-sink
mariadb-sourcemariadb-sink
minio-sourceminio-sink
mongodb-changes-stream-source
mongodb-sourcemongodb-sink
mqtt-sourcemqtt-sink
mqtt5-sourcemqtt5-sink
mysql-sourcemysql-sink
nats-sourcenats-sink
netty-httpnetty-http
nettynetty
oracle-database-sourceoracle-database-sink
postgresql-sourcepostgresql-sink
pulsar-sourcepulsar-sink
rabbitmq-source
redis-sourceredis-sink
rest-openapi-sink
salesforce-create-sink
salesforce-delete-sink
salesforce-sourcesalesforce-update-sink
scp-sink
sftp-sourcesftp-sink
sjms2sjms2
slack-sourceslack-sink
solr-sourcesolr-sink
splunk-hec-sink
splunk-sourcesplunk-sink
sqlserver-sourcesqlserver-sink
ssh-sourcessh-sink
syslogsyslog
telegram-sourcetelegram-sink
timer-source
twitter-directmessage-source
twitter-search-source
twitter-timeline-source
webhook-source
websocket-source
wttrin-source

This is pretty good coverage (except for OpenSearch®). Interestingly, there are also a couple of Kafka source and sink connectors—this is a bit “odd”, as Kafka connectors are designed to write to, or read from, Kafka and some other technology!

Kafka connector, Kafka source, and sink connectors will presumably work identically and read data from Kafka and write data back to Kafka—actually this could be useful and is what Kafka stream processors do, too.

There could be some use cases for this (e.g., moving data between topics, applying single message transforms to data, etc.). But watch out for infinite event loops (which is why MM2 is normally used for Kafka→Kafka mirroring as it can be configured to prevent them):

(Source: Adobe Stock)

I was trying to think of a nice visualization for this but still haven’t found it—any suggestions are welcome. Also, note that I haven’t tried most of these connectors out yet, so you will need to do your own testing and evaluation.

And just a reminder, when configuring Apache Camel Kafka Connectors, you need to look in 3 places:

  1. Camel component documentation
  2. Camel Kafka Connect basic configuration documentation, and
  3. Camel Kafka Connect specific documentation!