Feature Releases Technical Technical — Elasticsearch Technical — Kafka Tuesday 28th July 2020

Availability of a Bundled Elasticsearch  Connector

By Paul Aubrey

Instaclustr is pleased to announce we have now added an Elasticsearch Sink Connector, open sourced by lenses.io with an Apache 2.0 license, as a bundled and fully supported connector on Kafka Connect clusters created on the Instaclustr Managed Platform. It enables rapid, no-code integration of Apache Kafka and Elasticsearch to seamlessly move data into Elasticsearch.

The Elasticsearch Sink Connector allows you to stream Kafka topics into your Elasticsearch cluster using Kafka Connect. This integration is an important part of any use case where you wish to leverage the rich features of Elasticsearch or Kibana for analysis and visualization of data flowing from a wide variety of Kafka sources. 

The connector uses a specific query language called KCQL, which can be specified in the connector config for inserting data into Elasticsearch. Operations that can be performed include:

  1. Field selection
  2. Selection of target index
  3. Auto creation of indexes
  4. Adding index name suffixes with a data pattern
  5. Upsert and insert records.

The Elasticsearch Sink Connector now provides support for Elasticsearch 7.7 on top of the existing support for version 6.0. Our development team contributed to the Lenses project to make this enhancement. This broadens the compatibility of the Elasticsearch Connector and importantly provides support for users of the Elasticsearch Open Distro 1.8 that our Elasticsearch managed service is based on.

The Elasticsearch Connector adds to the existing libraries of bundled and fully supported Kafka Connect Connectors. More details of our bundled connectors can be found on the support pages for our Cassandra Connector and AWS S3 Connectors.

If you want to know more about the Instaclustr Managed Kafka Connect service or to understand how Kafka Connect fits into your integration strategy, or how to deploy your customized connector on our platform, please contact our Support Team.