# How to get OpenTelemetry traces from Apache Kafka clients into OpenSearch using Data Prepper and OpenTelemetry Collectors [Blog](/blog/)>[Technology](/blog/category/technical/)>How to get OpenTelemetry traces from Apache Kafka clients into OpenSearch using Data Prepper and OpenTelemetry Collectors How to get OpenTelemetry traces from Apache Kafka clients into OpenSearch using Data Prepper and OpenTelemetry Collectors ========================================================================================================================= November 11, 2025 | By [ Paul Brebner](https://www.instaclustr.com/blog/author/paul-brebner/) [ ](https://x.com/intent/tweet?text=How%20to%20get%20OpenTelemetry%20traces%20from%20Apache%20Kafka%20clients%20into%20OpenSearch%20using%20Data%20Prepper%20and%20OpenTelemetry%20Collectors&url=https://www.instaclustr.com/blog/how-to-get-opentelemetry-traces-from-apache-kafka-clients-into-opensearch-using-data-prepper-and-opentelemetry-collectors/) [ ](https://www.linkedin.com/shareArticle?mini=true&url=https://www.instaclustr.com/blog/how-to-get-opentelemetry-traces-from-apache-kafka-clients-into-opensearch-using-data-prepper-and-opentelemetry-collectors/&title=&summary=How%20to%20get%20OpenTelemetry%20traces%20from%20Apache%20Kafka%20clients%20into%20OpenSearch%20using%20Data%20Prepper%20and%20OpenTelemetry%20Collectors&source=) End-to-end distributed tracing of complex applications is a critical pillar of modern distributed systems observability development and operations. Previously, I’ve used [OpenTelemetry](https://opentelemetry.io/) to trace a complex Kafka application, including Kafka Streams, and looked at the traces in several open source OpenTelemetry GUIs, including [Jaeger, Uptrace, and SigNoz](https://www.instaclustr.com/blog/tracing-apache-kafka-with-opentelemetry/). And earlier, I used its predecessor, OpenTracing, to [trace Anomalia Machina](https://www.instaclustr.com/blog/anomalia-machina-6-application-tracing-opentracing/). [In part 1](https://www.instaclustr.com/blog/how-to-get-opentelemetry-traces-from-apache-kafka-clients-into-opensearch-using-jaeger/) of this blog, I explored option 1 using Jaeger. That approach was relatively easy to do, but it didn’t give the full functionality of the OpenSearch Trace Analytics visualizations. Now it’s time to investigate option 2 using a Data Prepper pipeline and see if we can unlock the full functionality. OpenTelemetry Java Auto instrumentation recap --------------------------------------------- [In part 1](https://www.instaclustr.com/blog/how-to-get-opentelemetry-traces-from-apache-kafka-clients-into-opensearch-using-jaeger/), we ran some Kafka clients with Java OpenTelemetry auto-instrumentation enabled. To recap, on the Kafka client side, this is what you have to run to automatically produce traces from a Java program: java -javaagent:opentelemetry-javaagent.jar -Dotel.traces.exporter=otlp -Dotel.resource.attributes=service.name=test\_producer -Dotel.metrics.exporter=none -jar producer.jar 1 2 3 java -javaagent:opentelemetry-javaagent.jar -Dotel.traces.exporter=otlp -Dotel.resource.attributes=service.name=test\_producer -Dotel.metrics.exporter=none -jar producer.jar The documentation and `opentelemetry-javaagent.jar` are available here. For a more interesting use case and topology, I ran a producer (`service.name=lotsofboxes`) writing to a topic, and two consumers in different groups (`service.name=consumerg1`, `service.name=consumerg2`) reading from the topic. Option 2: A complete and robust pipeline using OpenTelemetry Collectors and OpenSearch Data Prepper --------------------------------------------------------------------------------------------------- In practice, the preferred way of getting OpenTelemetry traces into OpenSearch is [as follows](https://docs.opensearch.org/latest/data-prepper/common-use-cases/trace-analytics/): [![How to get OpenTelemetry traces from Apache Kafka clients into OpenSearch using Data Prepper and OpenTelemetry Collectors diagram 1]()](https://www.instaclustr.com/wp-content/uploads/kafka-opensearch-data-prepper-image-1.jpg) It looks like I need two extra components/steps: OpenTelemetry Collectors on the application side, and OpenSearch Data Prepper between the Collectors and OpenSearch to collect the traces from the Collectors, process them, and insert the data correctly in OpenSearch to enable visualizations. Why is Data Prepper essential for this workflow? From this diagram, it looks like three internal pipelines are needed to process the traces correctly for use by OpenSearch: [![How to get OpenTelemetry traces from Apache Kafka clients into OpenSearch using Data Prepper and OpenTelemetry Collectors diagram 2]()](https://www.instaclustr.com/wp-content/uploads/kafka-opensearch-data-prepper-image-2.jpg) The documentation explains them as follows. There are three processors for the trace analytics feature: - `otel_traces` - The `otel_traces` processor receives a collection of span records from `otel-trace-source`, and performs stateful processing, extraction, and completion of trace-group-related fields. - I assume this is part of the `otel-trace-pipeline`. - `otel_traces_group` - The `otel_traces_group` processor fills in the missing `trace-group-related` fields in the collection of span records by looking up the OpenSearch backend. - I was getting an error about “missing trace groups” in option 1 above, so this is presumably the missing step. - `service_map` - The `service_map` processor performs the required preprocessing for trace data and builds metadata to display the `service-map` dashboards. - I assume this is part of the `service-map-pipeline`. I was puzzled about “trace groups” (as this is not an OpenTelemetry concept), but the documentation is sparse. AI suggested the following explanation: *In OpenSearch, when integrating with OpenTelemetry for distributed tracing, a trace group refers to a logical grouping of related traces or spans that share common characteristics or are associated with a particular service, operation, or application. It provides a way to organize and filter your trace data for easier analysis and visualization.* And groups are apparently customizable: *Trace groups can be defined based on various common characteristics, such as:* - *Service Name: All traces originating from or passing through a specific service can belong to a particular trace group (this appears to be the default).* - *Operation Name: Traces related to a specific operation within a service (e.g., “checkout,” “login”) can form a trace group.* - *Application: Traces from a particular application can be grouped together.* - *Custom Attributes: You can leverage custom attributes added to your OpenTelemetry spans to create highly specific trace groups based on business logic or other relevant metadata.* OpenSearch provides a generic sink that writes data to OpenSearch as the destination. The[ OpenSearch sink](https://docs.opensearch.org/latest/data-prepper/pipelines/configuration/sinks/opensearch/) has configuration options related to the OpenSearch cluster, such as endpoint, SSL, username/password, index name, index template, and index state management. The sink provides specific configurations for the trace analytics feature. These configurations allow the sink to use indexes and index templates specific to trace analytics. The following OpenSearch indexes are specific to trace analytics: - `otel-v1-apm-span` - The `otel-v1-apm-span` index stores the output from the [`otel_traces`](https://docs.opensearch.org/latest/data-prepper/pipelines/configuration/processors/otel-traces/) processor. - `otel-v1-apm-service-map` - The `otel-v1-apm-service-map` index stores the output from the [`service_map`](https://docs.opensearch.org/latest/data-prepper/pipelines/configuration/processors/service-map/) processor. So, a lot is going on in these pipelines, which explains why the service maps were not working using just Jaeger in option 1. Get, configure, and run Data Prepper ------------------------------------ According to the documentation, Data Prepper is: - [A server-side data collector and processor](https://docs.opensearch.org/latest/data-prepper/) - [And is vertically and horizontally scalable and can be clustered for redundancy.](https://docs.opensearch.org/latest/data-prepper/common-use-cases/trace-analytics/#trace-tuning) In practice, this probably means you would be running it on substantial servers and/or in the cloud. However, I just wanted to test the pipeline, so I ran it locally on my Mac. [The getting started guide is here](https://docs.opensearch.org/latest/data-prepper/getting-started/), and I got a local copy with this command: docker pull opensearchproject/data-prepper:latest 1 docker pull opensearchproject/data-prepper:latest Given the number of pipelines needed, and to run it with our managed OpenSearch, you need to write a specific configuration file; here’s an example. Here’s my configuration file (`opensearch_otel_config.yaml`): entry-pipeline: delay: "100" source: otel\_trace\_source: ssl: false buffer: bounded\_blocking: buffer\_size: 10240 batch\_size: 160 sink: - pipeline: name: "raw-trace-pipeline" - pipeline: name: "service-map-pipeline" raw-trace-pipeline: source: pipeline: name: "entry-pipeline" buffer: bounded\_blocking: buffer\_size: 10240 batch\_size: 160 processor: - otel\_traces: sink: - opensearch: hosts: \["https://ip:9200"\] insecure: true username: user password: password index\_type: trace-analytics-raw service-map-pipeline: delay: "100" source: pipeline: name: "entry-pipeline" buffer: bounded\_blocking: buffer\_size: 10240 batch\_size: 160 processor: - service\_map: sink: - opensearch: hosts: \["https://ip:9200"\] insecure: true username: user password: password index\_type: trace-analytics-service-map 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 entry-pipeline: delay: "100" source: otel\_trace\_source: ssl: false buffer: bounded\_blocking: buffer\_size: 10240 batch\_size: 160 sink: - pipeline: name: "raw-trace-pipeline" - pipeline: name: "service-map-pipeline" raw-trace-pipeline: source: pipeline: name: "entry-pipeline" buffer: bounded\_blocking: buffer\_size: 10240 batch\_size: 160 processor: - otel\_traces: sink: - opensearch: hosts: \["https://ip:9200"\] insecure: true username: user password: password index\_type: trace-analytics-raw service-map-pipeline: delay: "100" source: pipeline: name: "entry-pipeline" buffer: bounded\_blocking: buffer\_size: 10240 batch\_size: 160 processor: - service\_map: sink: - opensearch: hosts: \["https://ip:9200"\] insecure: true username: user password: password index\_type: trace-analytics-service-map This pipeline takes data from the OpenTelemetry Collector (`entry-pipeline`) and uses two other pipelines as sinks. These two separate pipelines serve two different purposes and write to different OpenSearch indexes. The first pipeline (`raw-trace-pipeline`) prepares trace data for OpenSearch and enriches and ingests the span documents into a span index within OpenSearch. The second pipeline (`service-map-pipeline`) aggregates traces into a service map and writes service map documents into a service map index within OpenSearch. Note that because there are 2 sinks, you need to provide the OpenSearch cluster host IP address and user/password twice (bolded)! Note that I changed the default sink configurations for insecure from false to true, as I was just testing. In production, you will likely set this back to false (The Data Prepper configuration documentation has more details). You run Data Prepper like this (note the mapping of port 21890): docker run --name data-prepper \\ -p 21890:21890 -v ${PWD}/opensearch\_otel\_config.yaml:/usr/share/data- prepper/pipelines/opensearch\_otel\_config.yaml \\ opensearchproject/data-prepper:latest 1 2 3 4 docker run --name data-prepper \\ -p 21890:21890 -v ${PWD}/opensearch\_otel\_config.yaml:/usr/share/data- prepper/pipelines/opensearch\_otel\_config.yaml \\ opensearchproject/data-prepper:latest Note that on a Mac (with Apple Silicon) you will get the following warning, but it still works ok: WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested 1 2 3 WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested If everything is configured correctly then you will get the following message when starting Data Prepper: org.opensearch.dataprepper.plugins.source.oteltrace.OTelTraceSource - Started otel\_trace\_source on port 21890 1 2 org.opensearch.dataprepper.plugins.source.oteltrace.OTelTraceSource - Started otel\_trace\_source on port 21890 Get, configure and run the OpenTelemetry Collector -------------------------------------------------- The final piece of the puzzle is the OpenTelemety Collector. To understand why the Collector is important, we need to backtrack a bit and have a look at the OpenTelemetry architecture. Here’s the high-level diagram, which clearly shows the central role of the Collector: [![How to get OpenTelemetry traces from Apache Kafka clients into OpenSearch using Data Prepper and OpenTelemetry Collectors diagram 3]()](https://www.instaclustr.com/wp-content/uploads/kafka-opensearch-data-prepper-image-3.png) Source: The Collector functions like a spider in a web and is [documented further here](https://opentelemetry.io/docs/collector/). The Collector offer a vendor-agnostic implementation of how to receive, process and export telemetry data. It removes the need to run, operate, and maintain multiple agents/collectors. This works with improved scalability and supports open source observability data formats (e.g. Jaeger, Prometheus, etc.), sending to one or more backends (also called frontends in the above diagram). Collectors are complex and support multiple components, including Receivers, Processors and Exporters, as shown in this diagram: [![How to get OpenTelemetry traces from Apache Kafka clients into OpenSearch using Data Prepper and OpenTelemetry Collectors diagram 4]()](https://www.instaclustr.com/wp-content/uploads/kafka-opensearch-data-prepper-image-4.png) I installed a Mac version of the Collector and used [these installation instructions](https://opentelemetry.io/docs/collector/installation/). Next, you need to configure it to send data to Data Prepper according to [these instructions](https://docs.opensearch.org/latest/data-prepper/common-use-cases/trace-analytics/#opentelemetry-collector) (you also need to add `otlp: protocols: http`). Here’s my configuration file (`otel-collector-config.yaml`): receivers: jaeger: protocols: grpc: otlp: protocols: grpc: http: zipkin: processors: batch/traces: timeout: 1s send\_batch\_size: 50 exporters: otlp/data-prepper: endpoint: localhost:21890 tls: insecure: true service: pipelines: traces: receivers: \[jaeger, otlp, zipkin\] processors: \[batch/traces\] exporters: \[otlp/data-prepper\] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 receivers: jaeger: protocols: grpc: otlp: protocols: grpc: http: zipkin: processors: batch/traces: timeout: 1s send\_batch\_size: 50 exporters: otlp/data-prepper: endpoint: localhost:21890 tls: insecure: true service: pipelines: traces: receivers: \[jaeger, otlp, zipkin\] processors: \[batch/traces\] exporters: \[otlp/data-prepper\] And run it like this: ./otelcol --config=otel-collector-config.yaml 1 ./otelcol --config=otel-collector-config.yaml If you get an error about the port being in use, then it’s likely that you already have another Collector running. I did – by accident! It turns out that the version of Jaeger I was using has a Collector built in – which was why it was working without an extra Collector and getting traces directly from the Java clients. Shutting down Jaeger (or running it on a different port) solves the problem and allows the new dedicated Collector to start running with Data Prepper as the sink. If it’s working correctly, then you will see messages like these: Starting GRPC server {"resource": {"service.instance.id": "…", "service.name": "otelcol", "service.version": "0.137.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "127.0.0.1:4317"} info otlpreceiver@v0.137.0/otlp.go:179 Starting HTTP server {"resource": {"service.instance.id": "…", "service.name": "otelcol", "service.version": "0.137.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "127.0.0.1:4318"} 1 2 3 4 5 6 7 Starting GRPC server {"resource": {"service.instance.id": "…", "service.name": "otelcol", "service.version": "0.137.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "127.0.0.1:4317"} info otlpreceiver@v0.137.0/otlp.go:179 Starting HTTP server {"resource": {"service.instance.id": "…", "service.name": "otelcol", "service.version": "0.137.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "127.0.0.1:4318"} Finally, go to the OpenSearch Dashboard Trace page and select Data Prepper as the data source. And now the full functionality of the OpenSearch Dashboard Trace Analytics should be available, including the Service Map, which visualizes the dependencies and service topologies as shown in this screenshot: [![How to get OpenTelemetry traces from Apache Kafka clients into OpenSearch using Data Prepper and OpenTelemetry Collectors diagram 5]()](https://www.instaclustr.com/wp-content/uploads/kafka-opensearch-data-prepper-image-5.png) Here’s a diagram of the final setup showing all the components and their relationships: [![How to get OpenTelemetry traces from Apache Kafka clients into OpenSearch using Data Prepper and OpenTelemetry Collectors diagram 6]()](https://www.instaclustr.com/wp-content/uploads/kafka-opensearch-data-prepper-image-6.png) Note that the order of running this pipeline matters. You have to work backwards from the final component to the first in this order for everything to work correctly: 1. OpenSearch 2. Data Prepper 3. OpenTelemetry Collector 4. Kafka Clients That’s it for part 2 of this exploration of how to get OpenTelemetry traces into OpenSearch. To summarize: - Option 1 ([part 1](/blog/how-to-get-opentelemetry-traces-from-apache-kafka-clients-into-opensearch-using-jaeger/)), using OpenSearch as a Jaeger database, works for an experiment with limited trace visualisation functionality. - Option 2 is more effort, has more components (Data Prepper and OpenTelemetry Collectors), requires more configuration, but unlocks the full OpenSearch Trace Analytics visualisation functionality, and is more scalable and robust for a production deployment. In a future article, we’ll explore how to use OpenSearch Observability and Trace Analytics with Apache Kafka in more detail, including what’s missing and what could be improved. ### About the author **[Paul Brebner](https://www.instaclustr.com/blog/author/paul-brebner/)** | Technology Evangelist at Instaclustr Paul has extensive R&D and consulting experience in distributed systems, technology innovation, software architecture, and engineering, software performance and scalability, grid and cloud computing, and data analytics and machine learning. [ Add Instaclustr as a preferred source on Google ](https://google.com/preferences/source?q=instaclustr.com) ![mail icon]()#### Get the latest articles for open sourceIn your inbox Sign up now ### Related content [ Apache Kafka tutorial: Get started with Kafka in 5 simple steps Apache Kafka is an open source, distributed event-streaming platform for building high-performance, real-time data pipelines and ... ](https://www.instaclustr.com/education/apache-kafka/apache-kafka-tutorial-get-started-with-kafka-in-5-simple-steps/) [ Getting started with OpenSearch®: 2 quick tutorials OpenSearch is an open source search and analytics suite used to derive insights from vast amounts of data... ](https://www.instaclustr.com/education/opensearch/getting-started-with-opensearch-2-quick-tutorials/) [ Running OpenSearch with Docker: Tutorial and best practices Learn how to run OpenSearch in Docker with a step-by-step tutorial covering single-node setup, multi-node clusters with Docker ... ](https://www.instaclustr.com/education/opensearch/running-opensearch-with-docker-tutorial-and-best-practices/) ×Sign upto ourNewsletter -----------------------