Scaling agent systems with Apache Kafka and A2A (Part 7): An experimental Kafka transport and the Semaphore Agent

What changes when agent requests and task updates travel through Kafka topics instead of HTTP connections?

In this installment of our series on scaling agent systems, we arrive at one of the planned destinations on our A2A + Kafka “campaign” map – an experimental Kafka A2A transport. We still use the same A2A semantics but transport them over Kafka.  In the Napoleonic era, this would have been like replacing dispatch riders with the faster semaphore telegraph system introduced in France during the 1790s.

The system used tall poles on towers with rotating wooden semaphores.  Operators in nearby towers could retransmit the signals along the line of towers, enabling messages to be transmitted hundreds of kilometres per hour.

The French semaphore system shares some interesting conceptual similarities with Apache Kafka, moving large volumes of information reliably across a geographically distributed network; multiple networks extending 100’s of km over the whole of France, and beyond. Relay towers functioned as brokers, producers and consumers, and operators reacted quickly to changes of state in the observed semaphores, relaying them to the next tower and so on. They also shared a major design feature with Kafka – the operators couldn’t understand the messages – only the people sending (encoding) and receiving (decoding) the messages could understand them using code books.

Kafka brokers also don’t understand messages – only the producers and consumers do.

Chappe’s telegraph (Public domain, https://commons.wikimedia.org/wiki/File:Chappe_semaphore.jpg)

About this series: This is Part 7 of Scaling agent systems with Apache Kafka and A2A. New to the topic? Start with Part 1: The evolution of agent systems. For the Java examples this experiment builds on, see Part 5: The Clockwork Agent and Part 6: The Quartz Chronometer Agent.

Why investigate A2A over Kafka?

The primary motivation for this experiment was to explore whether A2A could be implemented on top of Apache Kafka rather than its default transports of JSON-RPC over HTTP, SSE, and webhooks.

Kafka is a widely used platform for integration, event streaming, and large-scale distributed systems. Given its scalability, durability, fan-out capabilities, and replayable event log, it seems like a natural candidate for scaling Agent2Agent systems.

Does that mean standard A2A doesn’t scale?

No.

The default A2A HTTP binding, based on JSON-RPC and Agent Card discovery, scales well for traditional point-to-point interactions between clients and agents. A client sends a request, receives either an immediate response or a long-running task, and tracks progress through polling, SSE, or webhooks.

The challenges begin to appear when systems grow beyond simple client-agent communication. Polling generates repeated requests even when nothing has changed. SSE requires long-lived connections for every observer. Webhooks replace inbound traffic with outbound traffic and introduce delivery, retry, and idempotency concerns.

At the platform level, A2A is designed as a point-to-point protocol rather than a distributed event fabric. Agent task state is often held in-process, meaning horizontally scaled agent pools may require sticky routing or shared storage. Likewise, task lifecycle events are not automatically broadcast to multiple consumers for auditing, monitoring, analytics, or orchestration.

As workloads increase, with many agents, many concurrent long-running tasks, and many observers, additional infrastructure may be needed to provide fan-out, replay, durability, and centralized visibility. These capabilities sit outside the A2A protocol itself.

In short, the default A2A protocols scale normal client-agent interactions very well. However, they do not automatically provide large-scale event distribution, replay, auditing, or multi-observer task telemetry. Those concerns require supporting architecture beyond polling, streaming, and webhooks.

Different implementation choices introduce different scaling challenges:

Polling (GetTask):

• Generates extra RPCs even when task state is unchanged
• May return the same state multiple times
• Can become expensive with many clients and many tasks

SSE (Streaming)

• Requires one long-lived connection per observer
• Works well for individual clients but is less efficient for large numbers of subscribers
• Replay and reconnect behaviour depend on the agent implementation rather than a durable event log

Push (Webhooks)

• Requires the agent to POST updates to every registered endpoint
• Often still requires a follow-up GetTask request to retrieve full state
• Retry, delivery guarantees, and idempotency must be implemented separately

Running Many Agent Instances

• Task state frequently resides within a single process
• Load balancing can be complicated without sticky sessions or shared storage
• Requests related to the same task may reach different agent instances

Fan-Out and Orchestration

• No built-in one-to-many event distribution
• Audit pipelines, dashboards, and orchestration systems typically require additional infrastructure
• Replay capability is not provided by the protocol itself

The bottom line: Kafka is a candidate for that supporting event infrastructure because it provides a retained log and consumer groups which means A2A scales point-to-point agent communication effectively. The question is not whether Kafka replaces A2A, but how an event-streaming architecture might carry selected A2A interactions.

A2A is optimized for point-to-point communication. Kafka potentially complements it by adding durable event storage, replay, fan-out, and other distributed-systems capabilities that become increasingly valuable at scale.

How does the experimental A2A over Kafka design work?

For the new A2A Kafka design, we need two components:

1. A Kafka implementation of the A2A patterns
2. New A2A Kafka clients

Let’s see what this looks like in more detail, starting with some diagrams. First up, here’s a conceptual diagram showing how standard A2A modes were implemented in Kafka:

This diagram shows how three standard A2A interaction shapes map onto the same Kafka transport, using one Client, one broker, and one agent throughout. They are comparison blocks, not concurrent flows; just pick the one required pattern per task. Read top to bottom through the three shaded sections to compare event count, timing, and delivery semantics on identical infrastructure.

All three start the same way: the client produces message/send on a topic a2a.requests, the agent consumes the request, and replies go to topic a2a.updates.

• Pattern 1 (request/response) is the short path — artifact, COMPLETED, task snapshot, then the client consumes until terminal.
• Pattern 2 (streaming) adds WORKING, then a loop of artifact chunks consumed in order, ending with COMPLETED; on this binding the Kafka consumer replaces HTTP SSE.
• Pattern 3 (notifications) publishes the same kind of lifecycle events, but the client pulls them asynchronously from the log when ready — it does not receive an HTTP POST to a webhook URL.

In the A2A Kafka POC, standard A2A semantics are preserved in JSON-RPC-style envelopes on Kafka topics — but the three HTTP delivery paths (poll, SSE, push) are not implemented as three separate mechanisms. They collapse into produce to topic a2a.requests → consume from topic a2a.updates (or, potentially, a per-client reply topic).

Mapping interactions to Kafka topics

The Record key = taskId (ordering per task). The client can set metadata.replyTopic (ReplyRouting) so the agent publishes updates to a dedicated topic (used in future benchmarks).

Semaphore Agent: Adapting the Clockwork Agent Implementation

The Semaphore Agent adapts the asynchronous countdown from the earlier Clockwork and Quartz examples to the Kafka transport. It’s the same 10-second tick story as Clockwork/Quartz, but no HTTP, SSE, or webhooks.

The main components are:

Following a countdown from request to completion

The countdown sequence demonstrates the transport end to end with A2A Kafka. Here’s a diagram that shows concretely how our countdown example works.

This sequence diagram (as shown below) illustrates the countdown demo when Kafka is the entire A2A wire.

1. The client publishes a message/send envelope to the topic a2a.requests
2. The agent consumes it and publishes lifecycle events to the topic a2a.updates (status-update, artifact-update, final task).
3. The client consumes those updates until the task completes.
4. An optional AuditEventConsumer uses a separate consumer group on the same updates topic to illustrate fan-out (same events, multiple readers).
5. Discovery via a2a.agent-cards is shown as optional.
6. The point: poll, SSE, and push on HTTP are all collapsed into produce/consume on Kafka topics with the same JSON-RPC-shaped envelopes.

Explore the implementation on GitHub

The Kafka A2A implementation and demo contain approximately 2,114 lines of code, around 3.7 times larger than the earlier Quartz Chronometer example (566 Java LOC). The additional complexity comes from Kafka client plumbing, transport models, agents, and supporting infrastructure. Explore the code and demo here.

What the proof of concept does not yet implement

For a transport POC demo, the code is solid; for a complete, spec-advertisable Kafka binding, several important pieces are still missing or deliberately skipped.  Critical gaps (if you want “real A2A on Kafka”) are as follows:

1. Discovery and binding are unfinished:

• Countdown hard-codes a2a.requests / a2a.updates (there’s currently no card-driven lookup)
• Agent Cards use placeholder url: "kafka://a2a" with no actionable topic metadata
• No KafkaA2AClient.discover(skill) → resolve topics → send flow

Without that, peers can’t interoperate the way HTTP A2A does via supportedInterfaces.

2. No interoperability with standard a2a-java/ HTTP clients

This is by design, so not really a bug — but it’s critical if you expect a drop-in replacement for the Quartz Chronometer Demo. Generic clients speak HTTP JSON-RPC + SSE/push; this POC needs Kafka producers/consumers on both sides.

3. Several A2A operations aren’t on the wire

4. Countdown ≠ full Clockwork mechanisms

Kafka countdown is async countdown only. Not on Kafka yet:

• Ex 1 — sync time → immediate Message (no task)
• Ex 3 — input-required + confirm continuation

(TaskState.INPUT_REQUIRED exists in the model; countdown agent doesn’t use it.)

5. Multi-agent routing – not yet.

My initial design docs describe topic-per-type, tenant isolation, routers; but the code is limited to shared topics and a homogeneous pool of consumers.

What are the real limitations?

This Kafka transport remains an experimental proof of concept; it’s not fully tested or usable as a drop-in replacement for the default A2A protocol. The main limitations are:

Like a temporary Napoleonic-era encampment before the next advance, this proof of concept provides a useful staging point for demos, experimentation, and benchmarking. That’s where we’ll go next.

Bringing the series together: agent interactions and an event backbone

Across this series, we have moved from the foundations of agent systems to the Agent2Agent object model, runtime interactions, visual explanations, and working Java examples.

The Clockwork and Quartz agents made the interaction patterns concrete. The Semaphore Agent adds a different architectural perspective: selected A2A-style interactions carried through a shared Kafka event log, with independent consumers and replay of retained updates.

The takeaway is not that Kafka replaces A2A. A2A defines how agents exchange work and results; Kafka offers an event-streaming foundation for distributing and retaining the resulting events. This experiment shows one way to connect those ideas, while making the remaining compatibility, operational, and testing work explicit.

For developers exploring agent systems, that is the larger lesson of the series: separate the interaction model from the infrastructure carrying it, then evaluate each against the behavior your application actually needs.

Start with the Semaphore Agent code to follow a request through its lifecycle, observe the same updates from another consumer group, and explore replay. Completing discovery, operation coverage, routing, security, and validation would be potential next steps—not capabilities this proof of concept already provides.

Want to explore the entire series?

Scaling agent systems with Apache Kafka and A2A

• Part 1: The evolution of agent systems
• Part 2: Understanding the A2A object model
• Part 3: Agent2Agent protocol explained
• Part 4: Visualizing the Agent2Agent protocol
• Part 5: The Clockwork Agent—a Java A2A example
• Part 6: The Quartz Chronometer Agent—an example using the default A2A protocols
• Part 7: An experimental Kafka transport and the Semaphore Agent (you’re here)

For the Kafka infrastructure layer, try NetApp Instaclustr for Apache Kafka free. A managed Kafka service addresses cluster operations; it does not complete the missing A2A functionality or make this experimental transport production-ready.