iPaaS Insights

Apache's open source event distributed event streaming framework Apache Kafka gives developers the tools needed to create powerful data pipelines and applications. You can think of it as the central nervous system that processes events and messages that flows throughout your data infrastructure.

But if Kafka is the nervous system that translates your integrations into identifiable events, then Apache Camel is the connective tissue that binds those integrations together. Camel is the open source integration framework that gives developers the freedom to create cloud-native integrations. Connecting the two together lets you connect live data processed through Kafka to every integration you create via Camel.

Let's detail the steps it takes to connect Apache Kafka and Camel together, and the benefits you reap by doing so. For this example, we'll create an integration through Camel that writes data from your Kafka deployment into a text file.

Setting Up Kafka on Apache Camel

Camel has included a component specifically for Kafka since version 2.13, allowing users to easily connect to Kafka directly. Configuring this component to work with your specific build is simple. Below is a step-by-step guide on how to set up Kafka on your Camel runtime.

NOTE: For Maven users, before connecting Kafka onto your Camel deployment, you need to add this dependency to your pom.xml file:

<dependency>
    <groupId>org.apache.camel</groupId>
    <artifactId>camel-kafka</artifactId>
    <version>x.x.x</version> // replace 'x.x.x' with your version of Camel Core
</dependency>

Initializing Kafka

Once you have Kafka downloaded and extracted to your install directory, you'll need to initialize Kafka. First, run the ZooKeeper service:

$ bin/zookeeper-server-start.sh config/zookeeper.properties

Then, run the Kafka broker:

$ bin/kafka-server-start.sh config/server.properties

Next, you'll need to create a new Kafka topic to run events on. We'll create a my-test-topic topic for this example:

// replace the variable $KAFKA_HOME with your Kafka install directory
$KAFKA_HOME/bin/kafka-topics.sh --create \
  --zookeeper localhost:2181 \
  --replication-factor 1 \
  --partitions 1 \
  --topic my-test-topic

Now, you'll have to run a Kafka consumer and producer to write and read events, respectively. Start the consumer by running:

$KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-test-topic --from-beginning

And for the producer, run:

$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-test-topic

Running the Kafka Connector

After Kafka is installed and configured, it's time to run the Camel Kafka Connector. This will allow you to create connections with Camel's Kafka components. For our file example, you'll need to build the camel-kafka-connector project locally, then run the file connector with:

cd connectors/camel-log-kafka-connector/

You'll also need to unzip the camel-aws-sqs-kafka-connector archive to your plugin.path location. This will create a camel-file-kafka-connector-0.6.0-SNAPSHOT-package.zip file in your connector directory.

Let's unzip that file:

> cd /home/connectors/
> cp connectors/camel-file-kafka-connector/target/camel-file-kafka-connector-0.6.0-SNAPSHOT-package.zip .
> unzip camel-file-kafka-connector-0.6.0-SNAPSHOT-package.zip

Finally, let's run the file sink. This will create a text file at /tmp/kafkaconnect.txt.

$KAFKA_HOME/bin/connect-standalone.sh $KAFKA_HOME/config/connect-standalone.properties docs/examples/CamelFileSinkConnector.properties

And you're done!

What Can You Do with Kafka and Camel Together?

Now that you've connected Kafka and Camel together, what can your team accomplish? These two services go hand in hand in giving your developers access to real-time data for your integrations. Thus, your engineers and applications always have the most recent data to work with.

Kafka's event streaming platform ensures that data continuously flows from their sources throughout your data pipeline as actionable event-based information. Camel can interpret events processed and delivered by Kafka and send them to connected endpoints based on the properties of the event.

Once your Camel endpoints are configured correctly, Kafka-driven events can trigger responses to and from multiple applications at once. Apache currently operates 125 connectors with Camel Kafka, with support for a number of different applications and services. Compatible sources / destinations include Amazon Web Services, Microsoft Azure, Kubernetes, Dropbox, Salesforce, Google Mail, MongoDB, and Twitter.

Camel Kafka includes both source connectors (for delivering data onto your Kafka deployment) and sink connectors (for gathering data from Kafka). Not every supported service has both a source and sink connector, however. The GitHub connector, for example, only has a source connector for now — meaning you can pull data from GitHub into your Kafka environment through Camel, but not push data into it.

For a full list of currently available Camel Kafka connectors, click here.

Want to learn more about connecting Apache Kafka and Camel together? You can consult the official documentation for Kafka and Camel or Apache's installation and configuration guide for Camel's Kafka Connecter.