As mentioned on the following blog post by Lucas Jellema, Kafka is going to play a part in several Oracle products. For some usecases it might eventually even replace JMS. In order to allow for easy integration with Kafka, you can use Oracle Service Bus to create a virtualization layer around Kafka. Ricardo Ferreira from Oracle’s A-Team has done some great work on making a custom Kafka Service Bus transport available to us. Read more about this here, here  and here. The Kafka transport is not an ‘officially supported’ transport. Quote from the A-team blog: ‘The Kafka transport is provided for free to use “AS-IS” but without any official support from Oracle. The A-Team reserves the right of help in the best-effort capacity.’. I hope it will become an officially supported part of the Service Bus product in the future.

In this blog I summarize what I have done to get the end to end sample working for SOA Suite 12.2.1.2.0 and Kafka 0.10.1.0 based on the blogs I mentioned. This allows you to quickly start developing against Apache Kafka.

Setting up Apache Kafka

• Setting up Apache Kafka for development is easy. You follow the quickstart on: https://kafka.apache.org/quickstart. To summarize the quickstart:
• Download Apache Kafka: https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz
  
• Unzip it: tar -xzf kafka_2.11-0.10.1.0.tgz
• Go to the Kafka directory: cd kafka_2.11-0.10.1.0
• Start ZooKeeper: bin/zookeeper-server-start.sh config/zookeeper.properties
• Start a new console
• Start the Kafka broker: bin/kafka-server-start.sh config/server.properties
• Create a topic: bin/kafka-topics.sh –create –zookeeper localhost:2181 –replication-factor 1 –partitions 1 –topic test

Setting up the Kafka transport in OSB

Copy the following files:

• $KAFKA_HOME/libs/slf4j-api-1.7.21.jar
• $KAFKA_HOME/libs/kafka-clients-0.10.1.0.jar

To $OSB_DOMAIN/lib.

In my case this was /home/oracle/.jdeveloper/system12.2.1.2.42.161008.1648/DefaultDomain/lib. I’m using the JDeveloper IntegratedWebLogicServer

Download the Kafka transport from here: http://www.ateam-oracle.com/wp-content/uploads/2016/10/kafka-transport-0.4.1.zip

Extract the zip file.
Copy kafka-transport.ear and kafka-transport.jar to $MW_HOME/osb/lib/transports.

Start the domain

Execute install.py from the kafka-transport zipfile. Use wlst.sh in my case from: /home/oracle/Oracle/Middleware12212/Oracle_Home/oracle_common/common/bin/wlst.sh

Provide the required information. It will ask for Url, username, password of your WebLogic server and deploy the kafka-transport.jar and kafka-transport.ear to the specified server (AdminServer + cluster targets). If the deployments are already there, they are first undeployed by the script.

Stop the domain

The below part I got from the following blog. This is required to be able to configure the Kafka transport from the webinterface.

Locate the following file: $MW_HOME/osb/lib/osbconsoleEar/webapp/WEB-INF/lib/adflib_osb_folder.jar.

Extract this JAR and edit /oracle/soa/osb/console/folder/l10n/FolderBundle.properties.

Add the following entries:

desc.res.gallery.kafka=The Kafka transport allows you to create proxy and business services that communicate with Apache Kafka brokers.
desc.res.gallery.kafka.proxy=The Kafka transport allows you to create proxy services that receive messages from Apache Kafka brokers.
desc.res.gallery.kafka.business=The Kafka transport allows you to create business services that route messages to Apache Kafka brokers.

ZIP up the result as a new adflib_osb_folder.jar

Check the Service Bus console

After the above steps are completed, you can start the domain and use the Kafka transport from the servicebus console.

Setting up JDeveloper

Also you will not see possible options for consumer or producer settings but you can use the settings from: here and here

Running an end to end sample

Apache Kafka provides shell scripts to test producing and consuming messages:
– Producing: bin/kafka-console-producer.sh –broker-list localhost:9092 –topic test
– Consuming: bin/kafka-console-consumer.sh –bootstrap-server localhost:9092 –topic test –from-beginning

It helps to add a report, log or alert action to your Service Bus pipeline so you can see messages which have passed. As a report key I have used the Kafka offset from $inbound: ./ctx:transport/ctx:request/tp:headers/kafka:offset

And now?

As you can see, several steps need to be performed to install this custom transport. It is only supported on a best-effort basis by the A-Team. I could not see options for properties in the Service Bus Console as was shown in the blog posts mentioned at the start of this post, but that is not a real issue since if a fixed set would be provided and more options would become available in a new version of Kafka, this might become limiting. It is a shame custom transports are not visible in the component palette in JDeveloper. Once you know however you can use the Kafka transport by creating Proxy and Business services from File, New this also becomes a non-issue.

There are of course other solutions to take care of the integration with Kafka such as using Kafka connectors or create a custom service to wrap Kafka, but I like the way this custom transport allows you to integrate it with Service Bus. This allows you to make Kafka only available through this channel. This offers options like easily applying policies, monitoring, alerting, etc. I do expect in Oracle’s Cloud offering interaction with Kafka products running in the Oracle Cloud such as the Event Hub, will of course be much easier. We’re looking forward to it.