DEV Community

eidher
eidher

Posted on

Kafka Connect: FileStreamSourceConnector in distributed mode

#kafka #kafkaconnect #filestreamsourceconnector

1) Source connectors: Start our kafka cluster. This is the docker-compose.yml file:

version: '2' services: kafka-cluster: image: landoop/fast-data-dev:cp3.3.0 environment: ADV_HOST: 127.0.0.1 RUNTESTS: 0 ports: - 2181:2181 # Zookeeper - 3030:3030 # Landoop UI - 8081-8083:8081-8083 # REST Proxy, Schema Registry, Kafka Connect - 9581-9585:9581-9585 # JMX Ports - 9092:9092 # Kafka Broker
Enter fullscreen mode Exit fullscreen mode

Create and start the container:

docker-compose up kafka-cluster
Enter fullscreen mode Exit fullscreen mode

2) Create the topic we're going to write to:

docker run --rm -it --net=host landoop/fast-data-dev:cp3.3.0 bash kafka-topics --create --topic demo-2-distributed --partitions 3 --replication-factor 1 --zookeeper 127.0.0.1:2181
Enter fullscreen mode Exit fullscreen mode

3) In a browser go to 127.0.0.1:3030 -> Connect UI
Create a new connector -> File Source
Paste the next configuration and click the Create button:

name=file-stream-demo-distributed connector.class=org.apache.kafka.connect.file.FileStreamSourceConnector tasks.max=1 file=demo-file.txt topic=demo-distributed key.converter=org.apache.kafka.connect.json.JsonConverter key.converter.schemas.enable=true value.converter=org.apache.kafka.connect.json.JsonConverter value.converter.schemas.enable=true
Enter fullscreen mode Exit fullscreen mode

4) Create the source file:
Execute docker ps to see the containerId.

docker exec -it <containerId> bash touch demo-file.txt echo "add any content" >> demo-file.txt
Enter fullscreen mode Exit fullscreen mode

3) In a browser go to 127.0.0.1:3030 -> Kafka Topics UI
Click the topic defined previously (demo-distributed) and validate the content:

Image description

Top comments (0)