Base project for kafka-streams development and testing
- Linux
~ ❯ cat /etc/*-release ... NAME="Ubuntu" VERSION="17.04 (Zesty Zapus)" ...- docker
~ ❯ docker -v Docker version 17.12.0-ce, build c97c6d6- docker-compose
~ ❯ docker-compose -v docker-compose version 1.13.0, build 1719ceb- docker-machine
~ ❯ docker-machine -v docker-machine version 0.13.0, build 9ba6da9- virtualBox
~ ❯ vboxmanage -v ⏎ 5.1.22_Ubuntur115126- Intel(R) Core(TM) i7-4700HQ CPU @ 2.40GHz (8 cores)
- 1Gbit/s Killer E220x Gigabit Ethernet Controller
- 16GiB of Memory
- 1TB ATA Disk
- Create three docker-machine instances (node-1, node-2, node-3) with 4GB of memory and 2 CPU cores each:
# Use a loop if you like :) docker-machine create --driver virtualbox --virtualbox-no-dns-proxy --virtualbox-memory "4096" --virtualbox-cpu-count "2" node-1 docker-machine create --driver virtualbox --virtualbox-no-dns-proxy --virtualbox-memory "4096" --virtualbox-cpu-count "2" node-2 docker-machine create --driver virtualbox --virtualbox-no-dns-proxy --virtualbox-memory "4096" --virtualbox-cpu-count "2" node-3- Setup node-1 as swarm manager :
# Get node-1 ip address MANAGERIP=$(docker-machine ssh node-1 "ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p' | grep 99") # Init swarm group and set node-1 as manager docker-machine ssh node-1 "docker swarm init --advertise-addr $MANAGERIP --listen-addr $MANAGERIP"- Join node-2 and node-3 as workers:
# Get swarm group join token SWARMTOKEN=$(docker-machine ssh node-1 "docker swarm join-token -q worker") # join node-2 as workers docker-machine ssh node-2 "docker swarm join --token $SWARMTOKEN $MANAGERIP:2377" # join node-2 as workers docker-machine ssh node-3 "docker swarm join --token $SWARMTOKEN $MANAGERIP:2377"- Make sure that your Linux host machine has the right dns in
bash /etc/resolv.conf
~ ❯ cat /etc/resolv.conf nameserver 8.8.8.8- Check that all nodes are up
~ ❯ docker-machine ls ⏎ NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS node-1 - virtualbox Running tcp://192.168.99.100:2376 v17.12.1-ce node-2 - virtualbox Running tcp://192.168.99.101:2376 v17.12.1-ce node-3 - virtualbox Running tcp://192.168.99.102:2376 v17.12.1-ce - Check that swarm group is set up correctly
~ ❯ docker-machine ssh node-1 'docker node ls' ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS oqs2piqwdhl7axk59yunl4muu * node-1 Ready Active Leader wmu77zmdhq60ge579jxs6x7as node-2 Ready Active ukeq86oqsz0ove0lleic9xk0a node-3 Ready Active - connect to the swarm manager node
~ ❯ docker-machine ssh node-1 ## . ## ## ## == ## ## ## ## ## === /"""""""""""""""""\___/ === ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~ \______ o __/ \ \ __/ \____\_______/ _ _ ____ _ _ | |__ ___ ___ | |_|___ \ __| | ___ ___| | _____ _ __ | '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__| | |_) | (_) | (_) | |_ / __/ (_| | (_) | (__| < __/ | |_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_| Boot2Docker version 17.12.1-ce, build HEAD : 42357fc - Wed Feb 28 17:52:00 UTC 2018 Docker version 17.12.1-ce, build 7390fc6 docker@node-1:~$ - copy and paste the infra/kafka-cluster-swarm.yml content and run the command below
docker@node-1:~$ docker stack deploy --compose-file kafka-cluster-swarm.yml kafka-cluster- navigate to container visualizer at http://192.168.99.100:8080 to check stack services status :)
NB: You need to modify the stack Yaml file to suit configuration for your test here i show how to run them only
- on Linux host machine download Kafka and extract it somewhere
/tmp ❯ wget -qO- http://apache.crihan.fr/dist/kafka/1.0.0/kafka_2.12-1.0.0.tgz | tar xvz /tmp ❯ cd kafka_2.12-1.0.0- launch a perf test
# create test topic kafka_2.12-1.0.0 ❯ bin/kafka-topics.sh --create --zookeeper 192.168.99.100:12181,192.168.99.101:22181,192.168.99.102:32181 --replication-factor 1 --partitions 2 --topic benchmark-1-2-none # run perf script kafka_2.12-1.0.0 ❯ bin/kafka-producer-perf-test.sh --topic benchmark-2-2-none --num-records 15000000 --record-size 100 --throughput 15000000 --producer-props acks=1 bootstrap.servers=192.168.99.100:19092,192.168.99.101:29092 buffer.memory=67108864 compression.type=none batch.size=8196 1396651 records sent, 278272.8 records/sec (26.54 MB/sec), 1433.7 ms avg latency, 2278.0 max latency. 2113736 records sent, 422747.2 records/sec (40.32 MB/sec), 1423.1 ms avg latency, 1965.0 max latency. 2194914 records sent, 438982.8 records/sec (41.86 MB/sec), 1349.2 ms avg latency, 2281.0 max latency. 2185516 records sent, 437103.2 records/sec (41.69 MB/sec), 1398.3 ms avg latency, 2538.0 max latency. 2134530 records sent, 426906.0 records/sec (40.71 MB/sec), 1441.6 ms avg latency, 2425.0 max latency. 2186552 records sent, 437310.4 records/sec (41.71 MB/sec), 1360.0 ms avg latency, 2211.0 max latency. 2179966 records sent, 435993.2 records/sec (41.58 MB/sec), 1372.1 ms avg latency, 2256.0 max latency. 15000000 records sent, 407110.869860 records/sec (38.83 MB/sec), 1407.67 ms avg latency, 2538.00 ms max latency, 2010 ms 50th, 2385 ms 95th, 2487 ms 99th, 2536 ms 99.9th.