I have a Task Definition called kafka-zookeeper, it has an Apache Kafka broker and a Zookeeper node in it. Those containers need to communicate with each other (that's easy) and with the kafka and zookeeper containers from other kafka-zookeeper tasks, which is what I don't know how to do.
Basically, I need this:
------------------------ ___________________ | ___________________ | ___________________ | Task1 | | | Task2 | | | Task2 | |(kafka-zookeeper)| | |(kafka-zookeeper)| | |(kafka-zookeeper)| | _____________ | | | _____________ | | | _____________ | | | |<------ | | | | --->| | | | | Kafka |<-------->| Kafka |<-------->| Kafka | | | |___________| | | |___________| | | |___________| | | ^ | | ^ | | ^ | | | | | | | | | | | _____v_______ | | ______v______ | | ______v______ | | | | | | | | | | | | | | | Zookeeper |<-------->| Zookeeper |<-------->| Zookeeper | | | |___________|<------ | |___________| | --->|___________| | |_________________| | |_________________| | |_________________| | | ------------------------ But I don't know how to setup the communication between the tasks since the tasks' private IPs are determined once they're started. Also, if a task fails for some reason, when another one is created in its place the IP will be different. How do I deal with this?
I also think it's worth mentioning that zookeeper and kafka know about other instances by environment variables containing the IPs of the other instances.
I also can't give hostnames to containers because I'm using the awsvpc network mode.
Thanks in advance!
