I am trying to configure Kubernetes pods to send and receive UDP Multicast messages. I have two clusters, each is running on a separate Proxmox VM. Each cluster is using Flannel as the base CNI with Multus.
I have tried the following:
- I created the following Network Attachment Definition IPVLAN Network Attachment Definition and the following annotation to my pod IPVLAN Annotation the pod on the 2nd cluster has the same Network Attachment Definition and annotation (different IP and mac). With this configuration I can ping from 1 pod to the other, but using IPERF to send Multicast data fails.
- I tried the same thing with the following Network Attachment Definition MACVLAN Network Attachment Definition and this annotation MACVLAN Annotation with the same result: IPERF fails
- I removed Multus annotations and set the hostNetwork to true. This worked, IPERF was able to send Multicast data between the pods. However, another portion of the setup failed. I have another app running outside the cluster in a Docker container (this is a simulator that will not be part of the final deployment but is needed for testing). This app sends data to all pods through NodePort services. If I set hostNetwork to true the simulator can no longer reach the associated pod.
- I also tried the configuration mentioned in the answer to this thread Multicast between kubernetes pods, but with this I couldn't even ping from one pod to the other.
Here's how I use IPERF. On one pod I run IPERF Server command and get this output IPERF Server output. On the other pod I run IPERF Client command and get this output IPERF Client output.
One thing I notice is local IP is for the client is a 10.42.#.# address which is a cluster internal IP. I don't know how to change that.
