Downloaded 74 times
Uploaded byHungWei Chiu
Overview of kubernetes network functions
This document provides an overview of Kubernetes networking, including its functionalities, services, and challenges. It explains concepts like container networking, the role of the Container Network Interface (CNI), and how to implement Kubernetes networking effectively. Additionally, it addresses advanced networking scenarios including the use of hardware acceleration and the Multus CNI for multiple network interfaces in pods.
Overview of kubernetes network functions
- 1.
- 2. COSCUP2018 x openSUSE.Asia GNOME.Asia I amHung-Wei Chiu Co-organizer of SDNDS-TW Co-organizer of CNTUUG I love Linux Network/Kubernetes/SDN You can find me at: blog.hwchiu.com
- 3. COSCUP2018 x openSUSE.Asia GNOME.Asia Outline What networkfunctions about k8s How does above function implement What is the challenge about k8s
- 4. COSCUP2018 x openSUSE.Asia GNOME.Asia What networkfunction kubernetes providers?
- 5. COSCUP2018 x openSUSE.Asia GNOME.Asia What Container Network ○Connectivity ○ DNS Kubernetes services
- 6. COSCUP2018 x openSUSE.Asia GNOME.Asia Do YouKnow How Container Works?
- 7. COSCUP2018 x openSUSE.Asia GNOME.Asia Containersvs.VMs Containers areisolated, but share OS and where appropriate bins/libraries
- 8. COSCUP2018 x openSUSE.Asia GNOME.Asia HowDockerWorks We knowdocker is isolated, but how does it works? Linux kernel support the Namespaces mechanisms to partition kernel resources to different processes
- 9. COSCUP2018 x openSUSE.Asia GNOME.Asia HowDockerWorks Mount namespaces IPCnamespaces PID namespaces Network namespaces User namespaces UTS namespaces ○ Unix Time System
- 10. COSCUP2018 x openSUSE.Asia GNOME.Asia NetworkNamespace Isolate thenetwork functions. Including the ○ Network interfaces ○ Routing rules ○ Netfilter (iptables)
- 11.
- 12.
- 13. COSCUP2018 x openSUSE.Asia GNOME.Asia docker0 docker0 ns1 LinuxHost Linux Host Linux Host
- 14. COSCUP2018 x openSUSE.Asia GNOME.Asia docker0 docker0 docker0 ns1 ns1 veth1 veth0 LinuxHost Linux Host Linux Host Linux Host
- 15. COSCUP2018 x openSUSE.Asia GNOME.Asia docker0 docker0 docker0docker0 ns1 ns1ns1 vth1 vth0 veth eth0 LinuxHost Linux Host Linux Host Linux HostLinux Host
- 16. COSCUP2018 x openSUSE.Asia GNOME.Asia Before wetalk about service, we must know why service exist.
- 17. COSCUP2018 x openSUSE.Asia GNOME.Asia Pods/Deployments We candeploy our applications as a containers in the kubernetes. There’re many kind of the container we can deploy ○ Pod ○ Deployment ○ Statefulset ○ DaemonSet
- 18. COSCUP2018 x openSUSE.Asia GNOME.Asia Deployment Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster Deployment: ○Ngnix ○ Replica: 3 10.123.234.56 10.123.234.57 10.123.234.58
- 19. COSCUP2018 x openSUSE.Asia GNOME.Asia Access How weapplication access those Nginx servers? By IP address ○ 10.123.234.56:80 ○ 10.123.234.57:80 ○ 10.123.234.58:80 What’s the problem
- 20. COSCUP2018 x openSUSE.Asia GNOME.Asia Deployment Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster Deployment: ○Ngnix ○ Replica: 3 10.123.234.56 10.123.234.57 10.123.234.58
- 21. COSCUP2018 x openSUSE.Asia GNOME.Asia Deployment Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster Deployment: ○Ngnix ○ Replica: 3 10.123.234.56 10.123.234.57 10.123.234.75
- 22. COSCUP2018 x openSUSE.Asia GNOME.Asia Access How weapplication access those Nginx servers? By IP address ○ 10.123.234.56:80 ○ 10.123.234.57:80 ○ 10.123.234.58:80 ○ 10.123.234.75:80 It’s not easy for our application to handle those ip-changed situation.
- 23. COSCUP2018 x openSUSE.Asia GNOME.Asia The Serviceis used to solve this problem.
- 24. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster 10.123.234.5610.123.234.57 10.123.234.58 App Service Nginx
- 25. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Application toService ○ We use the DNS to access the service. ○ $(service).$(namespace).cluster.local Service to Pods ○ Service maintains all IP addresses of all Pods. ○ We call it endpoints
- 26. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster 10.123.234.5610.123.234.57 10.123.234.58 App Service Nginx nginx.default endpoints
- 27. COSCUP2018 x openSUSE.Asia GNOME.Asia How doeskubernetes implements those functions?
- 28. COSCUP2018 x openSUSE.Asia GNOME.Asia What Container NetworkConnectivity ○ Container Network Interface(CNI) Kubernetes Services ○ There’re many implementation we can choose
- 29.
- 30. COSCUP2018 x openSUSE.Asia GNOME.Asia ContainerNetworkInterface Cloud NativeComputing Foundation Project Consists of a specification and libraries. Configure network interfaces in Linux containers Concerns itself only with network connectivity of containers ○ Create/Remove
- 31. COSCUP2018 x openSUSE.Asia GNOME.Asia ContainerNetworkInterface Removing allocatedresources when the container is deleted
- 32.
- 33. COSCUP2018 x openSUSE.Asia GNOME.Asia FromtheGITHUB l rkt- container engine l Kubernetes - a system to simplify container operations l OpenShift - Kubernetes with additional enterprise features l Cloud Foundry - a platform for cloud applications l Apache Mesos - a distributed systems kernel l Amazon ECS - a highly scalable, high performance container management service
- 34. COSCUP2018 x openSUSE.Asia GNOME.Asia So, Howto use the CNI?
- 35. COSCUP2018 x openSUSE.Asia GNOME.Asia StepbyStep Create akubernetes cluster Setup your CNI plugin Deploy your first Pod
- 36. COSCUP2018 x openSUSE.Asia GNOME.Asia Just followthe installation to install the kubernetes
- 37. COSCUP2018 x openSUSE.Asia GNOME.Asia How dowe install the CNI?
- 38.
- 39. COSCUP2018 x openSUSE.Asia GNOME.Asia Handbyhand In thekubelet, we have the following parameters for CNI. --cni-bin-dir ○ /opt/cni/bin --cni-conf-dir ○ /etc/cni/net.d/ We should config the CNI for every k8s nodes.
- 40.
- 41.
- 42. COSCUP2018 x openSUSE.Asia GNOME.Asia Beforewestart Pod ○ Acollection of containers
- 43. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load thePod config ○ Multiple containers Find a node to deploy the pod Create a Pause container
- 44.
- 45. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load thePod config ○ Multiple containers Find a node to deploy the pod Create a Pause container Load the CNI config
- 46. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Loadthe CNI config from /etc/cni/net.d/…
- 47.
- 48. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load thePod config ○ Multiple containers Find a node to deploy the pod Create a Pause container Load the CNI config Execute the CNI
- 49. COSCUP2018 x openSUSE.Asia GNOME.Asia Callthebinary Load thebinary from the config Find the binary from the /opt/cni/bin/
- 50. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Callthe /opt/cni/bin/flannel Network Connectivity
- 51. COSCUP2018 x openSUSE.Asia GNOME.Asia Steps Load thePod config ○ Multiple containers Find a node to deploy the pod Create a Pause container Load the CNI config Execute the CNI Create target containers and attach to Pause container
- 52. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Callthe /opt/cni/bin/flannel Network Connectivity Busybox
- 53. COSCUP2018 x openSUSE.Asia GNOME.Asia Attachtoothercontainer docker run–net=… ○ bridge ○ host ○ containerID
- 54. COSCUP2018 x openSUSE.Asia GNOME.Asia Linux Host Pause Container Callthe /opt/cni/bin/flannel Network Connectivity Busybox Pod
- 55.
- 56. COSCUP2018 x openSUSE.Asia GNOME.Asia Kubernetesservice There’re threeimplementation now. ○ User-space Kube-Proxy ○ Kernel-space iptables (default) ○ Kernel-space ipvs We use the iptables to explain how service(ClusterIP) works
- 57. COSCUP2018 x openSUSE.Asia GNOME.Asia Service Node1 Nginx Node2 Nginx Node3 Nginx Kubernetes Cluster 10.123.234.5610.123.234.57 10.123.234.58 App Service Nginx nginx.default endpoints
- 58. COSCUP2018 x openSUSE.Asia GNOME.Asia LAB Get theService ○ kubectl get service
- 59. COSCUP2018 x openSUSE.Asia GNOME.Asia LAB Get theendpoints ○ kubectl get endpoints
- 60. COSCUP2018 x openSUSE.Asia GNOME.Asia LAB Get thepod ip address ○ kubectl get pods –o wide
- 61. COSCUP2018 x openSUSE.Asia GNOME.Asia Now.Trytofetchthenginx Service nameis k8s-nginx-cluster Use the nslookup to lookup the IP ○ nslookup k8s-nginx-cluster ○ nslookup k8s-nginx-cluster.default ■ default is the namespace of the service
- 62.
- 63. COSCUP2018 x openSUSE.Asia GNOME.Asia Trytofetchthenginx In thepod: curl k8s-nginx-cluster
- 64. COSCUP2018 x openSUSE.Asia GNOME.Asia Howitworks? First, wecan get the VIP from the hostname. ○ It’s just a DNS request. Second, we can access the nginx from that VIP address. ○ iptables!!
- 65. COSCUP2018 x openSUSE.Asia GNOME.Asia Lte’swatchtheiptables First, wecan use the service name to filter the iptables rules. sudo iptables-save | grep ”k8s- nginx-cluster”
- 66. COSCUP2018 x openSUSE.Asia GNOME.Asia Lte’swatchtheiptables Remember? There’rethree endpoints for the service now.
- 67. COSCUP2018 x openSUSE.Asia GNOME.Asia workflowsPackets Packets Packets Match Services’ ClusterIP Findthe endpoints DNAT KUBE-SERVICES KUBE-SVC-XXXX KUBE-SEP-XXXX Enter iptables PREROUTING Jump to other chain Jump to other chain Jump to other chain
- 68. COSCUP2018 x openSUSE.Asia GNOME.Asia Howdowechoosewhichonetouse? When wematch the clusterIP:Port, goto another custom chain. ○ 10.105.100.214:80
- 69. COSCUP2018 x openSUSE.Asia GNOME.Asia Howdowechoosewhichonetouse? Use therandom to choose what endpoint we use.
- 70. COSCUP2018 x openSUSE.Asia GNOME.Asia EP1 EP2EP3 P < 0.33 P < 0.5 EP1 EP2 EP3 P= 1/3 P= 2/3 * 1/2 = 1/3 P= 2/3 * 1/2 = 1/3
- 71. COSCUP2018 x openSUSE.Asia GNOME.Asia EP1 EP2EP3 P < 0.2 P < 0.25 EP1 EP2 EP3 EP4 EP5 P < 0.33 P < 0.5 EP4 EP5 P = 0.2 P = 4/5 * 1/4 = 1/5 P = 4/5 * 3/4 * 1/3 = 1/5 P = 4/5 * 3/4 * 2/3 * 1/2 = 1/5 P = 4/5 * 3/4 * 2/3 * 1/2 = 1/5
- 72. COSCUP2018 x openSUSE.Asia GNOME.Asia Howdowechoosewhichonetouse? K8S createa custom chain for each endpoints. First rule is a SNAT ○ The Ngnix want to access outside. Second is DNAT ○ Change the IP to one of the endpoints
- 73. COSCUP2018 x openSUSE.Asia GNOME.Asia Now, WeKnow The Basic Function Of Kubernetes Network.
- 74. COSCUP2018 x openSUSE.Asia GNOME.Asia What isthe next step of kubernetes network
- 75. COSCUP2018 x openSUSE.Asia GNOME.Asia challenges For differentuse cases ○ 5G/NFV/IoT Network features ○ High performance ○ Low latency Network infrastructure ○ Multiple network ■ Separate the data/control network.
- 76. COSCUP2018 x openSUSE.Asia GNOME.Asia Networkfeaturesc We usethe hardware/smart NIC for those requirements before. We also have some mechanism in the software. ○ DPDK ○ SR-IOV ○ … ○ etc
- 77. COSCUP2018 x openSUSE.Asia GNOME.Asia Network InterfaceCard Linux Kernel Network Stack Network Driver Application Network Interface Card Linux Kernel Network Stack Network Driver Application Kernel Space User Space DPDK
- 78. COSCUP2018 x openSUSE.Asia GNOME.Asia How integratethose with kubernetes?
- 79. COSCUP2018 x openSUSE.Asia GNOME.Asia CNI We usesome CNI for those functions. Intel had developed the CNI for those functions. ○ Call sriov-cni ○ https://github.com/intel/sriov-cni
- 80. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16)
- 81. COSCUP2018 x openSUSE.Asia GNOME.Asia Problem For thosecontainer using the DPDK/SR-IOV, it can’t use any kubernetes service now. Since the network function is handled by DPDK/SR-IOV now. How to solve this?
- 82. COSCUP2018 x openSUSE.Asia GNOME.Asia Multus There’s adiscussion in the github about that requirement. Intel develop a CNI plugin to support multiple network for a Pod. ○ It’s called Multus CNI Multus call CNIs one by one.
- 83.
- 84. COSCUP2018 x openSUSE.Asia GNOME.Asia , , You needto create first
- 85. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)
- 86. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)
- 87. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)
- 88. COSCUP2018 x openSUSE.Asia GNOME.Asia Node1 Node2Node3 PodA PodA PodA PodB PodA PodB Flannel (Control Network) br0 br0 br0 Data Network (192.168.0.0/16) Data Network (10.56.10/24)
- 89.