Configure exclusive ENI mode for node pools - Container Service for Kubernetes

The exclusive Elastic Network Interface (ENI) mode is a strategy that provides optimal network performance for pods. This mode is ideal for scenarios with strict network performance requirements. For example, in data analytics, real-time stream processing, or applications that are sensitive to network latency, such as video streaming, online gaming, or scientific computing, this network mode provides high network throughput and very low network latency. In high-frequency trading scenarios, this network mode provides multicast capabilities.

Limits

Terway v1.11.0 or later is required to add ECS instances. Terway v1.14.3 or later is required to add Node Lingjun instances. To upgrade the component, see Terway.
If you enable dual-stack for a cluster, adding nodes is subject to the ECS instance type limits of the shared ENI mode. The number of IPv4 addresses must equal the number of IPv6 addresses. This applies to both primary and secondary private addresses. For more information about the number of IPv4 and IPv6 addresses that an ECS instance supports, see Instance families.
IPv6 dual-stack is not supported for Node Lingjun instances.
Pods that use exclusive ENIs do not support eBPF network acceleration or NetworkPolicy.
You must use new nodes for exclusive ENIs. Existing ENIs on reused nodes are not used.

Configure the container network at the node pool level

The exclusive ENI mode is a node pool mode that Terway provides. For a detailed comparison with the shared ENI mode, see Shared ENI mode vs. Exclusive ENI mode. You can plan and create a node pool for exclusive ENIs. After the node pool is created, you can schedule pods to it.

1. Plan the exclusive ENI node pool

In the exclusive ENI mode, a single node supports a smaller number of pods. A worker node must have more than six ENIs to join the cluster. For more information about how to calculate the number of ENIs, see Instance families.
Plan the vSwitches and security groups for pods.

Terway supports multiple configuration methods, listed below in descending order of priority:

Configure a static IP address, a separate virtual switch, and a separate security group for a pod
Node-level network configuration
Cluster default configurations: Customize Terway configuration parameters
Important
Ensure that the vSwitches that correspond to the node zones are included in the preceding configurations. Otherwise, pods cannot be created.
Node Lingjun node pools do not support configuring a static IP address, a separate virtual switch, and a separate security group for a pod.

2. Create an exclusive ENI node pool and verify that the mode is enabled

Create a new node pool and add the label k8s.aliyun.com/exclusive-mode-eni-type: eniOnly to the nodes during creation. For more information, see Create and manage a node pool.
Configure a taint to prevent other pods from being scheduled to the exclusive ENI node pool.
Important
You must configure the label when you create the node pool. You cannot switch existing nodes to the exclusive ENI mode. If you configure the label incorrectly, you must delete the node pool and recreate it.

Run the following command to query the allocatable resources on a node and check whether the exclusive ENI mode is enabled.

kubectl describe node <node-name>

Expected output:

Capacity: aliyun/eni: 7 cpu: 16 ephemeral-storage: 123460788Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 31555380Ki pods: 213 Allocatable: aliyun/eni: 7 cpu: 15890m ephemeral-storage: 113781462033 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 28587828Ki pods: 213

If aliyun/eni is displayed in the output, the exclusive ENI mode is enabled.

3. Schedule pods to the exclusive ENI node pool

You can use NodeAffinity or a PodNetworking CustomResourceDefinition (CRD) to schedule pods to the exclusive ENI node pool.

NodeAffinity: Does not support pod-level configurations, such as using a static IP, a separate virtual switch, or a separate security group.
PodNetworking: Supports pod-level configuration of vSwitches, security groups, and static pod IPs. For more information, see Configure a static IP address, a separate virtual switch, and a separate security group for a pod.
In the following YAML example, set the eniType key in the eniOptions field to ENI to schedule pods to an exclusive ENI node pool.
```
apiVersion: network.alibabacloud.com/v1beta1 kind: PodNetworking metadata: name: enionly spec: eniOptions: eniType: ENI allocationType: type: Elastic selector: podSelector: matchLabels: network: enionly
```

FAQ

How do I know if a pod uses an exclusive ENI?

Terway creates a PodENI resource with the same name and in the same namespace as the pod. This resource records the network configuration information.

You can query the resource as follows:

kubectl get podeni nginx-9d557694f-rcdzs -oyaml

Expected output:

apiVersion: network.alibabacloud.com/v1beta1 kind: PodENI metadata: annotations: k8s.aliyun.com/pod-uid: 05590939-fc51-47ab-a204-3dd187233bca creationTimestamp: "2024-09-13T08:09:27Z" finalizers: - pod-eni generation: 1 labels: k8s.aliyun.com/node: cn-hangzhou.172.XX.XX.25 name: example-9d557694f-rcdzs namespace: default resourceVersion: "1131123" spec: allocations: - allocationType: type: Elastic eni: attachmentOptions: {} id: eni-xxxx mac: 00:16:3e:37:xx:xx securityGroupIDs: - sg-xxxx vSwitchID: vsw-xxxx zone: cn-hangzhou-j ipv4: 172.16.0.30 ipv4CIDR: 172.16.0.0/24 ipv6: 2408:4005:xxxx:xxxx:xxxx:xxxx:xxxx:9ad4 ipv6CIDR: 2408:4005:39c:xxxx::/64 zone: cn-hangzhou-j status: eniInfos: eni-xxxx: id: eni-xxxx status: Bind type: Secondary instanceID: i-xxxx phase: Bind