Merge pull request openshift#42 from bmeng/ovn-kube

add runbook for NodeWithoutOVNKubeNodePodRunning and V4SubnetAllocati…

Author: openshift-merge-robot

alerts/cluster-network-operator/NodeWithoutOVNKubeNodePodRunning.md

@@ -0,0 +1,48 @@
+# NodeWithoutOVNKubeNodePodRunning
+
+## Meaning
+
+The `NodeWithoutOVNKubeNodePodRunning` alert is triggered when one or more Linux
+nodes do not have a running OVNkube-node pod for a period of time.
+
+## Impact
+
+This is a warning alert. Existing workloads on the node may continue to have
+connectivity but any additional workloads will not be provisioned on the node.
+Any network policy changes will not be implemented on existing workloads on the
+node.
+
+## Diagnosis
+
+Check the nodes which should have the ovnkube-node running.
+
+ oc get node -l kubernetes.io/os!=windows
+
+Check the expected running replicas of ovnkube-node.
+
+ oc get daemonset ovnkube-node -n openshift-ovn-kubernetes
+
+Check the ovnkube-node pods status on the nodes.
+
+ oc get po -n openshift-ovn-kubernetes -l app=ovnkube-node -o wide
+
+Describe the pod if there is non-running ovnkube-node pod.
+
+ oc describe po -n openshift-ovn-kubernetes <ovnkube-node-name>
+
+Check the pod logs for the failing ovnkube-node pods
+
+ oc logs <ovnkube-node-name> -n openshift-ovn-kubernetes --all-containers
+
+## Mitigation
+
+Mitigation for this alert is not possible to understand in advance.
+
+If you are seeing that any of the ovnkube-node pods is not in Running status,
+you can try to delete the pod and let it being recreated by the daemonset
+controller.
+
+ oc delete po <ovnkube-node> -n openshift-ovn-kubernetes
+
+If the issue cannot be fixed by recreating the pod, reboot of the affected node
+might be an option to refresh the full stack (include OVS on the node).

alerts/cluster-network-operator/V4SubnetAllocationThresholdExceeded.md

@@ -0,0 +1,41 @@
+# V4SubnetAllocationThresholdExceeded
+
+## Meaning
+
+The `V4SubnetAllocationThresholdExceeded` alert is triggered when more than
+80% of subnets for nodes are allocated.
+
+## Impact
+
+This is a warning alert. No immediate impact to the cluster will be observed if
+this alert fires and it is a warning to be mindful of your remaining node
+subnet allocation. If your remaining subnets are exhausted, then no
+further nodes can be added to your cluster.
+
+## Diagnosis
+
+Check the network configuration on the cluster.
+
+ oc get networks.config.openshift.io/cluster -o jsonpath='{.spec.clusterNetwork}'
+
+ [{"cidr":"10.128.0.0/14","hostPrefix":23}]
+
+Calculate the IPv4 subnets capability.
+
+ subnet_capability = 2^[(32 - clusternetwork_netmask) - (32 - hostPrefix)]
+
+It will be 512 if the CIDR netmask is `/14` and hostPrefix is `23`, that means
+the cluster can have at most 512 nodes.
+
+Count the number of nodes to compare.
+
+ oc get node --no-headers | wc -l
+
+## Mitigation
+
+We do not support adding additional cluster networks for ovn-kuberntes.
+
+User will have to create a new cluster for more worker nodes.
+
+Choosing a larger cluster network CIDR which can hold more subnets could prevent
+this happening.