fix: [Linux] [NPM] handle #2977 for netpols without cidrs

[huntergregory]

Reason for Change:
#2977 actually could happen for any NetPol. Redoing #2978 to handle all NetPols. This fix forces the data plane to successfully apply IPSets before starting to add a policy if the previous RemovePolicy call failed to apply ipsets. This prevents the chance of NPM removing a policy, flushing all policy IPSets, but failing to delete policy IPSets, then NPM adding the policy back thinking that the policy IPSets are in the kernel with all members (when there are actually no members in the IPSets due to the flush).

Issue Fixed:
Fix #2977 completely

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• relevant PR labels added

Notes:

Logs for the Fix

Editing Policy

I0910 18:32:18.073907 1 dataplane.go:651] [DataPlane] Update Policy called for default/repro I0910 18:32:18.074560 1 dataplane.go:575] [DataPlane] Remove Policy called for default/repro I0910 18:32:18.075259 1 chain-management_linux.go:366] Executing iptables command with args [-w 60 -D AZURE-NPM-INGRESS -j AZURE-NPM-INGRESS-1739478793 -m set --match-set azure-npm-3888852483 dst -m set --match-set azure-npm-784554818 dst -m comment --comment INGRESS-POLICY-default/repro-TO-podlabel-pod:c-AND-ns-default-IN-ns-default] I0910 18:32:18.078654 1 chain-management_linux.go:366] Executing iptables command with args [-w 60 -D AZURE-NPM-EGRESS -j AZURE-NPM-EGRESS-1739478793 -m set --match-set azure-npm-3888852483 src -m set --match-set azure-npm-784554818 src -m comment --comment EGRESS-POLICY-default/repro-FROM-podlabel-pod:c-AND-ns-default-IN-ns-default] I0910 18:32:18.107254 1 restore.go:188] running this restore command: [iptables-nft-restore -w 60 -T filter --noflush] I0910 18:32:18.110645 1 dataplane.go:340] [DataPlane] [ApplyDataPlane] [DEL-NETPOL] starting to apply ipsets I0910 18:32:18.110706 1 ipsetmanager.go:460] [IPSetManager] dirty caches. toAddUpdateCache: to create: [], to update: [], toDeleteCache: map[nslabel-namespace:oed999999999999999:0xc0009a5a20 nslabel-namespace:oedasdf:0xc0009a5c20 nslabel-namespace:oedff:0xc0009a5c60 nslabel-namespace:oedrrrrrrrrrrrrr:0xc0009a5a60 nslabel-namespace:omncc:0xc0009a5ce0 nslabel-namespace:omnllllllllllll:0xc0009a5ae0 nslabel-namespace:platform:0xc0009a5ca0 nslabel-namespace:platform2:0xc0009a5b60 nslabel-namespace:platform222222:0xc0009a59a0 nslabel-namespace:platform2999:0xc0009a5960 nslabel-namespace:platform34:0xc0009a5ba0 nslabel-namespace:platform6534:0xc0009a5be0 nslabel-namespace:platform777777777:0xc0009a59e0 nslabel-namespace:platformwwwwwwwwwwwwwwww:0xc0009a5aa0 nslabel-namespace:ppsaa:0xc0009a5d20 nslabel-namespace:ppsvvvvvvvvvvvvvvv:0xc0009a5b20 podlabel-app.kubernetes.io/instance:datadog:0xc0009a5b40 podlabel-app.kubernetes.io/instance:datadog1111:0xc0009a5940 podlabel-app.kubernetes.io/instance:grafana-agent:0xc0009a5b80 podlabel-app.kubernetes.io/instance:grafana-agent444444444444:0xc0009a5980 podlabel-app.kubernetes.io/name:ingress-nginx:0xc0009a5bc0 podlabel-app.kubernetes.io/name:ingress-nginx00:0xc0009a5c80 podlabel-app.kubernetes.io/name:ingress-nginx88888888888888888:0xc0009a59c0 podlabel-app.kubernetes.io/name:ingress-nginxooooooooooooooooo:0xc0009a5a80 podlabel-base-app:ph-oed-distribution-adapter-api-v1000000000000000000000:0xc0009a5a00 podlabel-base-app:ph-oed-distribution-adapter-api-v133:0xc0009a5c00 podlabel-base-app:ph-oed-distribution-consumer-v122:0xc0009a5c40 podlabel-base-app:ph-oed-distribution-consumer-v1tttttttttt:0xc0009a5a40 podlabel-base-app:ph-omn-econnect-adapter-publisher-v1dd:0xc0009a5cc0 podlabel-base-app:ph-omn-econnect-adapter-publisher-v1xxxxxxxxxxxxx:0xc0009a5ac0 podlabel-base-app:ph-pps-sorting-servee:0xc0009a5d00 podlabel-base-app:ph-pps-sorting-serverbbbbbbbbbbbbbbbb:0xc0009a5b00 podlabel-pod:c:0xc0009a5d60] I0910 18:32:18.111370 1 restore.go:188] running this restore command: [ipset restore] I0910 18:32:18.113795 1 restore.go:299] continuing after line 34 for command [ipset restore] 2024/09/10 18:32:18 [1] skipping destroy line for set podlabel-base-app:ph-omn-econnect-adapter-publisher-v1xxxxxxxxxxxxx since the set is in use by a kernel component 2024/09/10 18:32:18 [1] error: on try number 1, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-3707620128]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 34: Set cannot be destroyed: it is in use by a kernel component ]] I0910 18:32:18.114877 1 restore.go:188] running this restore command: [ipset restore] I0910 18:32:18.117300 1 restore.go:299] continuing after line 1 for command [ipset restore] 2024/09/10 18:32:18 [1] skipping destroy line for set podlabel-app.kubernetes.io/name:ingress-nginx88888888888888888 since the set is in use by a kernel component 2024/09/10 18:32:18 [1] error: on try number 2, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-3982009430]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ]] I0910 18:32:18.117901 1 restore.go:188] running this restore command: [ipset restore] 2024/09/10 18:32:18 [1] skipping destroy line for set podlabel-app.kubernetes.io/name:ingress-nginxooooooooooooooooo since the set is in use by a kernel component 2024/09/10 18:32:18 [1] error: on try number 3, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-892848623]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ]] I0910 18:32:18.119335 1 restore.go:299] continuing after line 1 for command [ipset restore] I0910 18:32:18.119511 1 restore.go:188] running this restore command: [ipset restore] 2024/09/10 18:32:18 [1] skipping destroy line for set nslabel-namespace:ppsaa since the set is in use by a kernel component 2024/09/10 18:32:18 [1] error: on try number 4, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-796518079]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ]] I0910 18:32:18.121284 1 restore.go:299] continuing after line 1 for command [ipset restore] I0910 18:32:18.121358 1 restore.go:188] running this restore command: [ipset restore] 2024/09/10 18:32:18 [1] error: failed to apply ipsets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ] I0910 18:32:18.123115 1 dataplane.go:640] [DataPlane] remove policy has failed to apply ipsets. setting remove policy failure 2024/09/10 18:32:18 [1] syncNetPol error due to error syncing 'default/repro': [syncNetPol] error due to [syncAddAndUpdateNetPol] Error: failed to update translated NPMNetworkPolicy into Dataplane due to [DataPlane] error while updating policy: [DataPlane] [DEL-NETPOL] error while applying IPSets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ], requeuing E0910 18:32:18.123209 1 networkPolicyController.go:195] error syncing 'default/repro': [syncNetPol] error due to [syncAddAndUpdateNetPol] Error: failed to update translated NPMNetworkPolicy into Dataplane due to [DataPlane] error while updating policy: [DataPlane] [DEL-NETPOL] error while applying IPSets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ], requeuing I0910 18:32:18.128598 1 dataplane.go:651] [DataPlane] Update Policy called for default/repro I0910 18:32:18.128610 1 dataplane.go:654] [DataPlane] Policy default/repro is not found. I0910 18:32:18.128615 1 dataplane.go:407] [DataPlane] Add Policy called for default/repro I0910 18:32:18.128619 1 types.go:214] [DataPlane] enqueuing policy default/repro in netPolQueue I0910 18:32:18.128624 1 dataplane.go:421] [DataPlane] [ADD-NETPOL] new pending netpol count: 1 I0910 18:32:18.128652 1 networkPolicyController.go:191] Successfully synced 'default/repro' I0910 18:32:18.289086 1 dataplane.go:434] [DataPlane] adding policies [0xc00014dc30] I0910 18:32:18.289113 1 dataplane.go:340] [DataPlane] [ApplyDataPlane] [ADD-NETPOL-PRECAUTION] starting to apply ipsets I0910 18:32:18.289169 1 ipsetmanager.go:460] [IPSetManager] dirty caches. toAddUpdateCache: to create: [], to update: [], toDeleteCache: map[nslabel-namespace:oed999999999999999:0xc0009a5a20 nslabel-namespace:oedasdf:0xc0009a5c20 nslabel-namespace:oedff:0xc0009a5c60 nslabel-namespace:oedrrrrrrrrrrrrr:0xc0009a5a60 nslabel-namespace:omncc:0xc0009a5ce0 nslabel-namespace:omnllllllllllll:0xc0009a5ae0 nslabel-namespace:platform:0xc0009a5ca0 nslabel-namespace:platform2:0xc0009a5b60 nslabel-namespace:platform222222:0xc0009a59a0 nslabel-namespace:platform2999:0xc0009a5960 nslabel-namespace:platform34:0xc0009a5ba0 nslabel-namespace:platform6534:0xc0009a5be0 nslabel-namespace:platform777777777:0xc0009a59e0 nslabel-namespace:platformwwwwwwwwwwwwwwww:0xc0009a5aa0 nslabel-namespace:ppsaa:0xc0009a5d20 nslabel-namespace:ppsvvvvvvvvvvvvvvv:0xc0009a5b20 podlabel-app.kubernetes.io/instance:datadog:0xc0009a5b40 podlabel-app.kubernetes.io/instance:datadog1111:0xc0009a5940 podlabel-app.kubernetes.io/instance:grafana-agent:0xc0009a5b80 podlabel-app.kubernetes.io/instance:grafana-agent444444444444:0xc0009a5980 podlabel-app.kubernetes.io/name:ingress-nginx:0xc0009a5bc0 podlabel-app.kubernetes.io/name:ingress-nginx00:0xc0009a5c80 podlabel-app.kubernetes.io/name:ingress-nginx88888888888888888:0xc0009a59c0 podlabel-app.kubernetes.io/name:ingress-nginxooooooooooooooooo:0xc0009a5a80 podlabel-base-app:ph-oed-distribution-adapter-api-v1000000000000000000000:0xc0009a5a00 podlabel-base-app:ph-oed-distribution-adapter-api-v133:0xc0009a5c00 podlabel-base-app:ph-oed-distribution-consumer-v122:0xc0009a5c40 podlabel-base-app:ph-oed-distribution-consumer-v1tttttttttt:0xc0009a5a40 podlabel-base-app:ph-omn-econnect-adapter-publisher-v1dd:0xc0009a5cc0 podlabel-base-app:ph-omn-econnect-adapter-publisher-v1xxxxxxxxxxxxx:0xc0009a5ac0 podlabel-base-app:ph-pps-sorting-servee:0xc0009a5d00 podlabel-base-app:ph-pps-sorting-serverbbbbbbbbbbbbbbbb:0xc0009a5b00 podlabel-pod:c:0xc0009a5d60] I0910 18:32:18.289322 1 restore.go:188] running this restore command: [ipset restore] I0910 18:32:18.555426 1 dataplane.go:345] [DataPlane] [ApplyDataPlane] [ADD-NETPOL-PRECAUTION] finished applying ipsets I0910 18:32:18.555545 1 dataplane.go:340] [DataPlane] [ApplyDataPlane] [ADD-NETPOL] starting to apply ipsets I0910 18:32:18.555596 1 ipsetmanager.go:460] [IPSetManager] dirty caches. toAddUpdateCache: to create: [nslabel-namespace:omncc: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-pps-sorting-servee: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform7777777777777: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform6534: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:ppsvvvvvvvvvvvvvvv: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/instance:datadog: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform34: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-omn-econnect-adapter-publisher-v1dd: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-oed-distribution-adapter-api-v1000000000000000000000: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:omnllllllllllll: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:oedff: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:ppsaa: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/name:ingress-nginx: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-oed-distribution-adapter-api-v133: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:oed999999999999999: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/name:ingress-nginxooooooooooooooooo: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:oedasdf: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-oed-distribution-consumer-v122: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform2999: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/name:ingress-nginx88888888888888888: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/name:ingress-nginx00: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform777777777: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-oed-distribution-consumer-v1tttttttttt: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platformwwwwwwwwwwwwwwww: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-omn-econnect-adapter-publisher-v1xxxxxxxxxxxxx: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform: &{membersToAdd:map[] membersToDelete:map[]},podlabel-pod:c: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/instance:grafana-agent444444444444: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:oedrrrrrrrrrrrrr: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/instance:datadog1111: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform222222: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-pps-sorting-serverbbbbbbbbbbbbbbbb: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/instance:grafana-agent: &{membersToAdd:map[] membersToDelete:map[]}], to update: [], toDeleteCache: map[] I0910 18:32:18.555698 1 restore.go:188] running this restore command: [ipset restore] I0910 18:32:18.557605 1 dataplane.go:345] [DataPlane] [ApplyDataPlane] [ADD-NETPOL] finished applying ipsets I0910 18:32:18.557784 1 restore.go:188] running this restore command: [iptables-nft-restore -w 60 -T filter --noflush] I0910 18:32:18.563097 1 dataplane.go:439] [DataPlane] [BACKGROUND] added policies successfully 

Removing then Readding Policy

I0905 17:07:36.014721 1 dataplane.go:575] [DataPlane] Remove Policy called for default/repro I0905 17:07:36.014768 1 chain-management_linux.go:366] Executing iptables command with args [-w 60 -D AZURE-NPM-INGRESS -j AZURE-NPM-INGRESS-1739478793 -m set --match-set azure-npm-3888852483 dst -m set --match-set azure-npm-784554818 dst -m comment --comment INGRESS-POLICY-default/repro-TO-podlabel-pod:c-AND-ns-default-IN-ns-default] I0905 17:07:36.017573 1 chain-management_linux.go:366] Executing iptables command with args [-w 60 -D AZURE-NPM-EGRESS -j AZURE-NPM-EGRESS-1739478793 -m set --match-set azure-npm-3888852483 src -m set --match-set azure-npm-784554818 src -m comment --comment EGRESS-POLICY-default/repro-FROM-podlabel-pod:c-AND-ns-default-IN-ns-default] I0905 17:07:36.044933 1 restore.go:188] running this restore command: [iptables-nft-restore -w 60 -T filter --noflush] I0905 17:07:36.046489 1 dataplane.go:340] [DataPlane] [ApplyDataPlane] [DEL-NETPOL] starting to apply ipsets I0905 17:07:36.046532 1 ipsetmanager.go:460] [IPSetManager] dirty caches. toAddUpdateCache: to create: [], to update: [], toDeleteCache: map[nslabel-namespace:oed999999999999999:0xc000517a50 nslabel-namespace:oedasdf:0xc000517c50 nslabel-namespace:oedff:0xc000517c90 nslabel-namespace:oedrrrrrrrrrrrrr:0xc000517a90 nslabel-namespace:omncc:0xc000517d10 nslabel-namespace:omnllllllllllll:0xc000517b10 nslabel-namespace:platform:0xc000517cd0 nslabel-namespace:platform2:0xc000517b90 nslabel-namespace:platform222222:0xc0005179d0 nslabel-namespace:platform2999:0xc000517990 nslabel-namespace:platform34:0xc000517bd0 nslabel-namespace:platform6534:0xc000517c10 nslabel-namespace:platform777777777:0xc000517a10 nslabel-namespace:platformwwwwwwwwwwwwwwww:0xc000517ad0 nslabel-namespace:ppsaa:0xc000517d50 nslabel-namespace:ppsvvvvvvvvvvvvvvv:0xc000517b50 podlabel-app.kubernetes.io/instance:datadog:0xc000517b70 podlabel-app.kubernetes.io/instance:datadog1111:0xc000517970 podlabel-app.kubernetes.io/instance:grafana-agent:0xc000517bb0 podlabel-app.kubernetes.io/instance:grafana-agent444444444444:0xc0005179b0 podlabel-app.kubernetes.io/name:ingress-nginx:0xc000517bf0 podlabel-app.kubernetes.io/name:ingress-nginx00:0xc000517cb0 podlabel-app.kubernetes.io/name:ingress-nginx88888888888888888:0xc0005179f0 podlabel-app.kubernetes.io/name:ingress-nginxooooooooooooooooo:0xc000517ab0 podlabel-base-app:ph-oed-distribution-adapter-api-v1000000000000000000000:0xc000517a30 podlabel-base-app:ph-oed-distribution-adapter-api-v133:0xc000517c30 podlabel-base-app:ph-oed-distribution-consumer-v122:0xc000517c70 podlabel-base-app:ph-oed-distribution-consumer-v1tttttttttt:0xc000517a70 podlabel-base-app:ph-omn-econnect-adapter-publisher-v1dd:0xc000517cf0 podlabel-base-app:ph-omn-econnect-adapter-publisher-v1xxxxxxxxxxxxx:0xc000517af0 podlabel-base-app:ph-pps-sorting-servee:0xc000517d30 podlabel-base-app:ph-pps-sorting-serverbbbbbbbbbbbbbbbb:0xc000517b30 podlabel-pod:c:0xc000517d90] I0905 17:07:36.046668 1 restore.go:188] running this restore command: [ipset restore] I0905 17:07:36.048002 1 restore.go:299] continuing after line 34 for command [ipset restore] I0905 17:07:36.048105 1 restore.go:188] running this restore command: [ipset restore] 2024/09/05 17:07:36 [1] skipping destroy line for set podlabel-app.kubernetes.io/instance:grafana-agent444444444444 since the set is in use by a kernel component 2024/09/05 17:07:36 [1] error: on try number 1, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-1078088152]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 34: Set cannot be destroyed: it is in use by a kernel component ]] I0905 17:07:36.049213 1 restore.go:299] continuing after line 1 for command [ipset restore] I0905 17:07:36.049284 1 restore.go:188] running this restore command: [ipset restore] 2024/09/05 17:07:36 [1] skipping destroy line for set nslabel-namespace:oedrrrrrrrrrrrrr since the set is in use by a kernel component 2024/09/05 17:07:36 [1] error: on try number 2, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-4062452990]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ]] I0905 17:07:36.056466 1 restore.go:299] continuing after line 1 for command [ipset restore] I0905 17:07:36.056548 1 restore.go:188] running this restore command: [ipset restore] 2024/09/05 17:07:36 [1] skipping destroy line for set nslabel-namespace:platform6534 since the set is in use by a kernel component 2024/09/05 17:07:36 [1] error: on try number 3, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-3089587453]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ]] 2024/09/05 17:07:36 [1] skipping destroy line for set podlabel-base-app:ph-oed-distribution-consumer-v122 since the set is in use by a kernel component 2024/09/05 17:07:36 [1] error: on try number 4, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-2532171833]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ]] I0905 17:07:36.057622 1 restore.go:299] continuing after line 1 for command [ipset restore] I0905 17:07:36.057689 1 restore.go:188] running this restore command: [ipset restore] 2024/09/05 17:07:36 [1] error: failed to apply ipsets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ] 2024/09/05 17:07:36 [1] syncNetPol error due to error syncing 'default/repro': [syncNetPol] error: [cleanUpNetworkPolicy] Error: failed to remove policy due to [DataPlane] [DEL-NETPOL] error while applying IPSets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ] when network policy is not found, requeuing I0905 17:07:36.059208 1 dataplane.go:640] [DataPlane] remove policy has failed to apply ipsets. setting remove policy failure E0905 17:07:36.059293 1 networkPolicyController.go:195] error syncing 'default/repro': [syncNetPol] error: [cleanUpNetworkPolicy] Error: failed to remove policy due to [DataPlane] [DEL-NETPOL] error while applying IPSets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component ] when network policy is not found, requeuing I0905 17:07:36.064614 1 networkPolicyController.go:225] Network Policy default/repro is not found, may be it is deleted I0905 17:07:36.064630 1 dataplane.go:575] [DataPlane] Remove Policy called for default/repro I0905 17:07:36.064636 1 dataplane.go:590] [DataPlane] Policy default/repro is not found. Might been deleted already I0905 17:07:36.064647 1 networkPolicyController.go:191] Successfully synced 'default/repro' I0905 17:07:44.557662 1 dataplane.go:651] [DataPlane] Update Policy called for default/repro I0905 17:07:44.557683 1 dataplane.go:654] [DataPlane] Policy default/repro is not found. I0905 17:07:44.557688 1 dataplane.go:407] [DataPlane] Add Policy called for default/repro I0905 17:07:44.557693 1 types.go:214] [DataPlane] enqueuing policy default/repro in netPolQueue I0905 17:07:44.557697 1 dataplane.go:421] [DataPlane] [ADD-NETPOL] new pending netpol count: 1 I0905 17:07:44.557710 1 networkPolicyController.go:191] Successfully synced 'default/repro' I0905 17:07:44.768041 1 dataplane.go:434] [DataPlane] adding policies [0xc0003b4580] I0905 17:07:44.768068 1 dataplane.go:340] [DataPlane] [ApplyDataPlane] [ADD-NETPOL-PRECAUTION] starting to apply ipsets I0905 17:07:44.768171 1 ipsetmanager.go:460] [IPSetManager] dirty caches. toAddUpdateCache: to create: [], to update: [], toDeleteCache: map[nslabel-namespace:oed999999999999999:0xc000517a50 nslabel-namespace:oedasdf:0xc000517c50 nslabel-namespace:oedff:0xc000517c90 nslabel-namespace:oedrrrrrrrrrrrrr:0xc000517a90 nslabel-namespace:omncc:0xc000517d10 nslabel-namespace:omnllllllllllll:0xc000517b10 nslabel-namespace:platform:0xc000517cd0 nslabel-namespace:platform2:0xc000517b90 nslabel-namespace:platform222222:0xc0005179d0 nslabel-namespace:platform2999:0xc000517990 nslabel-namespace:platform34:0xc000517bd0 nslabel-namespace:platform6534:0xc000517c10 nslabel-namespace:platform777777777:0xc000517a10 nslabel-namespace:platformwwwwwwwwwwwwwwww:0xc000517ad0 nslabel-namespace:ppsaa:0xc000517d50 nslabel-namespace:ppsvvvvvvvvvvvvvvv:0xc000517b50 podlabel-app.kubernetes.io/instance:datadog:0xc000517b70 podlabel-app.kubernetes.io/instance:datadog1111:0xc000517970 podlabel-app.kubernetes.io/instance:grafana-agent:0xc000517bb0 podlabel-app.kubernetes.io/instance:grafana-agent444444444444:0xc0005179b0 podlabel-app.kubernetes.io/name:ingress-nginx:0xc000517bf0 podlabel-app.kubernetes.io/name:ingress-nginx00:0xc000517cb0 podlabel-app.kubernetes.io/name:ingress-nginx88888888888888888:0xc0005179f0 podlabel-app.kubernetes.io/name:ingress-nginxooooooooooooooooo:0xc000517ab0 podlabel-base-app:ph-oed-distribution-adapter-api-v1000000000000000000000:0xc000517a30 podlabel-base-app:ph-oed-distribution-adapter-api-v133:0xc000517c30 podlabel-base-app:ph-oed-distribution-consumer-v122:0xc000517c70 podlabel-base-app:ph-oed-distribution-consumer-v1tttttttttt:0xc000517a70 podlabel-base-app:ph-omn-econnect-adapter-publisher-v1dd:0xc000517cf0 podlabel-base-app:ph-omn-econnect-adapter-publisher-v1xxxxxxxxxxxxx:0xc000517af0 podlabel-base-app:ph-pps-sorting-servee:0xc000517d30 podlabel-base-app:ph-pps-sorting-serverbbbbbbbbbbbbbbbb:0xc000517b30 podlabel-pod:c:0xc000517d90] I0905 17:07:44.768338 1 restore.go:188] running this restore command: [ipset restore] I0905 17:07:45.036952 1 dataplane.go:345] [DataPlane] [ApplyDataPlane] [ADD-NETPOL-PRECAUTION] finished applying ipsets I0905 17:07:45.037206 1 dataplane.go:340] [DataPlane] [ApplyDataPlane] [ADD-NETPOL] starting to apply ipsets I0905 17:07:45.037273 1 ipsetmanager.go:460] [IPSetManager] dirty caches. toAddUpdateCache: to create: [nslabel-namespace:platform: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/instance:datadog1111: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform777777777: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-oed-distribution-adapter-api-v1000000000000000000000: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:oed999999999999999: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/instance:datadog: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-oed-distribution-consumer-v122: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:oedff: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/instance:grafana-agent444444444444: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform6534: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/name:ingress-nginx00: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-oed-distribution-consumer-v1tttttttttt: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-omn-econnect-adapter-publisher-v1xxxxxxxxxxxxx: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-pps-sorting-serverbbbbbbbbbbbbbbbb: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/instance:grafana-agent: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:omncc: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:ppsaa: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/name:ingress-nginxooooooooooooooooo: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:oedrrrrrrrrrrrrr: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:omnllllllllllll: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform34: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-oed-distribution-adapter-api-v133: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-pps-sorting-servee: &{membersToAdd:map[] membersToDelete:map[]},podlabel-pod:c: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/name:ingress-nginx88888888888888888: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platformwwwwwwwwwwwwwwww: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform222222: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:ppsvvvvvvvvvvvvvvv: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform2: &{membersToAdd:map[] membersToDelete:map[]},podlabel-app.kubernetes.io/name:ingress-nginx: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:oedasdf: &{membersToAdd:map[] membersToDelete:map[]},podlabel-base-app:ph-omn-econnect-adapter-publisher-v1dd: &{membersToAdd:map[] membersToDelete:map[]},nslabel-namespace:platform2999: &{membersToAdd:map[] membersToDelete:map[]}], to update: [], toDeleteCache: map[] I0905 17:07:45.037385 1 restore.go:188] running this restore command: [ipset restore] I0905 17:07:45.039177 1 dataplane.go:345] [DataPlane] [ApplyDataPlane] [ADD-NETPOL] finished applying ipsets I0905 17:07:45.039958 1 restore.go:188] running this restore command: [iptables-nft-restore -w 60 -T filter --noflush] I0905 17:07:45.044190 1 dataplane.go:439] [DataPlane] [BACKGROUND] added policies successfully 

[huntergregory]

/azp run Azure Container Networking PR

[huntergregory]

/azp run NPM Conformance Tests

[huntergregory]

/azp run NPM Scale Test

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).