[bug fix] handle rules exceeded error in listener rule synthesizer #4393
Conversation
k8s-ci-robot added the cncf-cla: yes k8s-ci-robot added approved 
| | ||
| s.logger.Info("unmatchedResLRs size", "size", len(unmatchedResLRs)) | ||
| for _, resLR := range unmatchedResLRs { | ||
| s.logger.Info("Unmatched", "res lr", *resLR) |
There was a problem hiding this comment.
Lets add these to debug logs.
k8s-ci-robot added the needs-rebase 
| @zac-nixon: The following test failed, say
Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
k8s-ci-robot added the lgtm 
k8s-ci-robot removed lgtm 
| /lgtm |
k8s-ci-robot added the lgtm | [APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: shraddhabang, zac-nixon The full list of commands accepted by this bot can be found here. The pull request process is described here Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| /lgtm |
4 participants
Issue
#4373
Description
Fixes situation described in #4373. When the listener rule synthesizer moves rules into the last priorities (50,000 - 49900) and the listener is at all the rule limit, then the synthesizer gets stuck. This is because we always attempt to create rules prior to deleting, in order to prevent requests from not getting 404 errors due to missing listener rules. At first, I thought that the modify rule path should handle this, but closer inspection told me that we only call modify rule if the priority of the new rule matched the priority of a rule that we were deleting. This means that the only time you can successfully replace a rule is when you're at the maximum rules for a listener is to modify the rule at the exact priority (and perform no other modifications!).
The solution I came up with is to create a greedy algorithm that attempts to create all desired rules, up until we detect the listener is it's max rule limit. At this point of time, we start flip-flopping between delete and create in order to minimize time between rules missing on the listener. Finally, once all missing rules are created, we will delete all remaining undesired rules.
Some other solutions that I played around with..
1/ Always call modify rule. This meant determining which unmatched SDK and result LRs to map together to modify. The downside to this approach is that we can't guarantee that the customer provided priority ordering would match on the listener. This is because the Modify api doesn't allow priority modification. I was afraid of misrouting scenarios because of corrupted priority ordering.
2/ Something along the lines of again trying to map unmatched sdk and result LRs and perform the create / delete in that order. It was pretty hard to come up with a correct algorithm to do this matching..
Checklist
README.md, or thedocsdirectory)BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯