fix: let Windows CNS use the InClusterConfig #3248
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
rbtr requested review from Copilot, jpayne3506, matmerr, nddq, pjohnst5 and thatmattlong 
thatmattlong reviewed Dec 6, 2024
thatmattlong reviewed Dec 6, 2024
thatmattlong previously requested changes Dec 6, 2024
thatmattlong reviewed Dec 6, 2024
| /azp run Azure Container Networking PR |
| Azure Pipelines successfully started running 1 pipeline(s). |
jpayne3506 previously approved these changes Dec 10, 2024
7 tasks
github-merge-queue bot pushed a commit to microsoft/retina that referenced this pull request Dec 12, 2024
# Description As of [AKS 1.30](https://github.com/Azure/AKS/releases/tag/2024-06-09), service account tokens refresh every ~1 hour when OIDC is enabled. They were previously valid for a year. [This setkubeconfigpath.ps1 script](https://github.com/Azure/azure-container-networking/blob/47b243c42fd16119a96ab6d06eb602ac2ce40e7d/npm/examples/windows/setkubeconfigpath.ps1) was at some point necessary on Windows to create a valid kubeconfig for Retina WIndows. It copies the token from the token file to create a custom kubeconfig from a template at startup and then we pass Retina Windows that file via --kubeconfig. The script runs at startup and never re-runs, so the token that exists at Pod start is the token CNS will try to use forever. Yearly token lifespans were long enough that no Retina Windows Pod was ever up long enough to hit token expiration. This becomes an issue with hourly token lifespans. An hour after Pod start, the token becomes invalid and Retina Windows can no longer auth to the API server. For PodSubnet clusters, this permanently prevents Retina Windows from being able to scale the IPAM pool and provide more Pod IPs. Fix for CNS which referenced: Azure/azure-container-networking#3248 ## Related Issue If this pull request is related to any issue, please mention it here. Additionally, make sure that the issue is assigned to you before submitting this pull request. ## Checklist - [x] I have read the [contributing documentation](https://retina.sh/docs/contributing). - [x] I signed and signed-off the commits (`git commit -S -s ...`). See [this documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification) on signing commits. - [x] I have correctly attributed the author(s) of the code. - [x] I have tested the changes locally. - [x] I have followed the project's style guidelines. - [ ] I have updated the documentation, if necessary. - [ ] I have added tests, if applicable. ## Screenshots (if applicable) or Testing Completed Please add any relevant screenshots or GIFs to showcase the changes made. ## Additional Notes Add any additional notes or context about the pull request here. --- Please refer to the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more information on how to contribute to this project.
github-merge-queue bot pushed a commit to microsoft/retina that referenced this pull request Dec 12, 2024
# Description As of [AKS 1.30](https://github.com/Azure/AKS/releases/tag/2024-06-09), service account tokens refresh every ~1 hour when OIDC is enabled. They were previously valid for a year. [This setkubeconfigpath.ps1 script](https://github.com/Azure/azure-container-networking/blob/47b243c42fd16119a96ab6d06eb602ac2ce40e7d/npm/examples/windows/setkubeconfigpath.ps1) was at some point necessary on Windows to create a valid kubeconfig for Retina WIndows. It copies the token from the token file to create a custom kubeconfig from a template at startup and then we pass Retina Windows that file via --kubeconfig. The script runs at startup and never re-runs, so the token that exists at Pod start is the token CNS will try to use forever. Yearly token lifespans were long enough that no Retina Windows Pod was ever up long enough to hit token expiration. This becomes an issue with hourly token lifespans. An hour after Pod start, the token becomes invalid and Retina Windows can no longer auth to the API server. For PodSubnet clusters, this permanently prevents Retina Windows from being able to scale the IPAM pool and provide more Pod IPs. Fix for CNS which referenced: Azure/azure-container-networking#3248 ## Related Issue If this pull request is related to any issue, please mention it here. Additionally, make sure that the issue is assigned to you before submitting this pull request. ## Checklist - [x] I have read the [contributing documentation](https://retina.sh/docs/contributing). - [x] I signed and signed-off the commits (`git commit -S -s ...`). See [this documentation](https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification) on signing commits. - [x] I have correctly attributed the author(s) of the code. - [x] I have tested the changes locally. - [x] I have followed the project's style guidelines. - [ ] I have updated the documentation, if necessary. - [ ] I have added tests, if applicable. ## Screenshots (if applicable) or Testing Completed Please add any relevant screenshots or GIFs to showcase the changes made. ## Additional Notes Add any additional notes or context about the pull request here. --- Please refer to the [CONTRIBUTING.md](../CONTRIBUTING.md) file for more information on how to contribute to this project.
rbtr force-pushed the fix/windows-tokens branch from 765c550 to c5b0288 Compare | /azp run Azure Container Networking PR |
| Azure Pipelines successfully started running 1 pipeline(s). |
| This pull request is stale because it has been open for 2 weeks with no activity. Remove stale label or comment or this will be closed in 7 days |
rbtr force-pushed the fix/windows-tokens branch from c5b0288 to 318ab1c Compare | This pull request is stale because it has been open for 2 weeks with no activity. Remove stale label or comment or this will be closed in 7 days |
paulyufan2 previously approved these changes Jun 13, 2025
Signed-off-by: Evan Baker <rbtr@users.noreply.github.com>
paulyufan2 previously approved these changes Jun 13, 2025
| /azp run Azure Container Networking PR |
| Azure Pipelines successfully started running 1 pipeline(s). |
jpayne3506 requested changes Jun 13, 2025
Signed-off-by: Evan Baker <rbtr@users.noreply.github.com>
jpayne3506 approved these changes Jun 13, 2025
paulyufan2 approved these changes Jun 13, 2025
| /azp run Azure Container Networking PR |
| Azure Pipelines successfully started running 1 pipeline(s). |
jpayne3506 added release/latest sivakami-projects pushed a commit that referenced this pull request Oct 23, 2025
* fix: let Windows CNS use the InClusterConfig Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> * remove pwsh from cmd Signed-off-by: Evan Baker <rbtr@users.noreply.github.com> --------- Signed-off-by: Evan Baker <rbtr@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Reason for Change:
As of AKS 1.30, service account tokens refresh every ~1 hour when OIDC is enabled. They were previously valid for a year.
This
setkubeconfigpath.ps1script was at some point necessary on Windows to create a valid kubeconfig for CNS (and NPM and others), but based on my testing we don't need it today. It copies the token from the token file to create a custom kubeconfig from a template at startup and then we pass CNS that file via--kubeconfig.The script runs at startup and never re-runs, so the token that exists at Pod start is the token CNS will try to use forever. Yearly token lifespans were long enough that no CNS Pod was ever up long enough to hit token expiration.
This becomes an issue with hourly token lifespans. An hour after Pod start, the token becomes invalid and CNS can no longer auth to the API server. For PodSubnet clusters, this permanently prevents CNS from being able to scale the IPAM pool and provide more Pod IPs.
Issue Fixed:
Azure/AKS#4679
Requirements:
Notes: