Recently, I was working on a migration from Azure Function Apps to Azure Kubernetes Service. While building a CI/CD pipeline, I decided to try applying a self-healing approach. The idea is that what if we can detect failures earlier and automatically recover from them?
(If you just need a full template, scroll to the end.)
How can problems be detected and reverted in Kubernetes deployment?
We use Deployment to manage workloads in Kubernetes. And the power of deployment that we can check status and rollback.
After applying, we can check the status of the deployment to detect problems.
If we just run this command:
kubectl rollout status deployment/some-deployment It will exit with 0 if deployment is ready. But if deployment is not ready and fails due to reasons, it will be stuck in this command and wait for deployment to complete. To escape this behaviour and exit with 1 if deployment is not ready on time we need to add a timeout argument:
kubectl rollout status deployment/some-deployment --timeout=1m This argument will timeout our deployment if it does not succeed within the timeframe.
Reverting changes.
To revert changes in Kubernetes we can use this command:
kubectl rollout undo deployment/some-deployment Before
After
Rollback Concept & Prerequisites.
The rollback mechanism in Kubernetes is designed to revert a deployment to a previous state if the current deployment is unhealthy or fails to meet the desired criteria.
In this case you should be aware that any changes that are not reversable can lead to a situation where rollback will not help.
Good example of those changes a schema change in database. For example during deployment you deploy schema changes and new code that supports it. But during deployment something went wrong and the new code has error and the app cannot start.
In this case if you rollback to a previous version the app will not work as intended because of the schema changes.
To successfully work with automatic rollbacks your team should always apply concept of Rolling Migrations.
Example Scenario.
Consider an application that requires a database schema update. The rolling migration process would involve the following steps:
- Deploy Code Supporting New Schema (Disabled): First, deploy the new application code that supports the new schema, but keep the new features disabled.
- Deploy Infrastructure Changes: Apply the database schema changes.
- Health Check: Introduce health checks to validate that the application can connect to the updated database schema.
- Enable New Features: Once the health checks pass, enable the new features in the application.
- Monitor Deployment: Continuously monitor the deployment to ensure everything is functioning correctly.
- Rollback if Necessary: If any issues are detected, rollback to the previous version of the application.
In this example the code is compatable with previous schema and even in the case of rollback the app will work as intended.
Better Deployment Status.
The deployment can get stuck and never completed because of the following:
- Insufficient quota
- Readiness probe failures
- Image pull errors
- Insufficient permissions
- Limit ranges
- Application runtime misconfiguration
Most of those problems are due to misconfigured deployments or clusters, but there is one that can be customised and can make our check even more robust. But Readiness probe failures allow us to extend those reasons to a more sophisticated one.
If we specify the URL for Readiness probe failures as our health check and extend it to validate more scenarios, like whether our application is able to connect external services like databases with the needed permissions, it will allow us to improve our Deployment Status Check.
Template Itself.
Azure DevOps yml pipeline template to validate AKS deployment and roll back if needed.
parameters: - name: environment type: string displayName: Required. Represents the stage of deployment. Usualy it is Development, Test, Acceptance, Production. It is needed so you can have multiple stages in the same pipeline. - name: azureServiceConnection displayName: Required. Azure service connection that will provide access to ARM. - name: aksName displayName: Required. Name of the AKS. - name: aksRg displayName: Required. Name of the resource group where AKS is. - name: aksSubscription displayName: Required. Azure subscription ID where AKS is. - name: namespace displayName: Required. Namespace of the k8s deployment to check. - name: deploymentName displayName: Required. Name of the k8s deployment to check. - name: timeout displayName: Optional. Timeout for the deployment check. The default is 1m. Example 1m, 2m, 10m default: 1m - name: dependsOn type: object displayName: Optional. Pass list of previous stages to depend on them. default: [] stages: - stage: AksDeploymentHealthAndRollback${{ parameters.environment}} displayName: ${{ parameters.environment}} — K8S Deployment Health and Rollback dependsOn: ${{ parameters.dependsOn }} jobs: - job: AksDeploymentHealthCheck displayName: ${{ parameters.environment}} — K8S Deployment Health steps: - checkout: self displayName: Checkout fetchTags: false - task: AzureCLI@2 displayName: K8S Deployment Health inputs: azureSubscription: ${{ parameters.azureServiceConnection}} scriptType: bash scriptLocation: inlineScript inlineScript: | set -e # fail when script is failing sudo az aks install-cli az aks get-credentials --name ${{ parameters.aksName }} \ --resource-group ${{ parameters.aksRg }} \ --subscription ${{ parameters.aksSubscription }} \ --overwrite-existing \ --file .kubeconfig-${{ parameters.aksName }} export KUBECONFIG=$(pwd)/.kubeconfig-${{ parameters.aksName }} # Set default namespace kubectl config set-context ${{ parameters.aksName }} --namespace=${{ parameters.namespace }} kubectl config get-contexts # Pass kubeconfig to kubelogin to access k8s API kubelogin convert-kubeconfig -l azurecli # Check the rollout status of the deployment kubectl rollout status deployment/${{ parameters.deploymentName }} --timeout=${{ parameters.timeout}} - job: ManualApprovalOfRollBack displayName: ${{ parameters.environment}} — Manual Approval Of Rollback dependsOn: AksDeploymentHealthCheck condition: failed() pool: server steps: - task: ManualValidation@0 displayName: Approve Rollback timeoutInMinutes: 1440 # task times out in 1 day - job: Rollback displayName: ${{ parameters.environment}} — K8S Deployment Rollback dependsOn: ManualApprovalOfRollBack condition: eq(dependencies.ManualApprovalOfRollBack.result, 'Succeeded') steps: - checkout: self displayName: Checkout fetchTags: false - task: AzureCLI@2 displayName: K8S Rollback inputs: azureSubscription: ${{ parameters.azureServiceConnection}} scriptType: bash scriptLocation: inlineScript inlineScript: | set -e # fail when script is failing sudo az aks install-cli az aks get-credentials --name ${{ parameters.aksName }} \ --resource-group ${{ parameters.aksRg }} \ --subscription ${{ parameters.aksSubscription }} \ --overwrite-existing \ --file .kubeconfig-${{ parameters.aksName }} export KUBECONFIG=$(pwd)/.kubeconfig-${{ parameters.aksName }} # Set default namespace kubectl config set-context ${{ parameters.aksName }} --namespace=${{ parameters.namespace }} kubectl config get-contexts # Pass kubeconfig to kubelogin to access k8s API kubelogin convert-kubeconfig -l azurecli # Rollback the deployment kubectl rollout undo deployment/${{ parameters.deploymentName }} - job: AksRollbackHealthCheck displayName: ${{ parameters.environment}} — K8S Rollback Health dependsOn: Rollback condition: eq(dependencies.Rollback.result, 'Succeeded') steps: - checkout: self displayName: Checkout fetchTags: false - task: AzureCLI@2 displayName: K8S Rollback Health inputs: azureSubscription: ${{ parameters.azureServiceConnection}} scriptType: bash scriptLocation: inlineScript inlineScript: | set -e # fail when script is failing sudo az aks install-cli az aks get-credentials --name ${{ parameters.aksName }} \ --resource-group ${{ parameters.aksRg }} \ --subscription ${{ parameters.aksSubscription }} \ --overwrite-existing \ --file .kubeconfig-${{ parameters.aksName }} export KUBECONFIG=$(pwd)/.kubeconfig-${{ parameters.aksName }} # Set default namespace kubectl config set-context ${{ parameters.aksName }} --namespace=${{ parameters.namespace }} kubectl config get-contexts # Pass kubeconfig to kubelogin to access k8s API kubelogin convert-kubeconfig -l azurecli # Check the rollout status of the deployment kubectl rollout status deployment/${{ parameters.deploymentName }} --timeout=${{ parameters.timeout}} - job: Clean displayName: Clean Up dependsOn: [ AksDeploymentHealthCheck, ManualApprovalOfRollBack, Rollback ] condition: always() steps: - checkout: none - script: | rm -rf ~/.kube/config displayName: Remove kube config Template behaviour:
Top comments (0)