Download to read offline
Uploaded byEdgaras Apšega
Lessons learned using GitOps
Vinted uses GitOps to deploy thousands of pods to Kubernetes. Some key points: - Vinted infrastructure has grown to 2000 deployments per day and over 10,000 running pods across 480+ physical nodes. - GitOps is a DevOps process using Git for deployment and management of containerized applications. It promotes repeatability, reliability, and efficiency. - Vinted uses Helm charts stored in Git to define application manifests and environments. Jenkins jobs trigger on commits to update image tags and deploy applications.
Lessons learned using GitOps
- 1. How Vinted usesGitOps to deploy thousands of pods to Kubernetes Edgaras Apšega Site Reliability Engineer / CNCF Ambassador @apsega
- 2. Vinted infrastructure 3x growthover 2 years Kubernetes production stats ● All services running on Kubernetes* ● 2000 deployments per day ● 10k+ running pods ● 480+ physical nodes (70k CPU cores; 190TB memory) *Almost
- 3.
- 4. GitOps Coined in August2017 by Weaveworks CEO Alexis Richardson GitOps is a DevOps process characterized by: Best practices of deployment, management and monitoring of containerized applications Experience for managing applications with fully automated pipelines/workflows using Git for development and operations Use of Git revision control system to track and approve changes to the infrastructure
- 5. Why GitOps? Infrastructure asa Code (IaC) Repeatability Reliability Efficiency Visibility Self-service / Internal Developer Platform (Backstage) Code reviews
- 6. Imperative vs. Declarative ImperativeDeclarative $ kubectl run busybox --image=busybox:1.29 --restart=Never --command -- sleep 3600 apiVersion: v1 kind: Pod metadata: labels: run: busybox name: busybox spec: containers: - command: - sleep - "3600" image: busybox:1.29 imagePullPolicy: Always name: busybox
- 7. Build Docker push Test Git commit and push Gitclone config repo Update manifests kubectl apply Git clone config repo Discover manifests GitOps continuous integration GitOps continuous deployment
- 8. Git strategies Single branch(multiple directories) Multiple branches Do use directories for GitOps environments kubernetes-deployments ├── demo-app │ ├── development │ ├── production │ └── staging └── guestbook ├── development ├── production └── staging Don’t use long-running branches for GitOps environments
- 9.
- 10. Configuration management Helm Kustomize Packagemanager Go templating language Environments per values files No parameters and templates - as close as you can get to Kubernetes manifests No parameters and templates - limiting in edge cases Overlays per environment More a language than a tool JSON with comments and templating Not Kubernetes specific Jsonnet
- 11.
- 12. ArgoCD App ofApps Pattern App of Apps manifest ArgoCD custom plugins
- 13. Kubernetes deployments Kubernetes deploymentsdirectory Application manifests kubernetes-deployments ├── guestbook │ ├── frontend │ │ ├── development-values.yaml │ │ ├── development-application.yaml │ │ ├── staging-values.yaml │ │ ├── staging-application.yaml │ │ ├── production-values.yaml │ │ └── production-application.yaml │ ├── backend │ │ ├── development-values.yaml │ │ ├── development-application.yaml │ │ ├── staging-values.yaml │ │ ├── staging-application.yaml │ │ ├── production-values.yaml │ │ └── production-application.yaml │ └── common-values.yaml └── demo-app ├── development-values.yaml ├── development-application.yaml ├── staging-values.yaml ├── staging-application.yaml ├── production-values.yaml └── production-application.yaml
- 14. Helm charts Centralized Helmcharts Helm chart values files
- 15. Helm charts (2) CentralizedHelm charts Helm chart values files in Kubernetes deployments repository
- 16.
- 17. Change image tagin deployments repo Jenkinsfile in code repo git clone yq write --inplace "${params.ENVIRONMENT}-values.yaml" --tag '!!str' image.tag "${params.IMAGE_TAG}" git push stage('Build') { when { branch 'master' } steps { echo 'Build and push Docker image' DockerImageBuildAndPublish('frontend') } } stage('Deploy App to Kubernetes') { when { branch 'master' } steps { KubernetesDeploymentsApply('frontend', 'production') } } Jenkins update image tag action
- 18. ArgoCD Sync waves Supportsapplication dependencies and defines deployments order Supports only definitions within same defined application Prolongs deployments Use when doing frequent deployments argocd app sync $(APP_NAME) --revision ${ARGOCD_APP_REVISION} Sync wave example Sync waves
- 19. Bonus: scaling ArgoCD forhigh load
- 20.
- 21. Application controller Too manyapplications: increase number of processes Too many clusters: shard application controller (increase replicas)
- 22. Typically running 3-4servers is enough reposerver.parallelism.limit Cache with webhooks Repo server
- 23. GitOps is awesome(while GitHub/GitLab works) Use directories for GitOps environments Separate code and configuration repositories Use common values for repetitive configuration Avoid using cross application dependencies
- 24.