Downloaded 204 times
Uploaded byJ On The Beach
An intro to Kubernetes operators
The document provides an introduction to Kubernetes operators, which are controllers that manage application deployment and operations on Kubernetes. It explains key concepts such as custom resource definitions (CRDs), the controller pattern, and the use of tools like kubebuilder and operator-sdk for building operators. Additionally, it discusses scenarios for when to create an operator versus using simpler configurations, along with methods for deploying and debugging operators.
In this document
Powered by AI
Overview of Kubernetes operators, presented by Lili Cosic, a Software Engineer at Red Hat.
Kubernetes as an open-source platform for managing containerized workloads, its origins, capabilities, and misconceptions.
Introduction to Kubernetes controllers, their functions, and the Informer pattern for handling resource events.
Explanation of the Workqueue mechanism in Kubernetes and how Custom Resource Definitions (CRDs) are structured.
Definition of what operators are within Kubernetes, their reliance on CRDs, and use cases for building operators.
Guidelines on when it's appropriate to create an operator versus using simpler Kubernetes objects like ConfigMaps.
Various methodologies and tools for building operators, including Go language libraries, Kubebuilder, and Operator SDK.
Details on deploying operators within Kubernetes, including YAML manifest files for CRDs and deployments.
Key features of the Operator Lifecycle Manager (OLM) and best practices for debugging operators in Kubernetes.
Discussion on utilizing Kubernetes for big data applications, with resources from special interest groups.
Expectations for future operator developments within Kubernetes, focusing on OLM and CRDs.
An open session for audience questions to conclude the presentation.
An intro to Kubernetes operators
- 1. An intro toKubernetes operators Lili Cosic Software Engineer at Red Hat Twitter: @LiliCosic Github: lilic
- 2. An Intro toKubernetes Operators - Lili Cosic2 Intro to Kubernetes
- 3. An Intro toKubernetes Operators - Lili Cosic3 Kubernetes What is Kubernetes? ● Open source platform for managing containerized workloads and services ● Containers, containers, containers ● Name originates from Greek - helmsman or pilot ● Google open-sourced in 2014 ● Based on Borg - Google’ internal project ● K_ _ _ _ _ _ _ _S -> k8s
- 4. An Intro toKubernetes Operators - Lili Cosic4 Kubernetes What is great about Kubernetes? ● Scalability of workloads ● Separation of workloads ● Native stable resources (Deployments, Pods) ● API / Custom Resources
- 5. An Intro toKubernetes Operators - Lili Cosic5 Kubernetes What is Kubernetes not? ● Not a PaaS ● Not limited to the types of apps supported ● Not opinionated on: ○ Deploying ○ Run CI/CD ○ Logging, monitoring or alerting
- 6. An Intro toKubernetes Operators - Lili Cosic6 Kubernetes controllers
- 7. An Intro toKubernetes Operators - Lili Cosic7 What is a controller? ● Reconciles given state ● Controller pattern ○ Non terminating loop that regulates the state of the system ○ A control loop that watches the shared state of the cluster via the API server and makes changes to move from current to desired state
- 8. An Intro toKubernetes Operators - Lili Cosic8 ReplicaSet controller Example controller ● ReplicaSet ○ Created by a Deployments ○ Creates Pods ● Reconciles to the correct (specified) number of pods running the cluster
- 9. An Intro toKubernetes Operators - Lili Cosic9 Controller Credit: github.com/kubernetes/sample-controller
- 10. An Intro toKubernetes Operators - Lili Cosic10 Informer pattern More important API functions ● ListWatcher ○ Helps you filter resources you want to watch ● ResourceEventHandler ○ Add, Update and Delete event trigger functions
- 11. An Intro toKubernetes Operators - Lili Cosic11 Workqueue? ● Resource Event Handler puts an item to the workqueue ● Workqueue consists of keys: ○ <resource_namespace>/<resource_name> ● Only one worker works on one item at a time ● First In First Out
- 12. An Intro toKubernetes Operators - Lili Cosic12 CRD - Custom Resource Definition
- 13. An Intro toKubernetes Operators - Lili Cosic13 CRD - custom resource definition apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: # name must match the spec fields below, and be in the form: <plural>.<group> name: crontabs.stable.example.com spec: # group name to use for REST API: /apis/<group>/<version> group: stable.example.com # list of versions supported by this CustomResourceDefinition versions: - name: v1 # Each version can be enabled/disabled by Served flag. served: true # One and only one version must be marked as the storage version. storage: true # either Namespaced or Cluster scope: Namespaced names: # plural name to be used in the URL: /apis/<group>/<version>/<plural> plural: crontabs # singular name to be used as an alias on the CLI and for display singular: crontab # kind is normally the CamelCased singular type. Your resource manifests use this. kind: CronTab # shortNames allow shorter string to match your resource on the CLI shortNames: - ct
- 14. An Intro toKubernetes Operators - Lili Cosic14 Operators, operators, operators
- 15. An Intro toKubernetes Operators - Lili Cosic15
- 16. An Intro toKubernetes Operators - Lili Cosic16 What is an “operator”? Operator is a K8s controller - specific to operating an application
- 17. An Intro toKubernetes Operators - Lili Cosic17 What is an “operator”? ● Non core Kubernetes controller ● Makes use of CRDs - custom resource definition ● Holds the knowledge of how an application needs to be deployed, managed and packaged ● Reconcile loop ● Extends the Kubernetes API to make the application a custom resource - part of the Kubernetes ecosystem ● Concept introduced by CoreOS
- 18. An Intro toKubernetes Operators - Lili Cosic18 Good examples of operators Awesome operators! ● github.com/coreos/prometheus-operator ● github.com/zalando-incubator/postgres-operator ● github.com/coreos/etcd-operator
- 19. An Intro toKubernetes Operators - Lili Cosic19 When to choose creating an operator? ● Application uses declarative API ● Resources are scoped to a namespace or a cluster ● Encapsulate business logic ● Build automation that watches for updates of Kubernetes objects ● Create or update resources via the Kubernetes native API ● Top level support from kubectl
- 20. An Intro toKubernetes Operators - Lili Cosic20 When to just use a ConfigMap or Secret? Sometimes creating an operator is not needed ● Existing well known config file format (e.g. mysql.cnf ) ● Config file is used to just configure an application running in a Pod ● No need to reconcile to the state
- 21. An Intro toKubernetes Operators - Lili Cosic21 101 ways to build an operator ● go language: ○ client-go & co. ○ operator-sdk ○ kubebuilder ● Other languages: ○ Kubernetes python and java clients and others ● Other tools: ○ helm ○ ansible
- 22. An Intro toKubernetes Operators - Lili Cosic22 Building operators using Kubernetes native clients
- 23. An Intro toKubernetes Operators - Lili Cosic23 Kubernetes native clients ● k8s.io/client-go ○ rest ○ discovery ○ dynamic ● k8s.io/api ● ks8.io/apimachinery
- 24. An Intro toKubernetes Operators - Lili Cosic24 Kubernetes native clients Pros ● Same as upstream controllers use ● Stability of K8s code ● Versioning based on Kubernetes releases ● Ability to finetune Cons ● Large ecosystem ● No abstractions/helpers ● A lot of inside knowledge to optimize correctly ● New major version on every Kubernetes minor version release
- 25. An Intro toKubernetes Operators - Lili Cosic25 Using Kubernetes native go clients Example of an “operator” github.com/kubernetes/sample-controller
- 26. An Intro toKubernetes Operators - Lili Cosic26 kubebuilder
- 27. An Intro toKubernetes Operators - Lili Cosic27 kubebuilder ● github.com/kubernetes-sigs/kubebuilder ● SDK for building Kubernetes APIs using CRDs ● Part of kubernetes-sigs repo ● Uses controller-runtime under the hood ● Doesn’t strictly advertise itself as operator builder tool
- 28. An Intro toKubernetes Operators - Lili Cosic28 Using kubebuilder Example of a go operator $ mkdir kubebuilder-operator && cd kubebuilder-operator $ kubebuilder init --domain k8s.io --license apache2 --owner "The JOnTheBeach Audience" $ kubebuilder create api --group ships --version v1beta1 --kind Sloop $ # Edit the logic code $ pkg/controller/sloop/sloop_controller.go
- 29. An Intro toKubernetes Operators - Lili Cosic29 operator-sdk
- 30. An Intro toKubernetes Operators - Lili Cosic30 operator-sdk ● github.com/operator-framework/operator-sdk ● Aimed at creating operators ● Part of operator-framework ● Operator types you can create: ○ go ○ helm ○ ansible ● Testing framework
- 31. An Intro toKubernetes Operators - Lili Cosic31 Using operator-sdk Example of a go operator $ operator-sdk new app-operator $ cd app-operator $ # Add a new API for the custom resource AppService $ operator-sdk add api --api-version=app.example.com/v1alpha1 --kind=AppService $ # Add a new controller that watches for AppService $ operator-sdk add controller --api-version=app.example.com/v1alpha1 --kind=AppService $ # Build operator $ operator-sdk build quay.io/example/app-operator
- 32. An Intro toKubernetes Operators - Lili Cosic32 operator-sdk - helm
- 33. An Intro toKubernetes Operators - Lili Cosic33 Using operator-sdk Example of a helm operator $ # Create a new helm operator project $ operator-sdk new nginx-operator --api-version=example.com/v1alpha1 --kind=Nginx --type=helm $ # Edit watches.yaml file to customize the operator logic
- 34. An Intro toKubernetes Operators - Lili Cosic34 operator-sdk - ansible
- 35. An Intro toKubernetes Operators - Lili Cosic35 Using operator-sdk Example of an ansible operator $ # Create a new ansible based operator project: $ operator-sdk new memcached-operator --api-version=cache.example.com/v1alpha1 --kind=Memcached --type=ansible $ # Edit watches file to customize the logic
- 36. An Intro toKubernetes Operators - Lili Cosic36 Deploying an operator
- 37. An Intro toKubernetes Operators - Lili Cosic37 Deploying (operator) on Kubernetes ● YAML ● Kustomize ● Jsonnet ● OLM ● Helm
- 38. An Intro toKubernetes Operators - Lili Cosic38 YAML $ # Register the CRD $ kubectl apply -f crd.yaml $ # Create any needed Role Based Access Control $ kubectl apply -f rbac.yaml $ # Deploy the operator $ kubectl apply -f deploy.yaml $ # Create an instance $ kubectl apply -f cr.yaml
- 39. An Intro toKubernetes Operators - Lili Cosic39 YAML - CRD manifest file apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: memcacheds.cache.example.com spec: group: cache.example.com names: kind: Memcached listKind: MemcachedList plural: memcacheds singular: memcached scope: Namespaced subresources: status: {} validation: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#types-kinds' type: string metadata: type: object spec: properties: size: format: int32 type: integer required: - size type: object status: properties: nodes: items: type: string type: array required: - nodes type: object version: v1alpha1 versions: - name: v1alpha1 served: true storage: true
- 40. An Intro toKubernetes Operators - Lili Cosic40 YAML - deployment manifest file apiVersion: apps/v1 kind: Deployment metadata: name: memcached-operator spec: replicas: 1 selector: matchLabels: name: memcached-operator template: metadata: labels: name: memcached-operator spec: serviceAccountName: memcached-operator containers: - name: memcached-operator image: memached:123 command: - memcached-operator imagePullPolicy: Always env: - name: WATCH_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: OPERATOR_NAME value: "memcached-operator"
- 41. An Intro toKubernetes Operators - Lili Cosic41 YAML - CR instance manifest file apiVersion: cache.example.com/v1alpha1 kind: Memcached metadata: name: example-memcached spec: size: 3
- 42. An Intro toKubernetes Operators - Lili Cosic42 OLM ● Upgrades ● Descriptions and metadata ● Dependency resolution ● Multiple versions of operator to install Key features
- 43. An Intro toKubernetes Operators - Lili Cosic43 Debugging an operator
- 44. An Intro toKubernetes Operators - Lili Cosic44 Debugging an operator ● `operator-sdk up local` - run ● Log all the things! ● Get logs of an operator ○ `kubectl logs <pod-name>` ● Use CR name to label any pods the operator creates ● `kubectl events -n NAMESPACE` ● `Spec.Paused`
- 45. An Intro toKubernetes Operators - Lili Cosic45 Kubernetes cluster stack for big data
- 46. An Intro toKubernetes Operators - Lili Cosic46 More information ● Kubernetes Special Interest Group Big Data ● User group meeting: Wednesdays at 18:00 UTC (biweekly) ● Slack channel - #ug-big-data
- 47. An Intro toKubernetes Operators - Lili Cosic47 Spark operator Spark on Kubernetes github.com/GoogleCloudPlatform/spark-on-k8s-operator
- 48. An Intro toKubernetes Operators - Lili Cosic48 Future Whats next for operators? ● OLM ● cluster-addons ● CRDs ○ GA in 1.16
- 49.