Post 2 of 70 in the series “Mastering Kubernetes: A Practical Journey from Beginner to CKA”

Introduction

Picture this: You’re managing a bustling restaurant. You have a head chef (master) coordinating the kitchen, sous chefs managing different stations (worker nodes), and a complex system ensuring every order gets prepared correctly and delivered on time. This is exactly how Kubernetes orchestrates your containerized applications across a cluster of machines.

What we’ll learn today:

• The fundamental Kubernetes architecture explained
• How master nodes coordinate the entire cluster
• What worker nodes do and why they’re essential
• How these components communicate to run your applications

Why this matters: Understanding Kubernetes architecture is like learning to read a map before starting a journey. Every DevOps engineer needs this foundation to troubleshoot issues, optimize performance, and design reliable containerized systems. Companies like Netflix, Spotify, and Airbnb rely on this architecture to serve millions of users daily. By the end, you’ll diagnose cluster issues like a certified engineer.

Series context: Welcome to our comprehensive Kubernetes journey! This is where we build the foundational knowledge that every subsequent topic will build upon. Think of this as your architectural blueprint for everything we’ll explore together.

Prerequisites

What you need to know:

• Basic understanding of containers and Docker
• Familiarity with Linux command line
• Understanding of distributed systems concepts (helpful but not required)

📌 Quick Refresher: Containers are isolated processes that package applications with their dependencies. Kubernetes manages these containers at scale across multiple machines.

Tools required:

• A computer with internet access
• Text editor for taking notes
• Optional: Access to a Kubernetes cluster (we’ll set this up in future posts)

Previous posts to read:

• Post #1: Course Introduction and Learning Path (foundational context)

Estimated time: 15-20 minutes to read and understand the concepts

Step-by-Step Tutorial

Theory First: The Kubernetes Cluster Architecture

Kubernetes follows a master-worker architecture where the cluster consists of at least one master node (control plane) and multiple worker nodes. Think of it as a conductor (master) leading an orchestra (worker nodes) to create beautiful music (running applications).

Step 1: Understanding the Master Node (Control Plane)

The master node is the brain of your Kubernetes cluster. It makes all the decisions but doesn’t run your actual applications.

Understanding the Control Plane Components (Master Node):

 # Master Node Components apiVersion: v1 kind: ConfigMap metadata: name: cluster-info namespace: kube-public data: master-components: | - kube-apiserver: The front door to Kubernetes - etcd: The cluster's memory/database - kube-scheduler: Decides where pods should run - kube-controller-manager: Ensures desired state - cloud-controller-manager: Manages cloud resources (if applicable) 

Let’s break down each component:

1. kube-apiserver

• The only component that talks to etcd directly
• Validates and processes all REST requests
• Acts as the gateway for kubectl commands

2. etcd in Kubernetes: The Cluster’s Brain

etcd is Kubernetes’ distributed key-value store that:

• Think of it as the cluster’s “single source of truth”
• Stores all cluster state (pods, nodes, configs)
• Implements leader election for consistency
• Maintains change history through raft logs

⚠️ Production Alert: etcd requires dedicated SSDs. HDDs cause cluster instability! This is your cluster’s most critical component.

3. kube-scheduler

• Watches for newly created pods with no assigned node
• Selects the best worker node based on resource requirements
• Considers factors like CPU, memory, and affinity rules

4. kube-controller-manager

• Runs multiple controllers in a single process
• Node Controller: Monitors node health
• Replication Controller: Maintains desired number of pods
• Service Controller: Creates cloud load balancers

Step 2: Understanding Worker Nodes

Worker nodes are your compute engines – where application pods live. But here’s what most miss: A single worker can host 100+ pods, but requires careful resource planning (covered in Post #15).

Key Components of Worker Node:

 # Worker Node Components apiVersion: v1 kind: Node metadata: name: worker-node-01 spec: components: - kubelet: Node agent that communicates with master - kube-proxy: Network proxy for services - container-runtime: containerd/CRI-O for running containers (Docker deprecated) 

Component Details:

1. kubelet explained

The kubelet is Kubernetes’ primary node agent responsible for:

• The primary node agent
• Communicates with kube-apiserver
• Manages pods and containers lifecycle
• Reports node and pod status back to master
• Pod lifecycle management (start/stop containers)
• Executing health checks (liveness/readiness probes)
• Mounting storage volumes

**Critical Insight:** If kubelet crashes, the node appears as `NotReady` and stops receiving new workloads. Always monitor `kubelet_service_restarts_total` metric.

2. kube-proxy

• Maintains network rules on nodes
• Implements Kubernetes Service concept
• Handles load balancing for services

3. Container Runtime

• Downloads images and runs containers
• Current default: containerd (Docker deprecated in Kubernetes 1.24+)
• Other options: CRI-O, Docker (via dockershim compatibility)

Step 3: How Components Communicate

The magic happens through API calls and network communication:

 # Example: How a pod creation flows through the system # 1. User runs kubectl command kubectl create deployment nginx --image=nginx # 2. kubectl sends REST API call to kube-apiserver # 3. kube-apiserver validates request and stores in etcd # 4. kube-scheduler notices new pod, selects node # 5. kubelet on selected node pulls image and starts container # 6. kube-proxy updates network rules for service discovery 

❓ Check Understanding: When a pod is created, which component…

• a) First stores its metadata? (Answer: etcd)
• b) Assigns it to a node? (Answer: kube-scheduler)
• c) Actually runs it? (Answer: kubelet)

Step 4: Examining a Real Cluster

If you have access to a Kubernetes cluster, you can explore the architecture:

 # View cluster information kubectl cluster-info # Expected output: # Kubernetes control plane is running at https://your-cluster-api-server # CoreDNS is running at https://your-cluster-api-server/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy # List all nodes in the cluster kubectl get nodes # Expected output: # NAME STATUS ROLES AGE VERSION # master-node Ready control-plane 30d v1.28.0 # worker-node-1 Ready <none> 30d v1.28.0 # worker-node-2 Ready <none> 30d v1.28.0 # Get detailed node information kubectl describe node master-node 

Step 5: Understanding Node Roles and Labels

 # View node labels to understand roles kubectl get nodes --show-labels # Master nodes typically have: # node-role.kubernetes.io/control-plane= # node.kubernetes.io/exclude-from-external-load-balancers= # Worker nodes might have: # node-role.kubernetes.io/worker=worker 

Step 6: Exploring System Pods

System pods run the Kubernetes components:

 # View system pods (master node components) kubectl get pods -n kube-system # Expected output includes: # etcd-master-node # kube-apiserver-master-node # kube-controller-manager-master-node # kube-scheduler-master-node # kube-proxy-xxxxx (one per node) 

Verification Steps:

1. ✅ You can explain the role of master vs worker nodes
2. ✅ You understand what each component does
3. ✅ You can identify system pods in a cluster
4. ✅ You grasp the communication flow

Real-World Scenarios

Scenario 1: E-commerce Platform Architecture

Imagine you’re running an e-commerce platform like Amazon:

 # Production cluster might look like: # 3 Master nodes (high availability) # - Handle 1000s of API requests per second # - Store product catalog, user data, order information in etcd # - Schedule pods across multiple availability zones # 50+ Worker nodes distributed across regions # - Frontend pods (React applications) # - Backend API pods (microservices) # - Database pods (with persistent storage) # - Cache pods (Redis) 

Master nodes coordinate everything: when a user places an order, the scheduler decides which worker node should handle the payment processing pod, while controllers ensure the right number of inventory-checking pods are always running.

Scenario 2: Development vs Production Clusters

Development Cluster:

• Single master node (cost-effective)
• 2-3 worker nodes
• Less strict resource requirements

Production Cluster:

• 3+ master nodes (high availability)
• 10+ worker nodes (workload distribution)
• Strict resource isolation and security

Best practices:

• Never run application pods on master nodes in production
• Always have odd number of master nodes (3, 5, 7) for etcd quorum
• Monitor etcd health closely – it’s your single point of failure
• Use node taints to dedicated worker nodes for specific workloads

Common mistakes:

• Running a single master in production (no high availability)
• Not monitoring etcd disk space and performance
• Mixing master and worker roles without understanding the implications

Troubleshooting Tips

Common Error 1: “The connection to the server was refused”

Solution:

 # Check if kube-apiserver is running kubectl get pods -n kube-system | grep apiserver # If not running, check the logs sudo journalctl -u kubelet -f 

Common Error 2: Nodes showing “NotReady” status

Solution:

 # Check node status details kubectl describe node <node-name> # Common causes: # - kubelet not running # - Network plugin issues # - Disk space full 

Common Error 3: etcd Issues

etcd disk full:

 # Check etcd health (on control plane node) sudo ETCDCTL_API=3 etcdctl --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt endpoint health 

etcd leader election failures:

• Usually indicates network partitioning between master nodes
• Check master node connectivity and etcd member status

Debug Commands:

 # Essential cluster health commands kubectl get --raw='/readyz?verbose' # Control plane health (replaces deprecated componentstatuses) kubectl get nodes -o wide # Detailed node information kubectl top nodes # Resource usage kubectl get events --sort-by=.metadata.creationTimestamp # Recent events 

Where to get help:

• Official Kubernetes Architecture Documentation
• Kubernetes Slack Community (#kubernetes-users channel)
• Stack Overflow kubernetes tag

Kubernetes Architecture FAQs

General Architecture Questions

Component-Specific Questions

Networking and Communication

Practical Operations

How can I check if my cluster is healthy?

A: Use these essential commands:

 kubectl get nodes # Node status kubectl get pods -n kube-system # System components kubectl get --raw='/readyz?verbose' # Control plane health kubectl top nodes # Resource usage 

My node shows “NotReady” – what should I check?

Common causes and solutions:

kubelet not running: sudo systemctl status kubelet
Network issues: Check CNI plugin pods
Disk space full: df -h on the node
Resource exhaustion: Check CPU/memory usage

Follow this diagnostic flow:

Sizing and Planning

Troubleshooting

Advanced Questions

Next Steps

What’s coming next: In Post #3, we’ll dive deep into “Docker vs ContainerD: What Changed and Why It Matters.” In our container runtime deep dive, you’ll discover why Docker was replaced and how it impacts your cluster’s security.

Additional learning:

• Explore the Kubernetes Components Interactive Tutorial
• Read about cluster networking concepts

Practice challenges:

1. Try This: If you have access to a cluster, run kubectl get pods -n kube-system and identify which pods correspond to master node components
2. Explore: Use kubectl describe on both a master and worker node – what differences do you notice in their configurations?
3. Think: Design a cluster architecture for a social media app with 1 million users – how many master/worker nodes would you recommend?

Community engagement: Share your cluster exploration results in the comments! What interesting configurations did you discover? Ask questions about anything that wasn’t clear – the Kubernetes community loves helping fellow learners.

🔗 Series Navigation

Previous: Post #1 – Course Introduction and Learning Path
Next: Post #3 – Docker vs ContainerD: What Changed and Why It Matters
Progress: You’re now 3% through the Kubernetes Fundamentals series! 🎉

💡 Pro Tip: Bookmark this post as a reference. As we explore pods, services, and deployments in upcoming posts, you’ll frequently need to remember how these components interact with the master and worker nodes we covered today.

📧 Never miss an update: Subscribe to get notified when new posts in this series are published. We’re building something amazing together!

Tags: kubernetes, cluster-architecture, master-node, worker-node, devops, containers, orchestration, cka-prep