DEV Community

InstaDevOps
InstaDevOps

Posted on • Originally published at instadevops.com

Container Security: From Development to Production

#kubernetes #docker #devops #security

Introduction

Containers have revolutionized application deployment, but they've also introduced new security challenges. While containers provide isolation, they're not a security silver bullet. A misconfigured container can expose your entire infrastructure to attack, leak sensitive data, or become a foothold for lateral movement within your network.

In this comprehensive guide, we'll explore container security best practices across the entire software development lifecycle—from writing Dockerfiles to running containers in production. Whether you're running Docker locally or managing thousands of containers in Kubernetes, these practices will help you build a robust security posture.

Understanding Container Security Risks

Before diving into solutions, let's understand the unique security challenges containers present:

Image Vulnerabilities

Container images often contain outdated packages with known vulnerabilities. A study by Snyk found that 75% of Docker Hub images contain at least one critical vulnerability.

Overprivileged Containers

Many containers run as root by default, violating the principle of least privilege. If compromised, attackers gain root access within the container.

Insecure Image Registries

Public registries contain malicious images disguised as legitimate software. Even private registries can be misconfigured to allow unauthorized access.

Secrets in Images

Developers accidentally commit credentials, API keys, and certificates into container images, exposing them to anyone with image access.

Container Escape

While rare, vulnerabilities in container runtimes can allow attackers to escape containers and access the host system.

Supply Chain Attacks

Base images from untrusted sources might contain backdoors or malware, compromising your entire application stack.

Secure Docker Image Development

Start with Minimal Base Images

Use minimal base images to reduce attack surface. Distroless or Alpine images contain far fewer packages than full OS images.

# Bad: Full Ubuntu image (78MB, hundreds of packages) FROM ubuntu:22.04 # Better: Alpine image (5MB, minimal packages) FROM alpine:3.18 # Best: Distroless image (smallest, no shell, no package manager) FROM gcr.io/distroless/nodejs18-debian11
Enter fullscreen mode Exit fullscreen mode

Multi-Stage Builds for Smaller Images

Use multi-stage builds to keep build tools out of production images:

# Build stage FROM node:18-alpine AS builder WORKDIR /app COPY package*.json ./ RUN npm ci --only=production COPY . . RUN npm run build # Production stage FROM gcr.io/distroless/nodejs18-debian11 WORKDIR /app COPY --from=builder /app/dist ./dist COPY --from=builder /app/node_modules ./node_modules COPY package*.json ./ USER nonroot CMD ["node", "dist/index.js"]
Enter fullscreen mode Exit fullscreen mode

Never Run as Root

Create and use a non-root user in your Dockerfile:

FROM node:18-alpine # Create app user RUN addgroup -g 1001 -S appuser && \  adduser -u 1001 -S appuser -G appuser # Set ownership WORKDIR /app COPY --chown=appuser:appuser . . # Install dependencies as root (needed for npm install) RUN npm ci --only=production # Switch to non-root user USER appuser # Run as non-root CMD ["node", "index.js"]
Enter fullscreen mode Exit fullscreen mode

Scan Images for Vulnerabilities

Integrate vulnerability scanning into your CI/CD pipeline:

# Using Trivy docker run aquasec/trivy image myapp:latest # Using Snyk snyk container test myapp:latest # Using Grype grype myapp:latest
Enter fullscreen mode Exit fullscreen mode 
# GitHub Actions example name: Container Security Scan on: push: branches: [main] pull_request: branches: [main] jobs: scan: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: Build image run: docker build -t myapp:${{ github.sha }} . - name: Run Trivy vulnerability scanner uses: aquasecurity/trivy-action@master with: image-ref: 'myapp:${{ github.sha }}' format: 'sarif' output: 'trivy-results.sarif' severity: 'CRITICAL,HIGH' - name: Upload Trivy results to GitHub Security uses: github/codeql-action/upload-sarif@v2 with: sarif_file: 'trivy-results.sarif' - name: Fail on high severity vulnerabilities uses: aquasecurity/trivy-action@master with: image-ref: 'myapp:${{ github.sha }}' exit-code: '1' severity: 'CRITICAL,HIGH'
Enter fullscreen mode Exit fullscreen mode

Don't Store Secrets in Images

Never bake secrets into images. Use environment variables, secret management tools, or mount secrets at runtime:

# Bad: Secret hardcoded in image ENV API_KEY="sk-1234567890abcdef" # Bad: Secret copied into image COPY secrets.env /app/ # Good: Secret passed at runtime # docker run -e API_KEY="$API_KEY" myapp # Better: Secret from secrets management # Kubernetes Secret, AWS Secrets Manager, HashiCorp Vault
Enter fullscreen mode Exit fullscreen mode 
# Kubernetes: Mount secrets as environment variables apiVersion: v1 kind: Pod metadata: name: myapp spec: containers: - name: app image: myapp:latest env: - name: API_KEY valueFrom: secretKeyRef: name: myapp-secrets key: api-key
Enter fullscreen mode Exit fullscreen mode

Use .dockerignore

Prevent sensitive files from being copied into images:

# .dockerignore .git .env .env.local *.log node_modules npm-debug.log secrets/ *.pem *.key .aws .gcp
Enter fullscreen mode Exit fullscreen mode

Pin Image Versions

Always use specific image versions, never latest:

# Bad: Unpredictable, can break without warning FROM node:latest # Bad: Still somewhat unpredictable FROM node:18 # Good: Specific version FROM node:18.17.1 # Best: Specific version + digest (immutable) FROM node:18.17.1-alpine@sha256:f1657204d3463bce763cefa5b25e48c28af6eb183b4b6b06f7d64a8e2f18314
Enter fullscreen mode Exit fullscreen mode

Limit Image Layer Count

Combine RUN commands to reduce layers and image size:

# Bad: Many layers RUN apt-get update RUN apt-get install -y curl RUN apt-get install -y git RUN apt-get clean RUN rm -rf /var/lib/apt/lists/* # Good: Single layer RUN apt-get update && \  apt-get install -y --no-install-recommends curl git && \  apt-get clean && \  rm -rf /var/lib/apt/lists/*
Enter fullscreen mode Exit fullscreen mode

Container Runtime Security

Use Read-Only Filesystems

Make container filesystems read-only to prevent tampering:

# Docker Compose services: app: image: myapp:latest read_only: true tmpfs: - /tmp - /var/run # Kubernetes apiVersion: v1 kind: Pod metadata: name: myapp spec: containers: - name: app image: myapp:latest securityContext: readOnlyRootFilesystem: true volumeMounts: - name: tmp mountPath: /tmp volumes: - name: tmp emptyDir: {}
Enter fullscreen mode Exit fullscreen mode

Drop Unnecessary Capabilities

Linux capabilities allow fine-grained privilege control. Drop all capabilities, then add only what's needed:

# Kubernetes Pod with minimal capabilities apiVersion: v1 kind: Pod metadata: name: myapp spec: containers: - name: app image: myapp:latest securityContext: capabilities: drop: - ALL add: - NET_BIND_SERVICE # Only if binding to ports <1024 allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1001 seccompProfile: type: RuntimeDefault
Enter fullscreen mode Exit fullscreen mode

Resource Limits

Prevent container resource exhaustion attacks:

# Docker Compose services: app: image: myapp:latest deploy: resources: limits: cpus: '1' memory: 512M reservations: cpus: '0.5' memory: 256M # Kubernetes resources: requests: memory: "256Mi" cpu: "500m" limits: memory: "512Mi" cpu: "1000m"
Enter fullscreen mode Exit fullscreen mode

Use Security Profiles

AppArmor and Seccomp restrict system calls containers can make:

# Kubernetes with AppArmor and Seccomp apiVersion: v1 kind: Pod metadata: name: myapp annotations: container.apparmor.security.beta.kubernetes.io/app: runtime/default spec: containers: - name: app image: myapp:latest securityContext: seccompProfile: type: RuntimeDefault
Enter fullscreen mode Exit fullscreen mode

Network Segmentation

Isolate containers with network policies:

# Kubernetes NetworkPolicy apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: myapp-netpol spec: podSelector: matchLabels: app: myapp policyTypes: - Ingress - Egress ingress: # Only allow traffic from specific pods - from: - podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 8080 egress: # Only allow traffic to database - to: - podSelector: matchLabels: app: postgres ports: - protocol: TCP port: 5432 # Allow DNS - to: - namespaceSelector: matchLabels: name: kube-system ports: - protocol: UDP port: 53
Enter fullscreen mode Exit fullscreen mode

Registry Security

Use Private Registries

Host images in private registries with access control:

# AWS ECR aws ecr get-login-password --region us-east-1 | \ docker login --username AWS --password-stdin \ 123456789.dkr.ecr.us-east-1.amazonaws.com # Google Container Registry gcloud auth configure-docker # Azure Container Registry az acr login --name myregistry
Enter fullscreen mode Exit fullscreen mode

Sign Images

Use Docker Content Trust or Notary to sign images:

# Enable Docker Content Trust export DOCKER_CONTENT_TRUST=1 # Push signed image docker push myregistry.com/myapp:v1.0 # Verify signature on pull docker pull myregistry.com/myapp:v1.0
Enter fullscreen mode Exit fullscreen mode

Implement Image Scanning in Registry

Many registries provide built-in vulnerability scanning:

# AWS ECR: Enable scanning on push aws ecr put-image-scanning-configuration \ --repository-name myapp \ --image-scanning-configuration scanOnPush=true # Google Artifact Registry: Enable vulnerability scanning gcloud artifacts repositories update myrepo \ --location=us-central1 \ --enable-scanning
Enter fullscreen mode Exit fullscreen mode

Use Admission Controllers

Prevent unsigned or vulnerable images from running:

# OPA Gatekeeper policy: Only allow images from approved registries apiVersion: constraints.gatekeeper.sh/v1beta1 kind: K8sAllowedRepos metadata: name: allowed-repos spec: match: kinds: - apiGroups: [""] kinds: ["Pod"] - apiGroups: ["apps"] kinds: ["Deployment", "StatefulSet"] parameters: repos: - "myregistry.com/" - "gcr.io/myproject/"
Enter fullscreen mode Exit fullscreen mode

Kubernetes-Specific Security

Pod Security Standards

Enforce pod security policies cluster-wide:

# Restricted Pod Security Standard (most secure) apiVersion: v1 kind: Namespace metadata: name: production labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/warn: restricted
Enter fullscreen mode Exit fullscreen mode

RBAC for Least Privilege

Grant minimal permissions to service accounts:

apiVersion: v1 kind: ServiceAccount metadata: name: myapp-sa namespace: production --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: myapp-role namespace: production rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "list"] - apiGroups: [""] resources: ["secrets"] resourceNames: ["myapp-secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: myapp-rolebinding namespace: production subjects: - kind: ServiceAccount name: myapp-sa namespace: production roleRef: kind: Role name: myapp-role apiGroup: rbac.authorization.k8s.io
Enter fullscreen mode Exit fullscreen mode

Secrets Encryption at Rest

Encrypt Kubernetes secrets in etcd:

# /etc/kubernetes/encryption-config.yaml apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: <BASE64_ENCODED_SECRET> - identity: {}
Enter fullscreen mode Exit fullscreen mode

Audit Logging

Enable comprehensive audit logging:

apiVersion: audit.k8s.io/v1 kind: Policy rules: # Log admin actions - level: RequestResponse users: ["admin"] # Log secret access - level: Metadata resources: - group: "" resources: ["secrets"] # Log pod exec/attach - level: Request verbs: ["exec", "attach"] resources: - group: "" resources: ["pods/exec", "pods/attach"]
Enter fullscreen mode Exit fullscreen mode

Runtime Security Monitoring

Use Falco for Threat Detection

Falco monitors kernel events to detect anomalous behavior:

# Falco rule: Detect shell in container - rule: Terminal Shell in Container desc: A shell was spawned in a container condition: > spawned_process and container and proc.name in (shell_binaries) output: > Shell spawned in container (user=%user.name container=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline) priority: WARNING # Detect outbound connections to suspicious IPs - rule: Outbound Connection to Suspicious IP desc: Container connected to known malicious IP condition: > outbound and container and fd.sip in (suspicious_ips) output: > Suspicious outbound connection (container=%container.name destination=%fd.sip:%fd.sport command=%proc.cmdline) priority: CRITICAL
Enter fullscreen mode Exit fullscreen mode

Container Runtime Security Tools 

# Deploy Falco as DaemonSet apiVersion: apps/v1 kind: DaemonSet metadata: name: falco namespace: falco spec: selector: matchLabels: app: falco template: metadata: labels: app: falco spec: serviceAccount: falco hostNetwork: true hostPID: true containers: - name: falco image: falcosecurity/falco:latest securityContext: privileged: true volumeMounts: - mountPath: /host/var/run/docker.sock name: docker-socket - mountPath: /host/dev name: dev-fs - mountPath: /host/proc name: proc-fs readOnly: true volumes: - name: docker-socket hostPath: path: /var/run/docker.sock - name: dev-fs hostPath: path: /dev - name: proc-fs hostPath: path: /proc
Enter fullscreen mode Exit fullscreen mode

Secrets Management

External Secrets Operator

Sync secrets from external vaults into Kubernetes:

apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: myapp-secrets spec: refreshInterval: 1h secretStoreRef: name: aws-secrets-manager kind: SecretStore target: name: myapp-secrets creationPolicy: Owner data: - secretKey: database-password remoteRef: key: prod/myapp/db-password - secretKey: api-key remoteRef: key: prod/myapp/api-key
Enter fullscreen mode Exit fullscreen mode

Sealed Secrets

Encrypt secrets in Git for GitOps workflows:

# Encrypt secret echo -n 'supersecret' | kubectl create secret generic mysecret \ --dry-run=client --from-file=password=/dev/stdin -o yaml | \ kubeseal -o yaml > mysealedsecret.yaml # Commit encrypted secret to Git git add mysealedsecret.yaml git commit -m "Add sealed secret"
Enter fullscreen mode Exit fullscreen mode

Security Scanning and Compliance

Comprehensive Security Scanning Pipeline 

# Complete security scanning pipeline name: Security Pipeline on: push: branches: [main] pull_request: branches: [main] jobs: security-scan: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 # 1. Scan source code for secrets - name: Secret scanning uses: trufflesecurity/trufflehog@main with: path: ./ # 2. SAST - Static Application Security Testing - name: Run Semgrep uses: returntocorp/semgrep-action@v1 # 3. Dependency scanning - name: Run Snyk uses: snyk/actions/node@master env: SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }} # 4. Build image - name: Build Docker image run: docker build -t myapp:${{ github.sha }} . # 5. Container scanning - name: Run Trivy uses: aquasecurity/trivy-action@master with: image-ref: 'myapp:${{ github.sha }}' severity: 'CRITICAL,HIGH' exit-code: '1' # 6. IaC scanning - name: Run Checkov uses: bridgecrewio/checkov-action@master with: directory: k8s/ framework: kubernetes # 7. License compliance - name: License scanning run: | npm install -g license-checker license-checker --production --onlyAllow 'MIT;Apache-2.0;BSD-3-Clause'
Enter fullscreen mode Exit fullscreen mode

Compliance and Hardening

CIS Benchmark Compliance

Follow CIS Docker and Kubernetes benchmarks:

# Run Docker Bench Security docker run --rm --net host --pid host --userns host --cap-add audit_control \ -v /var/lib:/var/lib \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /etc:/etc \ docker/docker-bench-security # Run kube-bench for Kubernetes kubectl apply -f https://raw.githubusercontent.com/aquasecurity/kube-bench/main/job.yaml kubectl logs job/kube-bench
Enter fullscreen mode Exit fullscreen mode

Incident Response

Container Forensics

When a container is compromised:

# 1. Don't delete the container immediately # Pause it to preserve state docker pause <container_id> # 2. Export filesystem for analysis docker export <container_id> > compromised-container.tar # 3. Inspect container configuration docker inspect <container_id> > container-config.json # 4. Check logs docker logs <container_id> > container-logs.txt # 5. Review network connections docker exec <container_id> netstat -antp # 6. Check running processes docker top <container_id> # 7. After analysis, kill and remove docker kill <container_id> docker rm <container_id>
Enter fullscreen mode Exit fullscreen mode

Best Practices Checklist

Development Phase

  • [ ] Use minimal base images (Alpine, Distroless)
  • [ ] Implement multi-stage builds
  • [ ] Run containers as non-root user
  • [ ] Pin image versions with digests
  • [ ] Use .dockerignore to exclude sensitive files
  • [ ] Never commit secrets to images
  • [ ] Scan images for vulnerabilities
  • [ ] Keep images under 500MB
  • [ ] Document security considerations

Build Phase

  • [ ] Automated vulnerability scanning in CI/CD
  • [ ] SAST scanning for code vulnerabilities
  • [ ] Dependency scanning for vulnerable packages
  • [ ] Secret scanning to prevent leaks
  • [ ] License compliance checking
  • [ ] Sign images before pushing to registry

Registry Phase

  • [ ] Use private registries with authentication
  • [ ] Enable vulnerability scanning on push
  • [ ] Implement image retention policies
  • [ ] Regular security audits of registry access
  • [ ] Enable image signing and verification

Runtime Phase

  • [ ] Use read-only filesystems
  • [ ] Drop unnecessary Linux capabilities
  • [ ] Enable security profiles (AppArmor/Seccomp)
  • [ ] Implement resource limits
  • [ ] Use network policies for isolation
  • [ ] Enable audit logging
  • [ ] Deploy runtime security monitoring (Falco)
  • [ ] Regular security updates and patching

Kubernetes Phase

  • [ ] Enforce Pod Security Standards
  • [ ] Implement RBAC with least privilege
  • [ ] Encrypt secrets at rest
  • [ ] Use admission controllers
  • [ ] Enable audit logging
  • [ ] Regular security updates
  • [ ] Network policies for pod isolation
  • [ ] Use external secrets management

Conclusion

Container security is not a one-time task but an ongoing process that spans the entire software development lifecycle. By implementing security best practices from development through production, you can significantly reduce your attack surface and protect your infrastructure.

Key takeaways:

  1. Shift Security Left: Catch vulnerabilities early in development
  2. Defense in Depth: Layer multiple security controls
  3. Principle of Least Privilege: Grant minimal necessary permissions
  4. Automate Security: Make security checks part of CI/CD
  5. Monitor Continuously: Detect and respond to threats in real-time
  6. Stay Updated: Regularly patch and update dependencies

Start with the basics—minimal images, non-root users, and vulnerability scanning—then progressively add advanced controls like admission controllers, runtime security monitoring, and comprehensive audit logging.

Need help implementing container security? InstaDevOps provides expert consulting and implementation for container security, Kubernetes hardening, and compliance. Contact us for a free consultation.

Need Help with Your DevOps Infrastructure?

At InstaDevOps, we specialize in helping startups and scale-ups build production-ready infrastructure without the overhead of a full-time DevOps team.

Our Services:

  • 🏗️ AWS Consulting - Cloud architecture, cost optimization, and migration
  • ☸️ Kubernetes Management - Production-ready clusters and orchestration
  • 🚀 CI/CD Pipelines - Automated deployment pipelines that just work
  • 📊 Monitoring & Observability - See what's happening in your infrastructure

Special Offer: Get a free DevOps audit - 50+ point checklist covering security, performance, and cost optimization.

📅 Book a Free 15-Min Consultation

Originally published at instadevops.com

Top comments (0)