Getting MEAN and Lean with MongoDB and Kubernetes MongoDB World 2016 Vadim Polyakov
Golden Age of Software Development {  } Developer :  ) User DeploymentTesting www. Native
About Me Inovalon - R&D, Enterprise Architecture, Software Dev, Healthcare AOL Advertising - High volume transactions, Big Data 3 Startups – Healthcare and Financial Services Electrical Engineer – Control Systems FIRST Robotics Mentor https://www.linkedin.com/in/vadimpolyakov
Impedance mismatch metaphor “Lossless” transmission line Impedance mismatch
Database impedance mismatch Existing concept: Database or Object-relational impedance mismatch ORM Tools/code (Hibernate, Entity Framework) Database technology •  Object Databases •  Document Databases Application and database schema are unified
Stack Impedance Client HTML,  CSS,  JS Server .Net,  Java Database SQL Client Mobile HTML,  CSS,  JS
Process impedance mismatch Dev QA UAT Prod Dev QA UAT Prod
Really tie the stack together Tie together a “thin line architecture” of tools that support agility and developer happiness. Solve: •  Stack Impedance •  Process impedance
9 The stack
Browser (HTML,  JS,  CSS) Server REST  API Data MEAN - Single Page App Routable Components [Service, controller, directive] Routable Components [Middleware] JSON Flexible/Composable Flexible schema Collections JavaScript/TypeScript
JSON and JS Pipeline URL View – HTML template Controller – Manage the view Service – Manage the model URL Business Logic Injectable Data Management MongoDB
Patterns and antipatterns Over-normalizing Chatty services Over-embedding Latency MongoDB Collection Project Extract :) MongoDB Business  and   Presentation   Logic :) Collection Collection Just right MongoDB Collection Collection Collection Collection Collection Join Aggregate :)
Beyond development Dev QA UAT Prod
Docker “Docker containers wrap up a piece of software ... This guarantees that it will always run the same, regardless of the environment it is running in.” (https://www.docker.com/what-docker) Dev QA UAT ProdDev QA UAT Prod
Docker Managed as code – Dockerfile Simple and fast to build images Git-like semantics (push, pull, diff…) Repository of images (public, private) Management API Starts in milliseconds (it’s a process)
Application – multiple containers and nodes nginx nginx content  volume HTML,  JS,  CSSnginx Nginx nginx API Node /data/db volumenginx Data MongoDB Git  Sync nginx nginx content  volume HTML,  JS,  CSSnginx Nginx nginx API Node /data/db volumenginx Data MongoDB Cluster   Config Kubernetes Git  Sync
Kubernetes Core elements •  Nodes •  Pods •  Replication Controllers •  Services Examples: •  Deploy a MEAN app •  MongoDB on Kubernetes
Kubernetes Components - Node Node - physical resources Registers with the master Resource capacity Labels -  tags used to filter the environment e.g. has kernel config for MongoDB
Kubernetes Components - Pod Pod -  One or more co-located components with shared context (cgroups, namespaces) MongoDB  Pod API  Pod Web  Pod nginx nginx content  volume HTML,  JS,  CSSnginx Nginx Git  Sync nginx API Node /data/db volumenginx Data MongoDB nginx nginx content  volume HTML,  JS,  CSSnginx Nginx Git  Sync nginx API Node /data/db volumenginx Data MongoDB
Kubernetes Components - Replication Controller Replication Controller -  Keep certain number of pod replicas running
Kubernetes Components - Services Services -  Set of pods -  Typically selected using labels -  Virtual IP External Services Headless Service
App deployment Deployment as code svc-www.json rc-www.json svc-app.json rc-app.json svc-arb.json svc-mongo-db1.json svc-mongo-db2.json pod-mongo-db1.json pod-mongo-db2.json rc-arb.json Web service and replication controller (2 pods) API service and replication controller (3 pods) MongoDB services MongoDB pods and arbiter replication controller
Application up kubectl create –f ./configdir Before the demo: •  Start cluster •  Assign Elastic IP to the cluster •  Update kubernetes security group to allow port 80 •  Label the nodes Demo: •  Deploy application •  Configure replica set •  Access the web app •  No ssh deployment After the demo: •  Teardown the cluster, run kube-down
Demo Deploy application to a new cluster Deployment: 1 min Demo Duration: 12 min
Demo Deploy application to a new cluster Deployment: 1 min Demo Duration: 12 min •  Recreate production environment for troubleshooting •  Create any environment for any purpose •  Blue-green deployment
MongoDB on Kubernetes “Datacenter as a Computer” Most nodes in the cluster are “cattle” not “pets” Some nodes are special Databases and DB servers are special •  Kernel config •  Data is an asset
MongoDB on Kubernetes Replication Controller? •  Restart automatically on failure •  Schedule on multiple nodes Naked Pods Services •  Discoverability •  Virtual IP •  DNS
MongoDB on Kubernetes: Demo MongoDB   Instance MongoDB   Instance MongoDB   Arbiter MongoDB   Instance MongoDB   Instance MongoDB   Instance MongoDB   Arbiter Replication  Controller MongoDB   Instance MongoDB   Instance MongoDB   Arbiter Replication  Controller MongoDB   Instance Replication  Controller MongoDB   Instance MongoDB   Arbiter Replication  Controller MongoDB   Instance Service MongoDB   Arbiter Replication  Controller MongoDB   Instance MongoDB   Instance ServiceService Service MongoDB   Arbiter Manual Intervention
Demo Recover from node failure •  Arbiter migration to healthy node •  Node recovery
Demo Recover from node failure •  Arbiter migration to healthy node •  Node recovery
Demo Recover from node failure •  Arbiter migration to healthy node •  Node recovery
Demo Recover from node failure •  Arbiter migration to healthy node •  Node recovery
Golden Age •  Aligned application stack – rapid development, eliminate boilerplate •  Tools like Docker and Kubernetes make deployments and rollbacks safe at any scale •  Manage all environments as code and reproduce them faithfully whenever I need them – remove process impedance mismatch for setup and troubleshooting {  } Developer :  ) User DeploymentTesting www. Native
Thank you https://www.linkedin.com/in/vadimpolyakov https://github.com/polyakov {  } Developer :  ) User DeploymentTesting www. Native

MongoDB World 2016: Get MEAN and Lean with MongoDB and Kubernetes

  • 1.
    Getting MEAN andLean with MongoDB and Kubernetes MongoDB World 2016 Vadim Polyakov
  • 2.
    Golden Age ofSoftware Development {  } Developer :  ) User DeploymentTesting www. Native
  • 3.
    About Me Inovalon -R&D, Enterprise Architecture, Software Dev, Healthcare AOL Advertising - High volume transactions, Big Data 3 Startups – Healthcare and Financial Services Electrical Engineer – Control Systems FIRST Robotics Mentor https://www.linkedin.com/in/vadimpolyakov
  • 4.
    Impedance mismatch metaphor “Lossless”transmission line Impedance mismatch
  • 5.
    Database impedance mismatch Existingconcept: Database or Object-relational impedance mismatch ORM Tools/code (Hibernate, Entity Framework) Database technology •  Object Databases •  Document Databases Application and database schema are unified
  • 6.
    Stack Impedance Client HTML,  CSS,  JS Server .Net,  Java Database SQL Client Mobile HTML,  CSS,  JS
  • 7.
    Process impedance mismatch DevQA UAT Prod Dev QA UAT Prod
  • 8.
    Really tie thestack together Tie together a “thin line architecture” of tools that support agility and developer happiness. Solve: •  Stack Impedance •  Process impedance
  • 9.
  • 10.
    Browser (HTML,  JS,  CSS) Server REST  API Data MEAN - Single Page App Routable Components [Service, controller, directive] Routable Components [Middleware] JSON Flexible/Composable Flexible schema Collections JavaScript/TypeScript
  • 11.
    JSON and JSPipeline URL View – HTML template Controller – Manage the view Service – Manage the model URL Business Logic Injectable Data Management MongoDB
  • 12.
    Patterns and antipatterns Over-normalizingChatty services Over-embedding Latency MongoDB Collection Project Extract :) MongoDB Business  and   Presentation   Logic :) Collection Collection Just right MongoDB Collection Collection Collection Collection Collection Join Aggregate :)
  • 13.
  • 14.
    Docker “Docker containers wrapup a piece of software ... This guarantees that it will always run the same, regardless of the environment it is running in.” (https://www.docker.com/what-docker) Dev QA UAT ProdDev QA UAT Prod
  • 15.
    Docker Managed as code– Dockerfile Simple and fast to build images Git-like semantics (push, pull, diff…) Repository of images (public, private) Management API Starts in milliseconds (it’s a process)
  • 16.
    Application – multiplecontainers and nodes nginx nginx content  volume HTML,  JS,  CSSnginx Nginx nginx API Node /data/db volumenginx Data MongoDB Git  Sync nginx nginx content  volume HTML,  JS,  CSSnginx Nginx nginx API Node /data/db volumenginx Data MongoDB Cluster   Config Kubernetes Git  Sync
  • 17.
    Kubernetes Core elements •  Nodes • Pods •  Replication Controllers •  Services Examples: •  Deploy a MEAN app •  MongoDB on Kubernetes
  • 18.
    Kubernetes Components -Node Node - physical resources Registers with the master Resource capacity Labels -  tags used to filter the environment e.g. has kernel config for MongoDB
  • 19.
    Kubernetes Components -Pod Pod -  One or more co-located components with shared context (cgroups, namespaces) MongoDB  Pod API  Pod Web  Pod nginx nginx content  volume HTML,  JS,  CSSnginx Nginx Git  Sync nginx API Node /data/db volumenginx Data MongoDB nginx nginx content  volume HTML,  JS,  CSSnginx Nginx Git  Sync nginx API Node /data/db volumenginx Data MongoDB
  • 20.
    Kubernetes Components -Replication Controller Replication Controller -  Keep certain number of pod replicas running
  • 21.
    Kubernetes Components -Services Services -  Set of pods -  Typically selected using labels -  Virtual IP External Services Headless Service
  • 22.
    App deployment Deploymentas code svc-www.json rc-www.json svc-app.json rc-app.json svc-arb.json svc-mongo-db1.json svc-mongo-db2.json pod-mongo-db1.json pod-mongo-db2.json rc-arb.json Web service and replication controller (2 pods) API service and replication controller (3 pods) MongoDB services MongoDB pods and arbiter replication controller
  • 23.
    Application up kubectl create–f ./configdir Before the demo: •  Start cluster •  Assign Elastic IP to the cluster •  Update kubernetes security group to allow port 80 •  Label the nodes Demo: •  Deploy application •  Configure replica set •  Access the web app •  No ssh deployment After the demo: •  Teardown the cluster, run kube-down
  • 24.
    Demo Deploy application to anew cluster Deployment: 1 min Demo Duration: 12 min
  • 25.
    Demo Deploy application to anew cluster Deployment: 1 min Demo Duration: 12 min •  Recreate production environment for troubleshooting •  Create any environment for any purpose •  Blue-green deployment
  • 26.
    MongoDB on Kubernetes “Datacenteras a Computer” Most nodes in the cluster are “cattle” not “pets” Some nodes are special Databases and DB servers are special •  Kernel config •  Data is an asset
  • 27.
    MongoDB on Kubernetes ReplicationController? •  Restart automatically on failure •  Schedule on multiple nodes Naked Pods Services •  Discoverability •  Virtual IP •  DNS
  • 28.
    MongoDB on Kubernetes:Demo MongoDB   Instance MongoDB   Instance MongoDB   Arbiter MongoDB   Instance MongoDB   Instance MongoDB   Instance MongoDB   Arbiter Replication  Controller MongoDB   Instance MongoDB   Instance MongoDB   Arbiter Replication  Controller MongoDB   Instance Replication  Controller MongoDB   Instance MongoDB   Arbiter Replication  Controller MongoDB   Instance Service MongoDB   Arbiter Replication  Controller MongoDB   Instance MongoDB   Instance ServiceService Service MongoDB   Arbiter Manual Intervention
  • 29.
    Demo Recover from node failure • Arbiter migration to healthy node •  Node recovery
  • 30.
    Demo Recover from node failure • Arbiter migration to healthy node •  Node recovery
  • 31.
    Demo Recover from node failure • Arbiter migration to healthy node •  Node recovery
  • 32.
    Demo Recover from node failure • Arbiter migration to healthy node •  Node recovery
  • 33.
    Golden Age •  Alignedapplication stack – rapid development, eliminate boilerplate •  Tools like Docker and Kubernetes make deployments and rollbacks safe at any scale •  Manage all environments as code and reproduce them faithfully whenever I need them – remove process impedance mismatch for setup and troubleshooting {  } Developer :  ) User DeploymentTesting www. Native
  • 34.