Downloaded 19 times
Uploaded byMatthias Furrer
Introducing Apache Mesos
The document introduces Apache Mesos, a cluster management system that optimizes resource utilization across various frameworks and applications. It explains the architecture, key components, and benefits of Mesos, including high availability and scalability, as well as its support for containers like Docker. The document highlights various frameworks for service scheduling and management, as well as installation and configuration steps for using Mesos and related technologies.
Introducing Apache Mesos
- 1. BASLE BERN BRUGGDÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH Introducing Apache Mesos Matthias Furrer Bruno Fuss A modern cluster manager optimizing resource utilization
- 2. Agenda TechEvent March 2017- Introducing Apache Mesos2 03/2017 1. Introduction Architecture (components) Distributions Other Offerings 2. Frameworks Service Scheduling and Management Frameworks Big Data Frameworks 3. HA Components Zookeeper Marathon 3. Configuration Mesos Marathon Kafka 4. Live Demo Mesos Kafka 5. Summary Use Cases / Benefits
- 3. TechEvent March 2017- Introducing Apache Mesos3 03/2017 Introduction
- 4. Introducing Apache Mesos Asoftware layer to aggregate servers into one processing unit TechEvent March 2017 - Introducing Apache Mesos4 03/2017 Distributed Cluster Management Software System Resources (Memory, Storage) abstraction from Hardware Metascheduler („scheduler of schedulers“) Distributed Systems Kernel / Distributed Datacenter OS Supports Mesos and Docker Containers Open Source – Runs under Apache License 2.0 Cross Plattform – Written in C++ – Runs on Linux/Posix systems, Experimental for Windows
- 5. Architecture - Features TechEventMarch 2017 - Introducing Apache Mesos5 03/2017 Scalability – Scales up to over 50‘000 nodes Resource isolation – Linux cgroups or Docker containers Efficiency – Resource scheduling accross multiple frameworks/applications High availability – Uses Apache Zookeeper (distributed configuration and synchronization service) Monitoring Interface – Web UI
- 6. Components TechEvent March 2017- Introducing Apache Mesos6 03/2017 Master – Mediating agent resources Agents – Manage resources on individual nodes and executes tasks Frameworks Applications – Systems running on mesos consisting of a scheduler and an executor (e.g. Spark, Storm ,Cassandra) Chronos – Fault tolerant job scheduler, can handle dependencies - acts as a cron for mesos Marathon – Framework for running long-running services – init.d for mesos
- 7. Architecture – HighLevel Overview TechEvent March 2017 - Introducing Apache Mesos7 03/2017 Zookeeper ensures high availability of masters Master daemon manages agent (slave) daemons Mesos frameworks run tasks on agents (slaves) Master decides how many resources to offer to each framework according to a given organizational policy (e.g fair sharing or strict priority). Framework running on top of Mesos consists of two components: a scheduler that registers with the master to be offered resources, and an executor process that is launched on agent nodes to run the framework’s taskSource: https://mesos.apache.org/documentation/latest/architecture
- 8. Architecture – ResourceOffering TechEvent March 2017 - Introducing Apache Mesos8 03/2017 1. Agent 1 reports to the master that it has 4 CPUs and 4 GB of memory free. The master then invokes the allocation policy module 2. The master sends a resource offer describing what is available on agent 1 to framework 3. The framework’s scheduler replies to the master with information about two tasks to run on the agent 4. Master sends the tasks to the agent, which allocates appropriate resources to the framework’s executor, which launches the two tasks. Because CPU/RAM Resources are still unallocated, the allocation module may now offer them to framework 2. Source: https://mesos.apache.org/documentation/latest/architecture
- 9. Distributions– Mesosphere DC/OS TechEventMarch 2017 - Introducing Apache Mesos9 03/2017 Commercial version: Mesosphere Datacenter Operating System Open Source version: DC/OS available - derived from Mesosphere’s Datacenter Operating System Based on Mesos and Marathon Supports Docker as well as native Mesos containers Container orchestration capabilities for Docker and Mesos containers including automatic workload recovery, security, networking, service discovery “Push-button installation” of complex distributed systems (including HDFS, Apache Spark, Apache Kafka, Apache Cassandra and more) with DC/OS Universe Runs on bare-metal, virtual (vSphere or OpenStack) and cloud (AWS, Azure, GCE)
- 10. Other Offerings TechEvent March2017 - Introducing Apache Mesos10 03/2017 Kubernetes (Google / Cloud Native Computing Foundation CNCF) Docker Swarm (Docker, Inc.) Rkt fleet (CoreOS, Inc.) Shipyard (Shipyard Project) CloudSlang (Hewlett Packard Enterprise) Ranger (Ranger Labs) Azure Service Fabric (Microsoft)
- 11. TechEvent March 2017- Introducing Apache Mesos11 03/2017 Frameworks
- 12. Service Scheduling andManagement Frameworks TechEvent March 2017 - Introducing Apache Mesos12 03/2017 Marathon – Launch and monitor long-running applications. Controlling a high availability environment, checking the applications' health and restarting services if nodes go down Chronos – Distributed, fully fault-tolerant job scheduler. Can interact with systems (e.g. Hadoop) even if the Mesos worker machine does not have the system installed Apache Aurora – Manages long-running services, cron jobs, and ad-hoc jobs Consul – Service Discovery and orchestration
- 13. Big Data Frameworks TechEventMarch 2017 - Introducing Apache Mesos13 03/2017 Apache Hadoop – Distributed Data Processing, MapReduce and Distributed File System (HDFS) Apache Spark – Execution engine for large-scale data processing. Batch processing, iterative processing, near real-time processing, and stream processing. Caching data in memory with Resilient Distributed Datasets (RDD) Apache Storm – Real-time distributed stream event-processing engine Apache Cassandra – Open source scalable NoSQL database. Provides query language Cassandra Query Language or CQL to communicate with database
- 14. Big Data Frameworks(II) TechEvent March 2017 - Introducing Apache Mesos14 03/2017 Apache Kafka – distributed publish-subscribe messaging system designed for speed, scalability, reliability, and durability Apache Samza – Distributed stream processing framework. Using Kafka for messaging and Hadoop YARN for resource management Elasticsearch-Logstash-Kibana (ELK) stack – End-to-End log analytics solution: Elasticsearch for search capabilities, Logstash as a log management software and Kibana as the visualization layer Mesos API – Develop custom frameworks – schedulers and executors
- 15. TechEvent March 2017- Introducing Apache Mesos15 03/2017 High Availability Components
- 16. Apache Zookeeper TechEvent March2017 - Introducing Apache Mesos16 03/2017 ZooKeeper – is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. Independent Apache Project Characteristics – Central Key/Value Store – Containing additional Information like time stamps, versioning, … – Simple – Fast – Designed for High Availability
- 17. Apache Zookeeper TechEvent March2017 - Introducing Apache Mesos17 03/2017 High Availability – one Leader – two or more Followers – Quorum Follower
- 18. Apache Zookeeper TechEvent March2017 - Introducing Apache Mesos18 03/2017 Data Model – Hierarchical Namespace – Z-Nodes (Zookeeper Nodes) – Nodes identified by Path • Ex.: /app1/p_1 Ephemeral Nodes Leader Election Conditional updates and watches Atomic Transactions
- 19. Apache Zookeeper onMesos Zookeeper takes hand in everywhere Mesos needs to store and call up distributed dynamic information – Service Locations – Service States – Client Notifications – Leader election for Mesos Masters and Frameworks (e.g. Marathon, etc.) 03/2017 TechEvent March 2017 - Introducing Apache Mesos19
- 20. Marathon Marathon is aApache Mesos framework for container orchestration – designed for long running and stateful Apps (e.g. Database, App. Server, etc.) Marathon is a meta framework – used for managing other Mesos Frameworks (Chronos, Storm, …) Container Support – Mesos Containers (default) – Docker Constainers (must be configured) 03/2017 TechEvent March 2017 - Introducing Apache Mesos20
- 21. Marathon Features – High Availability –Constraints – Service Discovery & Load Balancing (Mesos-DNS, Marathon-LB) – Health Checking – Event Subscription – User Interfaces for managing services (start, stop, check, …) • Marathon UI (GUI) • REST API – Basic Security (Authorisation, SSL, …) 03/2017 TechEvent March 2017 - Introducing Apache Mesos21
- 22. Marathon Example Mesos Masterand Agents (Slaves), Zookeeper and Marathon started by OS on each server (initd, systemd) Marathon starts frameworks on Mesos Agents (Chronos, JBoss, Jetty, Rails) Chronos, a fault tolerant job scheduler acts itself as framework and starts jobs (e.g. dumps a database, send Emails) 03/2017 TechEvent March 2017 - Introducing Apache Mesos22
- 23. TechEvent March 2017- Introducing Apache Mesos23 03/2017 Installation & Configuration
- 24. Installing and ConfiguringApache Mesos Installation – http://mesos.apache.org/gettingstarted – Different between CentOS 6 / 7 • Different Requirements (OS Packages, Libraries, …) – configure / make / make install – Zookeeper included 03/2017 TechEvent March 2017 - Introducing Apache Mesos24
- 25. Installing and ConfiguringApache Mesos Configuration – http://mesos.apache.org/documentation/latest/configuration – Provide parameters by config file or as startup parameter – Required parameters (M: on Master, A: on Agent) --zk="zk://mesos1:2181,mesos2:2181,mesos3:2181/mesos" (M,A) --quorum=2 (M) --work_dir=<workdir_path> (M,A) --containerizers=mesos,docker (A) 03/2017 TechEvent March 2017 - Introducing Apache Mesos25
- 26. Installing and ConfiguringMarathon Requirements • Apache Mesos 1.1.1+ • Apache ZooKeeper • JDK 1.8+ • Default JVM set to Java 8 Installation – https://mesosphere.github.io/marathon/docs/ – Dowload/unpack Kit (tar) 03/2017 TechEvent March 2017 - Introducing Apache Mesos26
- 27. Installing and ConfiguringMarathon Configuration – Library: MESOS_NATIVE_JAVA_LIBRARY • searches the common installation paths, /usr/lib and /usr/local/lib, for the Mesos native library. If the library lives elsewhere in your configuration, set the environment variable MESOS_NATIVE_JAVA_LIBRARY to its full path. – Parameters: Marathon uses --master to find the Mesos masters, and --zk to find ZooKeepers for storing state --master="zk://mesos1:2181,mesos2:2181,mesos3:2181/mesos" --zk="zk://mesos1:2181,mesos2:2181,mesos3:2181/marathon" 03/2017 TechEvent March 2017 - Introducing Apache Mesos27
- 28. Installing and ConfiguringApache Kafka for Mesos TechEvent March 2017 - Introducing Apache Mesos28 03/2017 Installation • According to https://github.com/mesos/kafka • Make sure to retrieve the Kafka download from https://archive.apache.org/dist/kafka from a version containing a fitting Scala library (Version indicated on the Mesos page may not be correct) to avoid java.lang.NoSuchMethodError Configuration • Set Mesos connection info in kafka-mesos.properties • Start Scheduler on ONE Mesos agent (Marathon can be used to manage) • Add Broker(s) • Start Broker(s)
- 29. TechEvent March 2017- Introducing Apache Mesos29 03/2017 Demo
- 30. mesos1 mesos2 mesos3 mesos4mesosn … mesosn-1 TVD Mesos Platform / Server extendable • additional Servers (on-premise) • Cloud • … 03/2017 TechEvent March 2017 - Introducing Apache Mesos30 BF2
- 31. Slide 30 BF2 BrunoFuss, 3/8/2017
- 32. mesos1 mesos2 mesos3 mesos4mesosn … mesosn-1 Mesos Master Mesos Master Mesos Agent Mesos Agent TVD Mesos Platform / Mesos Components Mesos Agent Mesos Agent HA concept Master: active/passive 03/2017 TechEvent March 2017 - Introducing Apache Mesos31
- 33. mesos1 mesos2 mesos3 mesos4 Mesos Master Mesos Master Mesos Agent Mesos Agent Zoo keeper Zoo keeper Zoo keeper TVDMesos Platform / Zookeeper Components HA concept: active/active 1 leader 2-n followers Zookeeper quorum= 2 mesosn … mesosn-1 Mesos Agent Mesos Agent 03/2017 TechEvent March 2017 - Introducing Apache Mesos32 BF3
- 34. Slide 32 BF3 BrunoFuss, 3/8/2017
- 35. TechEvent March 2017- Introducing Apache Mesos33 03/2017 Summary
- 36. Use Cases /Why use Mesos TechEvent March 2017 - Introducing Apache Mesos34 03/2017 Enhance resilience against failure Need for application-specific scaling Increase resource utilization Wide range of workloads must be supported – e.g. Batch (Hadoop), Analysis (Spark), real-time (Storm), data storage (HDFS, Cassandra) Open source with vivid community Developer API – minimal considerations about infrastructure details
- 37. Matthias Furrer Principal Consultant,Application Integration & SOA matthias.furrer@trivadis.com 03/2017 TechEvent March 2017 - Introducing Apache Mesos35 Bruno Fuss Senior Consultant, Infrastructure Managed Services Bruno.fuss@trivadis.com
- 38. Session Feedback –now TechEvent March 2017 - Introducing Apache Mesos36 03/2017 Please use the Trivadis Events mobile app to give feedback on each session Use "My schedule" if you have registered for a session Otherwise use "Agenda" and the search function If the mobile app does not work (or if you have a Windows smartphone), use your smartphone browser – URL: http://trivadis.quickmobileplatform.eu/ – User name: <your_loginname> (such as “svv”) – Password: sent by e-mail...