Download to read offline
Uploaded byArun Kumar
Cloudexpowest opensourcecloudcomputing-1by arun kumar
This document provides an overview of open source cloud computing presented by Mark R. Hinkle. It discusses key cloud concepts like virtualization formats, hypervisors, compute clouds, storage, platforms as a service, APIs, private cloud architecture, provisioning tools, configuration management, monitoring, and automation/orchestration tools. The presentation aims to educate about building clouds with open source software and managing them using open source management tools. Contact information is provided for Mark R. Hinkle for any additional questions.
Cloudexpowest opensourcecloudcomputing-1by arun kumar
- 1. Mark R. Hinkle Director,Cloud Computing Community Citrix Systems Inc. Twitter: @mrhinkle Email: mrhinkle@cloudstack.org Crash Course in Open Source Cloud Computing Cloud Computing Expo West 2011 Santa Clara, CA
- 2. 2 Agenda 1. Introduction 2. QuickCloud Computing Overview 3. Open Source Building Blocks for Cloud Computing 4. Open Source Tools for Cloud Management 5. Questions
- 3. 3 %whoami • Responsible forDriving Adoption of CloudStack Open Source Cloud Computing Software • Joined Citrix via Cloud.com acquisition July 2011 • Former manager of Zenoss Open Source project 100,000 users, 1.5 million downloads • Former Linux Desktop Advocate (Zealot?) • Former LinuxWorld Magazine Editor-in-Chief • Open Management Consortium organizer • Author - “Windows to Linux Business Desktop Migration” – Thomson • NetDirector Project - Open Source Configuration Management Project • Sometimes Author and Blogger at SocializedSoftware.com/NetworkWorld
- 4. Quick Cloud Computing Overview:Or the Obligatory “What is the Cloud?” Slides
- 5. 5 Five Characteristics ofClouds 1. On-Demand Self-Service 2. Broad Network Access 3. Resource Pooling 4. Rapid Elasticity 5. Measured Service
- 6. 6 Cloud Computing ServiceModels USER CLOUD a.k.a. SOFTWARE AS A SERVICE Single application, multi-tenancy, network-based, one-to-many delivery of applications, all users have same access to features. Examples: Salesforce.com, Google Docs, Red Hat Network/RHEL DEVELOPMENT CLOUD a.k.a. PLATFORM-AS-A-SERVICE Application developer model, Application deployed to an elastic service that autoscales, low administrative overhead. No concept of virtual machines or operating system. Code it and deploy it. Examples: Google AppEngine, Windows Azure, Rackspace Site, Red Hat Makara SYSTEMS CLOUD a.k.a INFRASTRUCTURE-AS-A-SERVICE Servers and storage are made available in a scalable way over a network. Examples: EC2,Rackspace CloudFiles, OpenStack, CloudStack, Eucalyptus, Ubuntu Enterprise Cloud, OpenNebula
- 7. 7 Deployment Models Public, Private& Hybrid Clouds
- 8. 8 Cloud Still Requires ArchitecturalDesign • Cloud Computing isn’t a magical solution apps need to be able to scale out • Design your architecture with the end in mind • Make your infrastructure easily replicable
- 9. Building Compute Clouds withOpen Source Software
- 10. 10 Why Open Source? •User-Driven Solutions to Real Problems • Lower barrier to participation • Larger user base, users helping users • Aggressive release cycles stay current with the state- of-the-art • Open data, Open standards, Open APIs
- 11. 11 Open Virtual MachineFormats Open Virtualization Format (OVF) is an open standard for packaging and distributing virtual appliances or more generally software to be run in virtual machines. Formats for hypervisors/cloud technologies: • Amazon - AMI • KVM – QCOW2 • VMware – VMDK • Xen – IMG • VHD – Virtual Hard Disk - Hyper-V
- 12. 12 Sourcing OSS VMs andCloud Appliances
- 13. 13 Open Source Hypervisors OpenSource • Xen, Xen Cloud Platform (XCP) • KVM – Kernel-based Virtualization • VirtualBox* - Oracle supported Virtualization Solutions • OpenVZ* - Container-based, Similar to Solaris Containers or BSD Zones • LXC – User Space chrooted installs Proprietary • VMware • Citrix Xenserver • Microsoft Hyper-V • OracleVM (Based on OS Xen)
- 14. 14 Open Source ComputeClouds Year Started License Hypervisors Supported 2008 GPL Xenserver, Xen Cloud Platform, KVM, VMware 2006 GPL Xen, KVM, VMware 2010 (Developed by NASA by Anso Labs previously) Apache VMware ESX and ESXi, Microsoft Hyper- V, Xen, KVM and Virtual Box Other open source compute software include Abiquo, Red Hat’s CloudForms and OpenNebula Numerous companies are building cloud software on OpenStack including Nebula, Piston Inc., CloudScaling
- 15. 15 Scale-Up or Scale-Out •Vertical Scaling (Scale-Up) – Allocate additional resources to VMs, requires a reboot, no need for distributed app logic, single-point of OS failure • Horizontal Scaling (Scale-Out) – Application needs logic to work in distributed fashion (e.g. HA-Proxy and Apache, Hadoop)
- 16. 16 Open Source Platform-as-a-Service Year StartedSponsors Hypervisors Supported 2011 VMware Spring, Rails, Node.js 2011 Joyent Node.js 2011 Red Hat Express – Ruby, Php and Python Flex – Jboss, Java EE6 2010 WSO2 Jboss, Java EE6
- 17. 17 Open Source Cloud ComputingStorage • GlusterFS – Scale Out NAS system aggregating storage over Ethernet or Infiniband • Ceph – Distributed file storage system developed by DreamHost • OpenStack Object Storage (SWIFT) – Long-term storage object storage system • Sheepdog – Distributed storage for KVM hypervisors • NFS – Old standby, tried and true, not designed for cloud scale or performance
- 18. 18 Cloud APIs Aren’tCreated Equal Open Source Abstractions • jclouds • libcloud • deltacloud • fog
- 19.
- 20. Managing Clouds with OpenSource Tools
- 21. 21 Automation Unlocks the Potentialof the Cloud • MeatCloud, Can’t Keep up with Cloud Computing • Devops & Agile IT Philosophy • Script Repetitive Tasks • Automate, Automate, Automate
- 22. 22 The Myth ofthe Nines Availability % Downtime per Year Downtime per Month Downtime per Week 99.9% (three nines) 8.76 hours 43.2 minutes 10.1 minutes 99.95% 4.38 hours 21.56 minutes 5.04 minutes 99.99% (four nines) 52.6 minutes 4.32 minutes 1.01 minutes 99.999% (five nines) 5.26 minutes 25.9 seconds 6.05 seconds 99.9999% (six nines) 31.5 seconds 2.59 seconds .0605 seconds Average polling interval for monitoring - 5 minutes Even superhuman operations people can’t be alerted and take action in under 5 minutes. One outage per year could drop service level to three nines or worse.
- 23. 23 4 Types ofManagement Tools Provisioning Installation of operating systems and other software Configuration Management Sets the parameters for servers, can specify installation parameters Orchestration/Automation Automate tasks across systems Monitoring Records errors and health of IT infrastructure
- 24. 24 Management Toolchains Configuration Patching and Provisioning Monitoring Toolchain (n): Aset of tools where the output of one tool becomes the input of another tool
- 25. 25 Open Source Provisioning Tools YearStarted License Installation Targets Kickstart ? GPL Most .dep and RPM based Linux distros Cobbler (Plus koan for PXE boot of VMs) 2007 GPL Red Hat, OpenSUSE Fedora, Debian, Ubuntu Spacewalk 2008 GPL Fedora, Centos Crowbar 2011 Apache (Bare metal provisioning)
- 26. 26 Open Source Configuration ManagementTools Year Started Language License Client/Server Cfengine 1993 C Apache Yes Chef 2009 Ruby Apache Chef Solo – No Chef Server - Yes Puppet 2004 Ruby GPL yes
- 27. 27 Open Source Monitoring Tools LicenseType of Monitoring Collection Methods Cacti / RRDTool GPL Performance SNMP, syslog Nagios GPL Availability SNMP,TCP, ICMP, IPMI, syslog Zabbix GPL Availability/ Performance and more SNMP, TCP/ICMP, IPMI, Synthetic Transactions Zenoss GPL Availability, Performance, Event Management SNMP, ICMP, SSH, syslog, WMI
- 28. 28 Open Source Automation/Orchestration Tools Year Started LanguageLicense Client/Serv er Support Organizatio n Capistrano 2006 Ruby MIT Yes None RunDeck 2010 Java Apache Yes DTO Solutions Func 2007 Python GPL Yes Fedora Project MCollectiv e 2009 Ruby Apache Yes PuppetLabs
- 29.
- 30. Slides Can beViewed and Downloaded at: http://www.slideshare.net/socializedsoftware/ Questions?
- 31. 31 Contact Me Professional: mrhinkle@cloustack.org Personal: mrhinkle@gmail.com Professional:919.228.8049 Professional: http://www.cloudstack.org Personal: http://www.socializedsoftware.com Twitter: @mrhinkle Mark R. Hinkle Director, Cloud Computing Community Citrix Systems Inc. Open Source Enthusiast
- 32. 32 Additional Resources • DevopsToolchains Group • DevOps Wikipedia Page • Open Cloud Initiative • NIST Cloud Computing Platform • Open Virtualization Format Specs • Clouderati Twitter Account • Planet DevOps
- 33. Crash Course inOpen Source Cloud Computing by Mark R. Hinkle is licensed under a Creative Commons Attribution- ShareAlike 3.0 United States License.
Editor's Notes
- #6 From the NIST Cloud Computing On-demand self-service. A consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with each service’s provider. Broad network access. Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs). Resource pooling. The provider’s computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the customer generally has no control or knowledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter). Examples of resources include storage, processing, memory, network bandwidth, and virtual machines. This is different than virtual private hosting which is constrained to a single host or hosted Exchange server with fixed storage limits. Rapid elasticity. Capabilities can be rapidly and elastically provisioned, in some cases automatically, to quickly scale out, and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time. Measured Service. Cloud systems automatically control and optimize resource use by leveraging a metering capability1 at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency for both the provider and consumer of the utilized service.
- #7 Cloud Software as a Service (SaaS) – The Application Cloud The capability provided to the consumer is to use the provider’s applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings. Cloud Platform as a Service (PaaS) – The Development Cloud The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations. Cloud Infrastructure as a Service (IaaS). – Systems Cloud The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).
- #8 Private cloud The cloud infrastructure is operated solely for an organization. It may be managed by the organization or a third party and may exist on premise or off premise. Public cloud The cloud infrastructure is made available to the general public or a large industry group and is owned by an organization selling cloud services. Hybrid cloud The cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load balancing between clouds).
- #12 An OVF package consists of several files, placed in one directory. A one-file alternative is the OVA package, which is a TAR file with the OVF directory inside. OVF is a packaging format for software appliances. From a technical point of view, an OVF is a transport mechanism for virtual machine templates. One OVF may contain a single VM, or many VMs (it is left to the software appliance developer to decide which arrangement best suits their application). OVFs must be installed before they can be run; a particular virtualization platform may run the VM from the OVF, but this is not required. If this is done, the OVF itself can no longer be viewed as a “golden image” version of the appliance, since run-time state for the virtual machine(s) will pervade the OVF. Moreover the digital signature that allows the platform to check the integrity of the OVF will be invalid. VHD – Virtual Hard Disk format started by Connectix (now part of Microsoft) made open through the Microsoft Open Specification Promise.
- #13 Software appliances are like toasters, they do one thing very well. Bitnami BitNami Cloud Images allow BitNami Stacks to run in a cloud computing environment. BitNami offers Amazon Machine Images (AMIs) for running BitNami Stacks on the Amazon Cloud, as well as BitNami Cloud Hosting, a service that simplifies the process of running open source applications on Amazon EC2. BoxGrinder BoxGrinder supports many virtualization and Cloud platforms like EC2, Xen, KVM, VMware. You can create an appliance based on Fedora, Red Hat Enterprise Linux or CentOS. You are of course free to write your own plugin to support any other virtualization platform or operating system. SUSE Studio SuSE Studio allows you to use a hosted build service and a on premise virtual build system. Has a RESTful API to make calls to SUSE Studio openSUSE, SUSE Enterprise Linux (SuSE) and JeOS Integrates with SUSE Lifecycle Management Server and WebYAST Can Share Images in the SUSE Studio Gallery
- #14 Top choices for Cloud Computing are Xen and KVM. OpenVZ, container virtualization for Linux, is an interesting option as it has a very minimal overhead to scale application space similar to containers like BSD Jails. Advantage is that memory allocation is soft and unutilized memory can be used by other applications.
- #15 CloudStack – www.cloudstack.org - CloudStack is a sponsored by Citrix systems released under GPLv3 that provides a highly capable IaaS solution for service providers and enterprises. Robust Web Interface Comprehensive API Secure-Single Sign-On Dynamic Workload Management Xenserver, Xen Cloud Platform, KVM, VMware, OracleVM support Secure AJAX Console for VMs Networking-as-a-Service (Create VLANs to segregate traffic) EC2 API Compatibility Usage Metering Eucalyptus – http://open.eucalyptus.com - IaaS platform originally targeted to provide migration path from Amazon EC2 to private cloud. Amazon AWS Interface Compatibility Supports Amazon AMI High Availability Network Management, Security Groups, Traffic Isolation Self Service S3 compatible Storage Bucket-Based Storage Xen and KVM Hypervisor Support (VMware in Enterprise Edition) User Group and Role-Based Management OpenStack – www.openstack.org - Sponsored by Rackspace, a hosting provider is made up by three primary projects. OpenStack Compute (Nova) – Nova is a cloud orchestration platform similar to Amazon EC2 Orchestration of popular hypervisors (Xen, Xenserver, KVM, Hyper-V, VMware, Linux Containers) Floating IP Addresses (keep IPs and DNS correct when restarting VMs) VNC proxy through the Web Apache 2.0 License Android/iOS Clients Block Storage Support (AOE, iSCSI, Sheepdog) OpenStack Storage (Swift) – Is a EBS style solution used for long term storage not real time. Swift is used creating redundant, scalable object storage using clusters of standardized servers to store petabytes of accessible data. Features: Store and Manage files Programmatically Create public and private folders Using Commodity Hardware Fault tolerant (Nodes/HDD) Scale-out, Scale-Up OpenStack Image Service(Glance) - OpenStack Image Service (code-named Glance) provides discovery, registration, and delivery services for virtual disk images. Features: Provides images-as-a-service Supports Raw, VHD, VDI, qcow2, VMDK, OVF Restful API Backend Options – Swift, Local, S3, HTTP Version Control and Logging OpenNebula – http://www.opennebula.org/ – Cloud Computing Toolkit Apache license
- #16 Scale Up Scale Out
- #17 CloudFoundry Cloud Foundry, a VMware-led project, for building a Platform as a Service (PaaS) offering. Cloud Foundry provides a platform for building, deploying, and running cloud apps using Spring for Java developers, Rails and Sinatra for Ruby developers, Node.js and other JVM frameworks including Grails. SmartOS Recent entry by Joyent, node.js PaaS. OpenShift A free Platform-as-a-Service that enables developers to deploy apps written in multiple frameworks and languages across clouds. Open source licensing is forthcoming. WSO2 Java PaaS.
- #18 GlusterFS is an open source scale-out NAS solution. The software is a powerful and flexible solution that simplifies the task of managing unstructured file data whether you have a few terabytes of storage or multiple petabytes. Ceph is a distributed network storage and file system designed to provide excellent performance, reliability, and scalability. Ceph is based on a reliable and scalable distributed object store, with a distributed metadata management cluster layered on top to provide a distributed file system with POSIX semantics. There are a variety of ways to interact with the system OpenStack Object Storage (code-named Swift) is open source software for creating redundant, scalable object storage using clusters of standardized servers to store petabytes of accessible data. It is not a file system or real-time data storage system, but rather a long-term storage system for a more permanent type of static data that can be retrieved, leveraged, and then updated if necessary. Primary examples of data that best fit this type of storage model are virtual machine images, photo storage, email storage and backup archiving. Having no central "brain" or master point of control provides greater scalability, redundancy and permanence. Sheepdog is a distributed storage system for QEMU/KVM. It provides highly available block level storage volumes that can be attached to QEMU/KVM virtual machines. Sheepdog scales to several hundreds nodes, and supports advanced volume management features such as snapshot, cloning, and thin provisioning.
- #19 Types of Tasks Accomplished by an API Provisioning (creating, re-creating, moving, or deleting components e.g. virtual machines, vlans) Configuration (assigning or changing attributes of the architecture such as security and network settings) Cloud Providers Jclouds – java API Abstraction Libcloud – started by CloudKick (now Rackspace) to abstract clouds, Apache incubator project Deltacloud – started by Red Hat to abstract clouds, Apache incubator project Fog - provider and abstraction level API across compute and storage, written in Ruby
- #20 Derived from the NIST Diagram
- #30 Automated Toolchain (For Linux guests) Bootstrapped image is launched fro a template in the cloud provider, then searches for the Cobbler server. Post Install from Cobbler kicks off Puppet with defined management class to configure server using roles After cobbler runs kicks off configuration management in Puppet. Then services can be started and stopped with RunDeck or post-install scripts Then RunDeck can insert new hosts in Zenoss or Nagios Finally as the network conditions change Zenoss can remediate via other tools based on situational awareness