Downloaded 37 times
Uploaded byKellyn Pot'Vin-Gorman
DevOps and DBA- Delphix
This document provides an overview of DevOps and how it relates to database administrators (DBAs). It discusses key DevOps concepts like continuous delivery, configuration management, and release coordination. Agile methodologies like Scrum, Kanban, and Extreme Programming are described. DevOps tools that can help DBAs are also covered, including virtualization platforms, containers, configuration management tools like Ansible, and the periodic table of DevOps tools. The document aims to explain how DevOps impacts and involves DBAs in its goal of faster, more reliable software delivery.
DevOps and DBA- Delphix
- 1. Kellyn Pot’Vin-Gorman |Technical Intelligence Manager for the Office of CTO DevOps and the Agile DBA
- 2. 2© 2017 DelphixCorporation Kellyn Pot’Vin-Gorman Technical Intelligence Manager, Delphix • Multi-platform DBA, (Oracle, MSSQL, MySQL, Sybase, Postgres…..) • Oracle ACE Director, (Alumni) • Oak Table Network • APEX Women in Technology Award, CTA 2014 • STEM education with Raspberry Pi and Python • President, Rocky Mtn Oracle User Group • Liaison, Denver SQL Server User Group • Author, blogger, (http://dbakevlar.com)
- 3. 3© 2017 DelphixCorporation https://en.wikipedia.org/wiki/DevOps DevOps DevOps is a short for DEVelopment and IT OPerationS. The term is used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably
- 4. 4© 2017 DelphixCorporation History of DevOps • First called “Agile Infrastructure” in 2008 • 2009, the first time DevOps was coined in Belgium • Has taken over most of the fellow terms of WinOps and in some cases, Agile. • Introduced at relational database events in 2010, but rarely was given much credit until the last five years in the DBA community. https://www.versionone.com/devops-101/what-is-devops/
- 5. 5© 2017 DelphixCorporation DevOps Relationship to Agile
- 6. 6© 2017 DelphixCorporation Don’t Confuse the Two… • Agile’s success pushed the need for DevOps • Agile and DevOps have similarities, but Agile focuses more on culture and DevOps focuses on organizational changes to create success. • DevOps uses the success of release implementation as a scoring method.
- 7. 7© 2017 DelphixCorporation How the DBA is Impacted by DevOps • Database Administrators heavily influence decisions. • Adverse to bleeding edge, high risk. • Introduction of the cloud, especially and SaaS, (Software as a Service). • Demand for decreased development cycles introduce risk, DBAs are seen as a roadblock to this. Still, DBA skills are in high demand towards DevOps initiatives…
- 8. 8© 2017 DelphixCorporation Common Terms, (Alphabetically) • Agile • Build Automation • Configuration Management • Continuous Delivery/Rapid Deploy • Functional Testing/Unit Testing/Test Driven Development • Lean/Scrum • Monitoring/Alerting • Packages/Containers/Virtualization • Release Coordination/Orchestration https://xebialabs.com/glossary/
- 9. 9© 2017 DelphixCorporation Build Automation DBAs are a natural fit, due to their history with scripting and automation experience for backups, monitoring and alerting.
- 10. 10© 2017 DelphixCorporation Configuration Management Concept & Development Testing & Verification Operations & Support Deployment to Production
- 11. 11© 2017 DelphixCorporation Continuous Delivery DBAs are much slower to embrace continuous delivery. The risk is high, with it’s “fail first, fail fast” approach. https://continuousdelivery.com/
- 12. 12© 2017 DelphixCorporation Rapid Deploy Software Content Enablement Service Need or Focus Area All Tiers of Specific Environment
- 13. 13© 2017 DelphixCorporation Release Coordination/Orchestration A demand for release tools and orchestration was at times a direct result to DBAs hesitation to risk production with a release. This helped automate and bypass the DBA to get code and features into production.
- 14. 14© 2017 DelphixCorporation Source/Version Control, but Data? We all know about version control with code, but how are we, as DBAs doing version control with data? This is a concept that hasn’t been discussed much and DataOps is the new way of bridging some of this challenge.
- 15. 15© 2017 DelphixCorporation The Goals of Each are the Same • Shorter Delivery Cycles • Higher Success Rate on Releases • Shorten Time to Address Issues • Continued Improvement in Communication • Increased Revenue
- 16. 16© 2017 DelphixCorporation For the DBA, Why Virtualization A technical approach in which users and applications do not use physical machines, but simulated systems running on actual, “real” hardware. Virtualization can be used to eliminate resource usage and enable savings for databases, network, file and application management, along with server infrastructure.
- 17. 17© 2017 DelphixCorporation ▶▶▶ Virtualize and Deployed▶ ▶ ▶ 600GB Many Types of Virtualization Storage Pool for Delphix QA DEV PATCH TEST PRODUCTION Database/App Tier 1 TB 1 TB Read From Production Each Virtual Database takes up around 5-10Gb upon creation, (dependent upon parameters) TEST Read AND Write
- 18. 18© 2017 DelphixCorporation Why Database Virtualization is Different • DBA is “only as good as their last backup”. • Many database tools take considerable time to recover. • DevOps is often about automation- automate an “undo” for development and testing that includes data rewind. • Include a self-service tool that will allow for rewind without DBA intervention. • Allows for data version control and DataOps, the next step in DevOps
- 19. 19© 2017 DelphixCorporation Timeline in a Project Waterfall- 5 Releases January February March April Agile with Virtualization- 12 Releases January February March April Database Refresh Development Production Release Testing Cycle The Goal of Agile Data Holdup to Development Cycle
- 20. 20© 2017 DelphixCorporation Packages/Containers Containers offer the ability to isolate application code and/or the whole infrastructure stack into a package able entity to ease deployment, even from the same kernel. This is a powerful tool for DevOps to ease deployment for complex tiers, applications and multiple data stores.
- 21. 21© 2017 DelphixCorporation “Containerize” for Ease of Delivery Create “Container” Rsync (UNIX/Linux) Robocopy (Windows) Reporting NFS iSCSI Development NFS iSCSI Testing NFS iSCSI Delphix Virtualization Engine Same Storage Size as Production Deployed using Jenkins
- 22. 22© 2017 DelphixCorporation Branching and Bookmarking • The ability to mark each iteration of development with a bookmark • Simplify to lock and deliver while testing a consistent image via a virtual database, (VDB) • If a something goes wrong, the ability to “bookmark”, (and subsequent snapshot) to deliver to development to address.
- 23.
- 24. 24© 2017 DelphixCorporation DevOps and Source Control
- 25. 25© 2017 DelphixCorporation Maturity
- 26. 26© 2017 DelphixCorporation Collaboration Practices/Tools • Methods- Scrum, KanBan, Extreme Programming, Continuous Delivery • Hybrid approaches combine, such as Scrumban and KanPlan • Collaboration- Jira, SlackStorm, Pivotal Tracker, HipChat, Rally, (now Agile Central) • Deployment- RapidDeploy, ElasticBox, JuJu, Otto • Automation- Jenkins, Chef, Ansible • Build Tools- Maven, Gradle, Make, Packer, MSBuild
- 27. 27© 2017 DelphixCorporation Agile Methodologies Scrum- A lightweight PM framework. Benefits include a flexible set of management and project protocols and strong collaboration focus. Lean/Kanban- Strong focus on team and deliverables. Efficiency and ability to see the “whole”. Extreme Programming, (XP)- Now the most popular, with continuous delivery key. Crystal- A lightweight agile framework. A removal of hurdles, including bureaucracy, obstacles and other impediments to the development cycle. Feature Driven Development, (FDD)- client and architecture centric software development, focused on the feature deliverable.
- 28. 28© 2017 DelphixCorporation What is Scrum? An Agile Framework Scrum Framework consists of: • A Product owner creating a wish list and the “sprint” begins • Sprint planning and backlog is created. • Team sets up schedule and beings to have daily scrum standups, (commonly 5 minutes) • Scrum master keeps team focused and collaborating, keeps track of status • Product is released • Sprint ends with feedback and lessons learned • Next sprint begins
- 29. 29© 2017 DelphixCorporation Example of a Scrum Sprint https://www.scrumalliance.org/community/articles/2014/april/devops-and-agile
- 30. 30© 2017 DelphixCorporation Lean Kanban Considered a very “visual” development process, based off of grocery store shelf stocking. • Uses standardized cues and refined processes • Goal to reduce waste and maximize value • Most often uses sticky notes and whiteboard to create a picture of the work to complete, what’s in process and what’s done. 1. Visualize Work 2. Limit Work in Process 3. Focus on Flow 4. Continuous Improvement
- 31. 31© 2017 DelphixCorporation Picture is Worth a 1000 Words
- 32. 32© 2017 DelphixCorporation Extreme Programming
- 33. 33© 2017 DelphixCorporation This Extreme Programming, (XP) • Code may come first in XP, but testing must already exist to know what the successful outcome will be. • Code is written by pairs of programmers, allowing for better collaboration. • Believes in the power of doing, vs. extensive planning. Failure is expected. • Always build foundations that can be built on later. • Rarely specialize- everyone develops, tests, designs, etc.
- 34. 34© 2017 DelphixCorporation XP at a Glance
- 35. 35© 2017 DelphixCorporation Crystal Alistair Cockburn said, “Crystal is a family of human- powered, adaptive, ultra light, ‘stretch-to-fit’ software development methodologies.” • People are the most important aspect of Crystal. • Development is self-sufficient and self-organizing, (very different from Scrum) • Very lightweight • Very little documentation, overhead management or reporting.
- 36. 36© 2017 DelphixCorporation Feature Driven Development
- 37. 37© 2017 DelphixCorporation DevOps Tools for the DBA
- 38. 38© 2017 DelphixCorporation The Periodic Table of DevOps Tools https://xebialabs.com/periodic-table-of-devops-tools/
- 39. 39© 2017 DelphixCorporation Ansible • Learn about Playbooks • Powerful collection of steps, (aka plays) • Plays consist of a list of tasks • Handlers allow for remote operations • YAML Syntax • Security configurations based on hosts and users • Ability to run as secondary users, (privilege escalation) • Huge support community and best practices are mature
- 40. 40© 2017 DelphixCorporation Example of an Ansible Call - name: Transfer and execute a script. hosts: server remote_user: test_user sudo: yes tasks: - name: Transfer the script copy: src=test.sh dest=/home/test_user mode=0777 - name: Execute the script command: sh /home/test_user/test.sh
- 41. 41© 2017 DelphixCorporation Jenkins Jenkins is an open source automation server- • Easiest executed via a jetty servlet • Has an easy to access URL, (commonly host:8080) • Automation is done via individual projects. • Has numerous plugins to support environments • Supports multiple scripting languages, including shell, batch commands and Ant scripts. • Supports robust revision control and compatible with version control systems like CVS and Subversion. • Supports notifications like qmail and sendmail.
- 42. 42© 2017 DelphixCorporation Jenkins Groovy Script Example // prepare variables for subscript execution: String subscript = "cdmserver-monitor-instance- status.groovy"; println "executing subscript ${subscript}" // variables to be passed to the subscript status = (action == "start" ? "started" : "stopped") def s = ScriptlerConfiguration.getConfiguration().getScriptById(su bscript) File scriptSrc = new File(ScriptlerManagment.getScriptDirectory(), s.getScriptPath());
- 43. 43© 2017 DelphixCorporation Jenkins Plugin for Delphix https://github.com/delphix/delphix-jenkins-plugin
- 44. 44© 2017 DelphixCorporation Puppet • Review, automate, deploy and invest in all your software. • Control and enforce consistency across environments and platforms. • Share, test and enforce changes on-premise and in the cloud. Similar to Puppet • Chef • Jenkins
- 45. 45© 2017 DelphixCorporation Docker • Virtualization is important, this is OS level • Resource level isolation at the kernel level • Technology such as OverlayFS, allows for containers, which removes overhead of other virtualization technologies. Docker Alternatives: • Open Container Initiative, (OCI) • Kubernetes, (Google) • CoreOS and rkt • Apache Mesos • Canonical and LXD
- 46. 46© 2017 DelphixCorporation GitHub • Git is an open-source version control system that was started by Linus Trovalds • If you aren’t using it now, start. Yes, as a DBA… • Great option to highlight your scripting work GitHub Alternatives • Subversion • CVS • Mercurial
- 47. 47© 2017 DelphixCorporation Creating a GitHub Repository Project
- 48. 48© 2017 DelphixCorporation Github Repository Example
- 49. 49© 2017 DelphixCorporation So, Know Your Tools Incredible variation of tools and knowing tool classifications doesn’t mean that the tools are alike. Many groups are often comparing one feature of a given tool.
- 50. 50© 2017 DelphixCorporation From the DBA Perspective DevOps include the following focus areas for Database Administrators: • Automating repeatable and complex processes • Ensuring the databases and data stores aren’t the bottleneck to the development cycle and critical data is protected. • Locating heterogeneous tools/scripting languages that can support multiple tiers of technology. • Understanding that automation does not lessen your value. • Embracing new skills to add more value to the business and to your career.
- 51. 51© 2017 DelphixCorporation Summary- DBA Focus Areas • Update skills, (scripting, monitoring, automating) to remove bottlenecks • Enhance monitoring utilities that take infrastructure and development into overall view • Recognize a need to become more skilled in scripting and become knowledgeable of DevOps tools. • Data is EVERYWHERE! Recognize the integral part Database Administrators play in DevOps and learn more about the next generation- DataOps! DBA DevOpsSkills
- 52. Kellyn Pot’Vin-Gorman Technical IntelligenceManager kellyn@delphix.com http://dbakevlar.com
Editor's Notes
- #4 DevOps derives from both development and operations, groups that DBAs often have a foot in each of. There is a high focus on collaboration, geared on methodologies, process and practice. The goal is to release more frequently, more successfully and with less bugs.
- #5 Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure” The term DevOps was popularized through a series of "devopsdays" starting in 2009 in Belgium
- #6 Agile and DevOps aren’t one in the same, but as it’s well known, DevOps came out of Agile’s success. Agile= culture, where DevOps focuses more on the organization changes.
- #10 Build automation is the process of automating the creation of a software build and the associated processes including: compiling computer source code into binary code, packaging binary code, and running automated tests.
- #11 Configuration management (CM) is a systems engineering process for establishing and maintaining consistency of a product's performance, functional, and physical attributes with its requirements, design, and operational information throughout its life. A DBA’s desire for low risk and stability assists here as we desire routine that results in expected outcomes.
- #12 Continuous delivery (CD) is a software engineering approach in which teams produce software ... incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.
- #13 This is another area that introduces risk, so DBAs can be very adverse to it, but the focus on a single feature, helps minimize the impact and often isolate any issues.
- #14 Release Orchestration is the use of tools like XLRelease which manage software releases from the development stage to the actual software release itself.
- #15 I’m going to add to this definition with Data version control.
- #17 In computing, virtualization means to create a virtual version of a device or resource, such as a server, storage device, network or even a database. The framework divides the resource into one or more execution environments. For data, this can result in a golden copy or source that is used for a centralized location and removal of duplicated data. For read and writes, having unique data for that given copy, while duplicates are kept to singular.
- #18 Point out the engine and size after we’ve compressed and de-duplicated. Note that each of the VDBs will take approximately 5-10G vs. 1TB to offer a FULL read/write copy of the production system It will do so in just a matter of minutes. That this can also be done for the application tier!
- #20 Over 80% of time is waiting for RDBMS, (relational databases) to be refreshed. Developers and Testers are waiting for data to do their primary functions. This allows for faster and less costly migrations to the cloud, too.
- #21 Package software into standardized units for development, shipment and deployment. A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.
- #22 We refer to a container as a template in our product.
- #23 This is a cornerstone to developers and testers, so as DBAs, we know the pain when a developer comes to us to flashback a database and before that, recover or logically recover, (import or datapump) independent objects. What is The developer/tester could do this for themselves?
- #24 This is the interface for Developers and testers- they can bookmark before important tasks or rewind to any point in the process. They can bookmark and branch for full development/testing needs.
- #25 This may appear to be a traffic disaster of changes, but for developers with Agile experience, a “sprint” looks just like this. You have different sprints that are quick runs and merges where developers are working separately on code that must merge successfully at the correct intersection and be deployed. Versioning with source control is displayed at the top, using Virtual images. You can see each iteration of the sprints. In the middle section is the branches of that occur during the development process. A virtual can be spun from a virtual, which means that it’s easier for developers to work from the work another developer has produced. Stopping points and release via a clone is simply minutes vs. hours or days.
- #26 The maturity of the DevOps environment will decide how silo’d or how blended the role you’ll have in DevOps vs. your standard role as a DBA.
- #27 Methods provide a format or guide to work from. Hybrid approaches often implement best. Collaboration methods ensure that communication continues when team members return to their desks Deployment tools help with documenting and lessons learned Build tools help with automation and orchestration
- #28 Scrum focuses on features, bug fixes and backlog debt. Serves very large teams, including those 800+ Lean’s goal is to eliminate all waste, over demand on resources and ability to deliver faster and more effectively each time. XP is one of the most controversial due to the ability to deliver even to large companies every couple minutes if required.. Very disciplined approach. Crystal is often known under Crystal Clear, Yellow Orange and others.
- #32 Often uses a whiteboard with sticky notes…
- #37 Like Rapid Deploy, more focused on delivering one feature as the product.
- #42 Ant is another java based built tool that’s part of Apache open-source project. Similar to Make and written in XML.
- #43 This Groovy script executes another script, making it valuable in environments that already have a number of mature scripts in place that should be reused in automation.