DevOps for Network Engineers
- 1. Stefan Vallin, Ph.D. Principal Engineer 29 June 2016 Use Things You Can Program, and Program the Things You Use DevOps for Networking
- 2. Walls of Confusion Newand Changed Requirements Users Product Mgr New/Changed App Code Configuration Change Application Developers Network Engineers Operations
- 3. What is DevOps? http://dev.mobify.com/blog/devops-101-best-practices/ “Thecool-kids way of stringing stuff together with shell scripts. It exists because we can now wrap things in shell scripts that we used to only dream of; the world is now programmable at a much larger scale, and we have many new tools and techniques for taking advantage of it.” Takes a new or enhanced feature all the way to production— everyone is adding value to the customer A method that stresses communication, collaboration, integration, automation, and cooperation across organizational units to deliver features quickly and efficiently. Based on lean and agile principles DevOps is the marrying of process, infrastructure, and product New and Changed Requirements Customers Users New/ChangedApp Code Configuration Change Application Developers Network Engineers Operations
- 4. Why Should YouCare? Better Customer Experience Faster Better Increased capacity to innovate Let a small team multiply their effectiveness Faster time to value Fewer failures Recover faster from failures More predictable behavior
- 5. Why Should YouCare? We want more change! Dev “ ” We want less change! Ops “ ” You can have both!
- 6. Network Automation Historically DevOps Tools:GIT, Python, NETCONF • An IT/OSS project • Waterfall • Few network domain-experts • Driven from the network • Agile • Designed and implemented by network engineers with a DevOps mentality
- 7.
- 8. • Sharing • Collaboration •Learning • Understanding other roles • Understanding business • No blame-culture People, Process, Culture Source: IBM: “DevOps for Dummies” People, Process Culture
- 9. Across Roles Product Owners DevOps Team AT&T Domain2.0:Product Owners Business People People, Process Culture Everyone need to change- -not just network engineers - Meet in same practices - Meet in same tools - Learn and educate - More hands-on
- 10. • Hypothesis-driven • Slicesthat can be implemented in a week • Teams have a good understanding of the flow of work from the business all the way through to customers • Visibility • Regularly seek customer feedback, and incorporate this feedback into the design of their products. Lean Product Management Source: Puppe Labs: “2016 State of DevOps Report”
- 11. DevOps Networking Skills Culture Networking Software Engineering Education? “ There remains much to do before this vision can be implemented, including pivots from networking craft to software engineering, and from carrier operations models to cloud “DevOps” models. ” AT&T Domain 2.0: People, Process Culture
- 12. • An increasedknowledge of other IT disciplines • More knowledge of the business • More understanding of applications • More emphasis on programming Changes in Network Professionals’ Skills Stallings: Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud People, Process Culture
- 13.
- 14. • Infrastructure ascode • Automation • Deploy with repeatable, reliable processes • Self-service environments • Automated environment for de-provisioning • Automated recovery roll-back and roll-forward • Hypothesis-driven development • Dev and ops team shall use same tools • Develop and test against production-like systems • Amplify feedback loops • Continuous integration • Automated testing • Continuous deployment • Availability monitoring • Monitor and validate operational quality • Release management • App performance monitoring • Load-testing and auto-scale Practices
- 15. Before After Before andAfter Tools: Ticketing, Excel, Device CLI Tools: GIT, Python, NETCONF • Operator role picks up ticket in ITSM tool • Implements the configuration change manually or pseudo- automatically Cut and paste snippets • Closes ticket • Users provision their network services directly • DevOps role implements automation • What do I do manually? • How can I automate? • How can I validate it? • How do I handle failures?
- 16. So DevOps isjust scripting? More than a shell script in: ~/my-cool-automation-scripts Distribution, Execution, Monitoring, Error-Recovery, Logging, Validation, Versioning, Feature Management, No-Hands
- 17. Open Source, OpenAPIs, OpenFlow, Open vSwitch, OpenDaylight, OpenConfig… • Things can be said about all of the above, but clear values coming out of it • Standard APIs like NETCONF • Scripting on the devices • Linux-based devices, the shell prompt • Networking devices are now open for programming Enabler: The Open Networking Trend ~ Use Things You Can Program, and Program the Things You Use ~ Stallings: Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud
- 18. Deep-Dive: DevOps is aboutManaging Change — What is Changing?
- 19. Separation of Concerns,Change life-cycles Run-time fine-grained configuration changes for individual services • Modify QoS policy • Add e-mail user • Modify firewall rule • Add backend to load-balancer Network Services VNFs/VMs Applications Platforms Backbone / DC Network Short Lifecycle Long Lifecycle Do Not Blur/Mix Make resources available to provide services • Install • Start • Stop • Upgrade • Provision L2 network in data center connecting VNFs
- 20. Nature of ConfigurationChanges • Static “Golden Configs” • Configuration templates with variable substitution • Configuration templates with loops, if-statements • Scripted configuration based on stateServices VNFs/VMs Applications Platforms Backbone / DC Network Short Lifecycle Long Lifecycle Do Not Blur/Mix Different requirements for tools
- 21. Comparison: Changes inDifferent Domains Examples Changes Change Rate Applications Latest code-base for web-shop In-house developed code. Software changes Daily IT-Platforms New patch on WordPress Server New Database Server External Software releases and patches. Weekly mysql config file Config file contents Network Functions New VPN Add leg to VPN Add new VAS to VPN Change QoS params for VPN Configuration changes Thousands per day New device OS version New and upgraded devices Quarterly
- 22.
- 23. Typical Tools Services Applications VNFs VMs Platforms Backbone / DCNetwork Short Lifecycle Long Lifecycle • Service Orchestrators • Scripts • Vendor tools and UIs • And the below • Deployment: • Chef, Puppet • Juju charms, Heat Templates, TOSCA • Code and static configs • GIT • Build & Test • Jenkins
- 24. Application Centric Stack Applications Stitched VMs •IT Tools like Chef, Puppet, TOSCA • IT Tools like Chef, Puppet, TOSCA • Fine-grained configuration tools for networking might be needed
- 25. Network Centric Stack Network Services Virtual Network VNFs •Tools like Chef, Puppet, TOSCA not optimal • Service-Orchestrators • IT Tools like Chef, Puppet, TOSCA • BUT, exposed networking • AND Fine-tuned for packet-forwarding
- 26.
- 27. • Winner inService Agility: • Business Innovation and Product Owners in close co-op with DevOps team • Data-Models and Running Code • Sign of lagging behind • Organization controlled by “Architects in PowerPoint” • OSS-projects • Internal “purchase-orders” Summary Start creating your DevOps culture now
- 28. Material and Reading •John Willis, ”The Convergence of DevOps” • http://itrevolution.com/the-convergence-of-devops/ • Kyle Young, Mobify: “DevOps 101: Best Practices for Optimizing and Automating Your Infrastructure" • http://dev.mobify.com/blog/devops-101-best-practices/ • David Tesar, “DevOps Practices” • http://www.itproguy.com/devops-practices/ • Adrian Cockcroft: “Ops, DevOps and PaaS (NoOps) at Netflix” • http://perfcap.blogspot.com/2012/03/ops-devops-and-noops-at-netflix.html • Ashton, Metzler & Associates: “The Changing Role of the IT & Network Professional” • http://www.ashtonmetzler.com/Quali%20Fifth%20Paper%20V2.0.pdf • Lori MacVittie:”Network Engineers: Don’t Fear the Code” • http://www.networkcomputing.com/careers/network-engineers-dont-fear-code/1347793692
- 29. Material and Reading •IBM: “DevOps for Dummies” • https://www.ibm.com/marketing/iwm/iwm/web/signup.do?source=swg-rtl-sd-wp&S_PKG=ov18162 • Barry O’Reilly: “How to Implement Hypothesis-Driven Development” • https://www.thoughtworks.com/insights/blog/how-implement-hypothesis-driven-development • Julien Stroheker: “DevOps: Where do I start ? Cheat Sheet” • https://blogs.technet.microsoft.com/juliens/2016/02/14/devops-where-do-i-start-cheat-sheet/ • Stefan Vallin and Caroline Chappel, Light Reading Upskill: “Virtualization, Automation” • http://www.lightreading.com/virtualization-automation/l/d-id/722320 • Stallings: Foundations of Modern Networking: SDN, NFV, QoE, IoT, and Cloud • Puppet, “State of DevOps Report” • https://puppet.com/resources/white-paper/2016-state-of-devops-report
Editor's Notes
- #3 There is a constant feed of new and changed requirements from customers and users. Normally the developers and network engineers are not in close contact with the end-users. In contrast they get new requirements thrown over the wall without interaction. In turn: - the developers develop new apps to fulfill the requirements and throw over the wall to operations team. - network engineers manually reconfigures the network and throw that over the wall to the operations team. These two walls create: - confusion: from requirements to implementation, from reconfigured network to operational issues - time lag: the hand-over process is time-consuming
- #21 It is very important to distinguish different lifecycles. Two important layers are illustrated in the figure. BOTTOM: The long-lived lifecycle with running VMs/VNFs and TOP: the short-lived providing end-user services: telco:services, IT:applications Walk through the example bullets. Talk bottom-up Design separately, maybe different tools (more on this soon). BUT, they act together in a client-server relationship. Needs clear API linkage between the layers. The top-layer requests resources from the bottom layer. The top-layer does run-time config of the bottom layer. The bottom layer notifies the top layer on changes. The top layer need to adopt.
- #22 It is very important to distinguish different lifecycles. Two important layers are illustrated in the figure. BOTTOM: The long-lived lifecycle with running VMs/VNFs and TOP: the short-lived providing end-user services: telco:services, IT:applications Walk through the example bullets. Talk bottom-up Design separately, maybe different tools (more on this soon). BUT, they act together in a client-server relationship. Needs clear API linkage between the layers. The top-layer requests resources from the bottom layer. The top-layer does run-time config of the bottom layer. The bottom layer notifies the top layer on changes. The top layer need to adopt.
- #25 Look at the tools to the right. There are more well-known tools for the bottom layer. This also means that there is a risk that these tools are applied in the top-layer since they are known. Top-layer: APIs and fine-grained operations Bottom-layer: more templating Also, the “services” to use an over-loaded word in the top-layer varies much more than in the bottom-layer. A business VPN is very different between service providers, whereas starting a MYSQL server is more main-stream. Tools in the top-layer also depends heavily on if you are an IT Application-centric shop or a network service provider. For the first category the tools in the bottom layer might be ok.
- #26 If you are selling applications, the tools for long-lived life-cycles are appropriate also in the top-layer. The networking can be abstracted in the bottom layer. VIMs like OpenStack will hide/hard-code networking. Same with the tuning for the VMs, less effort in tuning the VIM for different VMs. I am not saying that IT is easy. Main challenge: Deployment of new application code several times per day. Needs extreme control on DevOps process for changing the application code itself. If you have internal control this is of course easier. External dev team, puts extreme focus on regression tests and deployment process.
- #27 For a network centric offering: Only the underlay network is performed in the bottom layer The top layer does all overlay networking The network layer can not be abstracted by for example OpenStack and VMWare or in the automation tools like Juju, HEAT,… More effort in tuning the virtual infrastructure VIMs for packet processing applications In the previous slide we talked about application centric offerings. There we said that a major challenge is that the application code might change several times per day. For a network centric offering it is the configuration that changes extremely frequently, thousands per day. the application code is changed with very low frequency. Since the top-layer is about high-frequency fine-grained configuration changes for varying offerings we need: Explicit data-models for the offerings APIs that support fine-grained operations
- #28 Process: Learning Sharing Collaboration Understanding business, other roles Practices: Automation Lean and agile Infrastructure as code Self-Service Automated rollback and recovery Tools: Share tools across organisation Configuration changes of different nature, different tools Pick-up tools from distributed programming, tools act upon text, not vendor blobs