Download as PDF, PPTX
![Look at your purpose and those processes which aid it ● Make sure the process is correct from the beginning to end then look at ways to speed up that process ● Value Stream - the name of the process which provides value to the business ● Working from left to right e.g. like a timeline business / development => customer / operations ● Flow [rate] - speed work goes through the process. #1 Workflow](https://image.slidesharecdn.com/practicaldevsecopsmatt-tesauroonug-fall-2022-221030185302-0f8e02cc/75/Practical-DevSecOps-Fundamentals-of-Successful-Programs-12-2048.jpg)
Practical DevSecOps: Fundamentals of Successful Programs
- 1. Old West Edition Practical DevSecOps Fundamentalsof Successful Programs
- 2. HOWDY! ● Reformed programmer& AppSec Engineer ● Noname Security - Distinguished Engineer ● 14 years in the OWASP Community ○ OWASP DefectDojo (core maintainer) ○ OWASP AppSec Pipeline (co-leader) ○ OWASP WTE (leader) ● 22 + years using FLOSS & Linux ● Currently a Go language fanboy ● Ee Dan in Tang Soo Do Mi Guk Kwan (2nd degree black belt) ● Founder 10Security
- 3. Starting from Zero Steps 2Take Example of how to expand your horizon 1 Final Thought Conclusions and key takeaways How this whole thing got started 3 Ways of DevOps Crawl, walk run with DevSecOps TABLE OF CONTENTS 2 4 3 1
- 4. Starting from Zero 1 And maybeheading towards hero
- 5.
- 6. The Iron HorseStraddles America ● Radically changed travel in the US ● Travel time across the US ○ Pre-train: 6 months + $1,000 ○ Post-train: 1 week + $150 ● Town that had a stop prospered ○ Those that didn’t, faded away
- 7. Trains == Change ●Trains changed the landscape for better or worse ● The US ‘got smaller’ - travel was in reach to more people ● Expanded markets, more customers ● ‘Cost’ of going west went way down
- 8. Trains <==> DevSecOps ●Trains changed the landscape for better or worse DevSecOps changed IT for better or worse ● The US ‘got smaller’ - travel was in reach to more people Batch / change size got smaller (CICD) ● Expanded markets, more customers Increased agility, more customers ● ‘Cost’ of going west went way down ‘Cost’ of experiments goes way down
- 9. When will wesee this? DevSecOps AppSec Your bit of IT here
- 10. 3 Ways of DevOps 2 Beforethere was DevSecOps, there was DevOps
- 11. Workflow ? ? The 3 waysof DevOs 1 3 2
- 12. Look at yourpurpose and those processes which aid it ● Make sure the process is correct from the beginning to end then look at ways to speed up that process ● Value Stream - the name of the process which provides value to the business ● Working from left to right e.g. like a timeline business / development => customer / operations ● Flow [rate] - speed work goes through the process. #1 Workflow
- 13. Each Step Repeatable #1Workflow ● Remove all haphazard and ad hoc work from the process ● Repeat until stable ○ I like doing the first couple of times manually with a ‘run book’ ● Scripting languages are your friends ● Config Mgmt - Puppet, Chef, Salt, Ansible, Terraform, Helm, … ● Create deployable artifacts from a branch/release e.g. .rpm, .deb, .msi, … ● Make sure what you do can be done once or 10,000 times
- 14. Never Pass alongDefects #1 Workflow ● While working left to right, don’t pass on failures ● Test early and often ● Increase rigor of testing as you work left to right ● When a failure occurs, end that flow and start a new one (after corrections) ● The further right you are, the more expensive failure is ○ Concentrate your early work on the left side (intake) ● e.g. in AppSec, defects are false positive findings
- 15. Local Optimizations witha Global View #1 Workflow ● Ensure no single-step optimizations degrade the overall performance of the workflow ● Find the bottleneck in your workflow and start there ○ Upstream changes will just back things up ○ Downstream changes won’t manifest since input is constrained ● Each new optimization creates a new bottleneck ○ Iterate on these
- 16. Increase the flowof work #1 Workflow ● Make sure you have a well-defined, repeatable process first ● Look for manual steps that can be automated ● Look for duplicate work that can be removed or eliminated ● Measuring/tracking time taken at each step is crucial ● Ask yourself: Where does the flow ebb?
- 17. Workflow Improve Feedback ? The 3ways of DevOs 1 3 2
- 18. Open yourself toupstream and downstream information #2 Improve Feedback ● Feedback loops occur when information is gathered from ○ upstream (business / development) ○ downstream (customer / operations) ● Make visible problems, concerns, potential improvements ○ Share this broadly inside the company ● Learn as you move left to right so improvements aren’t lost ● Requests are opportunities to better fulfill business’s needs ○ There’s rarely enough feedback, capture and look for more ○ Feedback collected can be used to optimally improve the system
- 19. Understand and Respondto your Customers #2 Improve Feedback ● Customers are also inside your business ● Customer is more than the end ‘consumer’ at the end of the process ○ Each step is the customer of the previous step ○ Understand what the next steps need from you to succeed ● Remember, feedback isn’t guaranteed ○ Encourage it by responding ● Responses are required of external and internal customers ● Make feedback & responding quick, easy and readily available
- 20. Shorten Feedback Loops #2Improve Feedback ● Remove any intermediaries and impediments to feedback ● Communicate as directly as possible, skipping steps/people if possible ○ e.g . The person who finds a problem communicates with the person who fixes the problem ● The more hands that hold the feedback, the more change to get garbled ● If possible, intermediaries should be software, not people ● Whispered secret across a classroom ○ How much change occurs?
- 21. Amplify All Feedback #2Improve Feedback ● Shout it from the mountain tops ● No heroes quietly fixing things or applying workarounds ● Open, honest communication of feedback, especially of problems ○ File a bug report ○ Halt the process at that step (e.g. pull the Andon cord to stop the line) ● Make having problems OK and hiding problems a fireable offense ○ GM and all those green status sheets…
- 22. Embed Knowledge whereNeeded #2 Improve Feedback ● Keep specialized knowledge out of people’s heads and into the system ○ Special configurations, business requirements, etc ○ Check it into source control - automatic versioning! ● git blame anyone? Find out where/when regressions occurred ● Moving left to right, keep info in the stage that requires it ● Docs to build a package stored in the repo for that package ● Deploy automation in the repo with configuration templates, etc
- 23. Workflow Improve Feedback Continual Experimentation &Learning The 3 ways of DevOs 1 3 2
- 24. Create a cultureof innovation and experimentation #3 Continual Experiments ● The fundamentals are now solid, what can your new knowledge buy you? ● The business culture must allow for and embrace innovation & experimentation ● Two essential things must be understood by the business ○ We can learn from failed experiments / risks we take ○ Mastery comes from repetition and practice ● And you won’t be a master the first N times you practice
- 25. “I fear notthe man who has practiced ten thousand kicks once, But I fear the man who has practiced one kick ten thousand times.”
- 26. Rituals are createdthat reward risk taking #3 Continual Experiments ● Reward risk + learning ● Don’t just talk about rewarding risk, walk the walk ● Trying new things and failing is OK when you gain knowledge ● Consider this creating your own feedback in a very tight loop ● Get real about this: ○ Failures should be noted positively in annual reviews if and only if a lesson was learned ● Edison invented the lightbulb by running out of things that didn’t work
- 27. Mgmt allocates timefor projects to improve the system #3 Continual Experiments ● Plan to improve or you’re planning on stagnation ● Invest in improving the system created ○ By providing value to the business, it should want to maximize that return ● Prune any technical debt - all debt is not bad ○ Some is good, none has opportunity costs, too much will crush you ● Amplifying feedback helps sell this to the business ● Can keep mistakes from being repeated
- 28. Faults are introducedto increase resilience #3 Continual Experiments ● Practice emergencies so emergencies feel routine ● Think fire drills aka Chaos Monkey ● You need to be a very mature org to do this ● Wonderful feedback look ○ How would your programming change if you knew the DB could disappear at any second? ● How else to check redundancy? ○ e.g. Think ‘trying to restore from backups’ after they were encrypted by malware
- 29. Try crazy oraudacious things #3 Continual Experiments ● Stretch out of your comfort zone ● Requires embracing failures since many of these won’t work ● Forces out-of-the-box thinking ● Provides new perspectives on existing systems ○ You may thing A will break first, but B falls over instead ● Can help find false bottlenecks, bad assumptions, the dreaded “unknown unknowns” ● Yet another source of feedback so make sure and learn from it publicly
- 30. The Phoenix Project _If youhaven’t_ _read that yet… Right after this talk is over, go out and get this book & “Beyond The Phoenix Project” to read them. Period
- 31. Steps 2 Take 3 Iteratefrom zero to hero
- 32.
- 33. Key features ofAppSec Pipelines AppSec Pipelines ● Designed for iterative improvement ● Provides a re-usable path for AppSec activities ● Provides a consistent process for both the team and your constituency ● One way flow with well defined states ● Relies heavily on automation ● Grow functionality organically over time ● Gracefully interconnects with the development process (e.g customers)
- 34.
- 35. Gen 1 Pipelines Lookat your team’s purpose and those processes which aid it
- 36. Spending time optimizing anything other thanthe critical resource is an illusion _W. Edwards Deming_
- 37. AppSec Personnelle Critical Resource ●They are the critical resource - optimize their work ● Automate things that don’t require a human brain ● Drive up consistency ● Increase tracking of work status ● Increase flow through the pipeline ● Increase visibility and metrics ● Reduce any dev team friction with application security
- 38.
- 39. Then, once your house isin order…
- 40. Gen 2 Pipelines Lookoutside at your team’s purpose and those processes which aid it
- 41. Gen 2 Pipeline DevPipeline AppSec Pipeline Drop tool(s) into their pipeline
- 42. Weaponizing CICD Gen 2Pipelines ● Zero false positives ○ Pretend FPs give you anaphylactic shock ● Health Checks vs Scanning ○ Run health checks all the time ● Home of specific issue tests ○ Find a vulnerability, write a test ● Cadence for longer running tests ○ These NEVER break a build ○ Every X builds or every Y days
- 43.
- 44.
- 45. Single source oftruth for findings OWASP DefectDojo ● AppSec Programs, QA / QE, Product Security, Pen Testers ○ Custom report generation ○ Metrics and Dashboards ○ App and Infrastructure findings ○ 150+ security tools supported ● OWASP Flagship project, 8+ years being open sourced from Rackspace ● Community and contributor friendly ○ 305 contributors ● Github: 2.4k stars, 1.1k forks, monthly releases
- 46. The Heart ofyour DevSecOps OWASP DefectDojo
- 47. Gen 3 Pipelines Lookto scale your team’s reach and dramatically increase speed and visibility
- 48. What is aGen 3 Pipeline? Gen 3 Pipelines ● A way to conduct AppSec testing in an automated fashion ● Run by the AppSec team for the AppSec team to: ○ Provide visibility into software security ○ Provide security findings to the dev teams ● A means to scale the AppSec team coverage ○ Not in-depth testing but “you must be this high” ○ Allow some testing to be ‘pre-calculated’ for manual assessments ○ Event-driven (mostly) or on a schedule ● Creates a baseline of security
- 49. What a Gen3 Pipeline isn’t: Gen 3 Pipelines ● Magic pixie dust ● A gate blocking production deploys ● Something to add to existing CICD systems ● Pipelines create artifacts ○ CICD artifacts are a deployed app ○ AppSec Pipeline artifacts are security findings
- 51.
- 52.
- 53. AppSec Pipeline evolution 1 2 3 Gen1 Your team's purpose and what aids it Gen 2 Look outside your teams purpose and what aids it Gen 3 Event Driven, inter-connected pipelines
- 54. Iteration will berequired
- 55. Iteration will berequired Obligatory coffee stain
- 56. 1 Final Thought 4 Conclusionsand Key Takeaways
- 57. ● Reverse IPLookup ● IP reputation ● IP blacklist check ● Domain reputation ● Known IP (aka we own it) ● Check IP against our cloud account OK, AppSec, what else? DFIR ● Who registered the domain ● How long registered ● Shodan lookup ● Geolocate IP ● Whois ● Check against threat intel feed(s)
- 58. Useage 5,100 Runs 25k+ Container Executions Size 15Repos 4 months Event-driven Pipeline numbers
- 59. 2014 2015 2016 Numberof Assessments 44 224 414 Headcount N/A -3.5 -2 Percentage Increase N/A 450% 107% AppSec Program Numbers
- 60.
- 61. What are youwaiting for… ● Build a Pipeline & remove painful drudgery from your life ● Co-opt good ideas from other disciplines ● Get your DevSecOps on!
- 62. TLDR for thistalk
- 63. CREDITS: This presentationtemplate was created by Slidesgo, including icons by Flaticon, and infographics & images by Freepik THANKS! matt.tesauro@owasp.org mattt@nonamesecurity.com Twitter: matt_tesauro Do you have any questions?